2010-09-20 22:08:53 +02:00
<!-- doc/src/sgml/runtime.sgml -->
2000-03-31 05:27:42 +02:00
2011-01-31 20:58:49 +01:00
<chapter id="runtime">
2009-01-09 14:37:18 +01:00
<title>Server Setup and Operation</title>
2000-06-18 23:24:54 +02:00
2005-01-23 01:30:59 +01:00
<para>
2020-09-03 17:45:26 +02:00
This chapter discusses how to set up and run the database server,
2004-12-27 00:06:56 +01:00
and its interactions with the operating system.
2000-06-18 23:24:54 +02:00
</para>
2020-09-03 17:45:26 +02:00
<para>
The directions in this chapter assume that you are working with
plain <productname>PostgreSQL</productname> without any additional
infrastructure, for example a copy that you built from source
according to the directions in the preceding chapters.
If you are working with a pre-packaged or vendor-supplied
version of <productname>PostgreSQL</productname>, it is likely that
the packager has made special provisions for installing and starting
the database server according to your system's conventions.
Consult the package-level documentation for details.
</para>
2000-06-18 23:24:54 +02:00
<sect1 id="postgres-user">
2002-11-11 21:14:04 +01:00
<title>The <productname>PostgreSQL</productname> User Account</title>
2000-06-18 23:24:54 +02:00
2001-05-13 00:51:36 +02:00
<indexterm>
<primary>postgres user</primary>
</indexterm>
2000-06-18 23:24:54 +02:00
<para>
2010-02-03 18:25:06 +01:00
As with any server daemon that is accessible to the outside world,
2002-03-09 06:11:38 +01:00
it is advisable to run <productname>PostgreSQL</productname> under a
separate user account. This user account should only own the data
that is managed by the server, and should not be shared with other
2003-03-24 15:32:51 +01:00
daemons. (For example, using the user <literal>nobody</literal> is a bad
2020-09-03 17:45:26 +02:00
idea.) In particular, it is advisable that this user account not own
the <productname>PostgreSQL</productname> executable files, to ensure
that a compromised server process could not modify those executables.
</para>
<para>
Pre-packaged versions of <productname>PostgreSQL</productname> will
typically create a suitable user account automatically during
package installation.
2000-06-18 23:24:54 +02:00
</para>
1999-05-20 07:39:29 +02:00
2000-06-18 23:24:54 +02:00
<para>
2002-03-09 06:11:38 +01:00
To add a Unix user account to your system, look for a command
2000-06-18 23:24:54 +02:00
<command>useradd</command> or <command>adduser</command>. The user
2004-12-27 00:06:56 +01:00
name <systemitem>postgres</systemitem> is often used, and is assumed
throughout this book, but you can use another name if you like.
2000-06-18 23:24:54 +02:00
</para>
</sect1>
<sect1 id="creating-cluster">
2002-11-11 21:14:04 +01:00
<title>Creating a Database Cluster</title>
2000-06-18 23:24:54 +02:00
2001-05-13 00:51:36 +02:00
<indexterm>
<primary>database cluster</primary>
</indexterm>
<indexterm>
<primary>data area</primary>
<see>database cluster</see>
</indexterm>
2000-06-18 23:24:54 +02:00
<para>
Before you can do anything, you must initialize a database storage
2001-11-28 01:13:30 +01:00
area on disk. We call this a <firstterm>database cluster</firstterm>.
2015-07-29 00:42:59 +02:00
(The <acronym>SQL</acronym> standard uses the term catalog cluster.) A
2004-12-27 00:06:56 +01:00
database cluster is a collection of databases that is managed by a
2002-03-09 06:11:38 +01:00
single instance of a running database server. After initialization, a
2005-06-21 06:02:34 +02:00
database cluster will contain a database named <literal>postgres</literal>,
which is meant as a default database for use by utilities, users and third
party applications. The database server itself does not require the
<literal>postgres</literal> database to exist, but many external utility
2021-11-02 17:54:35 +01:00
programs assume it exists. There are two more databases created within
each cluster during initialization, named <literal>template1</literal>
and <literal>template0</literal>. As the names suggest, these will be
used as templates for subsequently-created databases; they should not be
2017-11-23 15:39:47 +01:00
used for actual work. (See <xref linkend="managing-databases"/> for
2004-12-27 00:06:56 +01:00
information about creating new databases within a cluster.)
1999-05-20 07:39:29 +02:00
</para>
2000-06-18 23:24:54 +02:00
<para>
2015-07-29 00:42:59 +02:00
In file system terms, a database cluster is a single directory
2001-11-28 01:13:30 +01:00
under which all data will be stored. We call this the <firstterm>data
directory</firstterm> or <firstterm>data area</firstterm>. It is
2001-12-27 22:37:34 +01:00
completely up to you where you choose to store your data. There is no
2001-11-28 01:13:30 +01:00
default, although locations such as
2000-06-18 23:24:54 +02:00
<filename>/usr/local/pgsql/data</filename> or
2020-09-03 17:45:26 +02:00
<filename>/var/lib/pgsql/data</filename> are popular.
The data directory must be initialized before being used, using the program
<xref linkend="app-initdb"/><indexterm><primary>initdb</primary></indexterm>
which is installed with <productname>PostgreSQL</productname>.
</para>
<para>
If you are using a pre-packaged version
of <productname>PostgreSQL</productname>, it may well have a specific
convention for where to place the data directory, and it may also
provide a script for creating the data directory. In that case you
should use that script in preference to
running <command>initdb</command> directly.
Consult the package-level documentation for details.
</para>
<para>
To initialize a database cluster manually,
run <command>initdb</command> and specify the desired
file system location of the database cluster with the
2009-04-27 18:27:36 +02:00
<option>-D</option> option, for example:
2000-06-18 23:24:54 +02:00
<screen>
2017-10-09 03:44:17 +02:00
<prompt>$</prompt> <userinput>initdb -D /usr/local/pgsql/data</userinput>
2000-06-18 23:24:54 +02:00
</screen>
2002-03-09 06:11:38 +01:00
Note that you must execute this command while logged into the
<productname>PostgreSQL</productname> user account, which is
described in the previous section.
2000-06-18 23:24:54 +02:00
</para>
1999-05-20 07:39:29 +02:00
2000-06-18 23:24:54 +02:00
<tip>
1999-05-20 07:39:29 +02:00
<para>
2000-06-18 23:24:54 +02:00
As an alternative to the <option>-D</option> option, you can set
the environment variable <envar>PGDATA</envar>.
2003-03-24 15:32:51 +01:00
<indexterm><primary><envar>PGDATA</envar></primary></indexterm>
1999-05-20 07:39:29 +02:00
</para>
2000-06-18 23:24:54 +02:00
</tip>
2009-12-10 07:32:28 +01:00
<para>
Alternatively, you can run <command>initdb</command> via
2017-11-23 15:39:47 +01:00
the <xref linkend="app-pg-ctl"/>
2017-10-09 03:44:17 +02:00
program<indexterm><primary>pg_ctl</primary></indexterm> like so:
2009-12-10 07:32:28 +01:00
<screen>
2017-10-09 03:44:17 +02:00
<prompt>$</prompt> <userinput>pg_ctl -D /usr/local/pgsql/data initdb</userinput>
2009-12-10 07:32:28 +01:00
</screen>
This may be more intuitive if you are
using <command>pg_ctl</command> for starting and stopping the
2017-11-23 15:39:47 +01:00
server (see <xref linkend="server-start"/>), so
2009-12-10 07:32:28 +01:00
that <command>pg_ctl</command> would be the sole command you use
for managing the database server instance.
</para>
2000-06-18 23:24:54 +02:00
<para>
<command>initdb</command> will attempt to create the directory you
2015-07-29 00:42:59 +02:00
specify if it does not already exist. Of course, this will fail if
<command>initdb</command> does not have permissions to write in the
parent directory. It's generally recommendable that the
<productname>PostgreSQL</productname> user own not just the data
directory but its parent directory as well, so that this should not
be a problem. If the desired parent directory doesn't exist either,
you will need to create it first, using root privileges if the
grandparent directory isn't writable. So the process might look
like this:
2000-06-18 23:24:54 +02:00
<screen>
2015-07-29 00:42:59 +02:00
root# <userinput>mkdir /usr/local/pgsql</userinput>
root# <userinput>chown postgres /usr/local/pgsql</userinput>
2000-06-18 23:24:54 +02:00
root# <userinput>su postgres</userinput>
2001-11-28 21:49:10 +01:00
postgres$ <userinput>initdb -D /usr/local/pgsql/data</userinput>
2000-06-18 23:24:54 +02:00
</screen>
</para>
1999-05-20 07:39:29 +02:00
2000-06-18 23:24:54 +02:00
<para>
<command>initdb</command> will refuse to run if the data directory
2015-07-29 00:42:59 +02:00
exists and already contains files; this is to prevent accidentally
overwriting an existing installation.
</para>
2000-06-18 23:24:54 +02:00
<para>
Because the data directory contains all the data stored in the
2002-03-09 06:11:38 +01:00
database, it is essential that it be secured from unauthorized
2000-06-18 23:24:54 +02:00
access. <command>initdb</command> therefore revokes access
2002-03-09 06:11:38 +01:00
permissions from everyone but the
2018-04-07 23:45:39 +02:00
<productname>PostgreSQL</productname> user, and optionally, group.
Group access, when enabled, is read-only. This allows an unprivileged
user in the same group as the cluster owner to take a backup of the
cluster data or perform other operations that only require read access.
</para>
<para>
Note that enabling or disabling group access on an existing cluster requires
the cluster to be shut down and the appropriate mode to be set on all
directories and files before restarting
<productname>PostgreSQL</productname>. Otherwise, a mix of modes might
exist in the data directory. For clusters that allow access only by the
owner, the appropriate modes are <literal>0700</literal> for directories
and <literal>0600</literal> for files. For clusters that also allow
reads by the group, the appropriate modes are <literal>0750</literal>
for directories and <literal>0640</literal> for files.
2000-06-18 23:24:54 +02:00
</para>
2000-11-25 21:33:54 +01:00
2001-11-28 01:13:30 +01:00
<para>
2019-07-22 19:28:25 +02:00
However, while the directory contents are secure, the default
client authentication setup allows any local user to connect to the
database and even become the database superuser. If you do not
trust other local users, we recommend you use one of
2004-06-24 21:26:59 +02:00
<command>initdb</command>'s <option>-W</option>, <option>--pwprompt</option>
2019-07-22 19:28:25 +02:00
or <option>--pwfile</option> options to assign a password to the
database superuser.<indexterm>
2017-10-09 03:44:17 +02:00
<primary>password</primary>
<secondary>of the superuser</secondary>
2011-04-08 17:36:05 +02:00
</indexterm>
2022-05-31 21:59:47 +02:00
Also, specify <option>-A scram-sha-256</option>
so that the default <literal>trust</literal> authentication
2019-07-22 19:28:25 +02:00
mode is not used; or modify the generated <filename>pg_hba.conf</filename>
file after running <command>initdb</command>, but
<emphasis>before</emphasis> you start the server for the first time. (Other
reasonable approaches include using <literal>peer</literal> authentication
or file system permissions to restrict connections. See <xref
linkend="client-authentication"/> for more information.)
2001-11-28 01:13:30 +01:00
</para>
2000-11-25 21:33:54 +01:00
<para>
2003-05-15 17:50:21 +02:00
<command>initdb</command> also initializes the default
2017-10-09 03:44:17 +02:00
locale<indexterm><primary>locale</primary></indexterm> for the database cluster.
2003-05-15 17:50:21 +02:00
Normally, it will just take the locale settings in the environment
and apply them to the initialized database. It is possible to
specify a different locale for the database; more information about
2017-11-23 15:39:47 +01:00
that can be found in <xref linkend="locale"/>. The default sort order used
2008-09-23 11:20:39 +02:00
within the particular database cluster is set by
<command>initdb</command>, and while you can create new databases using
different sort order, the order used in the template databases that initdb
creates cannot be changed without dropping and recreating them.
There is also a performance impact for using locales
2017-10-09 03:44:17 +02:00
other than <literal>C</literal> or <literal>POSIX</literal>. Therefore, it is
2005-01-04 01:05:45 +01:00
important to make this choice correctly the first time.
2000-11-25 21:33:54 +01:00
</para>
2005-01-08 23:13:38 +01:00
<para>
<command>initdb</command> also sets the default character set encoding
for the database cluster. Normally this should be chosen to match the
2017-11-23 15:39:47 +01:00
locale setting. For details see <xref linkend="multibyte"/>.
2005-01-08 23:13:38 +01:00
</para>
2007-11-04 22:48:03 +01:00
2016-07-02 17:22:36 +02:00
<para>
2017-10-09 03:44:17 +02:00
Non-<literal>C</literal> and non-<literal>POSIX</literal> locales rely on the
2016-07-02 17:22:36 +02:00
operating system's collation library for character set ordering.
This controls the ordering of keys stored in indexes. For this reason,
a cluster cannot switch to an incompatible collation library version,
2016-08-02 23:13:10 +02:00
either through snapshot restore, binary streaming replication, a
different operating system, or an operating system upgrade.
2016-07-02 17:22:36 +02:00
</para>
2015-07-29 00:42:59 +02:00
<sect2 id="creating-cluster-mount-points">
<title>Use of Secondary File Systems</title>
<indexterm zone="creating-cluster-mount-points">
<primary>file system mount points</primary>
</indexterm>
<para>
Many installations create their database clusters on file systems
2017-10-09 03:44:17 +02:00
(volumes) other than the machine's <quote>root</quote> volume. If you
2015-07-29 00:42:59 +02:00
choose to do this, it is not advisable to try to use the secondary
volume's topmost directory (mount point) as the data directory.
Best practice is to create a directory within the mount-point
directory that is owned by the <productname>PostgreSQL</productname>
user, and then create the data directory within that. This avoids
permissions problems, particularly for operations such
2017-10-09 03:44:17 +02:00
as <application>pg_upgrade</application>, and it also ensures clean failures if
2015-07-29 00:42:59 +02:00
the secondary volume is taken offline.
</para>
</sect2>
2019-04-26 11:50:16 +02:00
<sect2 id="creating-cluster-filesystem">
<title>File Systems</title>
2015-07-29 00:42:59 +02:00
<para>
2019-04-26 11:50:16 +02:00
Generally, any file system with POSIX semantics can be used for
PostgreSQL. Users prefer different file systems for a variety of reasons,
including vendor support, performance, and familiarity. Experience
suggests that, all other things being equal, one should not expect major
performance or behavior changes merely from switching file systems or
making minor file system configuration changes.
2007-11-08 16:21:03 +01:00
</para>
2019-04-26 11:50:16 +02:00
<sect3 id="creating-cluster-nfs">
<title>NFS</title>
<indexterm zone="creating-cluster-nfs">
<primary>NFS</primary>
</indexterm>
<para>
It is possible to use an <acronym>NFS</acronym> file system for storing
the <productname>PostgreSQL</productname> data directory.
<productname>PostgreSQL</productname> does nothing special for
<acronym>NFS</acronym> file systems, meaning it assumes
<acronym>NFS</acronym> behaves exactly like locally-connected drives.
<productname>PostgreSQL</productname> does not use any functionality that
is known to have nonstandard behavior on <acronym>NFS</acronym>, such as
file locking.
</para>
<para>
The only firm requirement for using <acronym>NFS</acronym> with
<productname>PostgreSQL</productname> is that the file system is mounted
using the <literal>hard</literal> option. With the
<literal>hard</literal> option, processes can <quote>hang</quote>
indefinitely if there are network problems, so this configuration will
require a careful monitoring setup. The <literal>soft</literal> option
will interrupt system calls in case of network problems, but
<productname>PostgreSQL</productname> will not repeat system calls
interrupted in this way, so any such interruption will result in an I/O
error being reported.
</para>
<para>
It is not necessary to use the <literal>sync</literal> mount option. The
behavior of the <literal>async</literal> option is sufficient, since
<productname>PostgreSQL</productname> issues <literal>fsync</literal>
calls at appropriate times to flush the write caches. (This is analogous
to how it works on a local file system.) However, it is strongly
recommended to use the <literal>sync</literal> export option on the NFS
<emphasis>server</emphasis> on systems where it exists (mainly Linux).
Otherwise, an <literal>fsync</literal> or equivalent on the NFS client is
not actually guaranteed to reach permanent storage on the server, which
could cause corruption similar to running with the parameter <xref
linkend="guc-fsync"/> off. The defaults of these mount and export
options differ between vendors and versions, so it is recommended to
check and perhaps specify them explicitly in any case to avoid any
ambiguity.
</para>
<para>
In some cases, an external storage product can be accessed either via NFS
or a lower-level protocol such as iSCSI. In the latter case, the storage
appears as a block device and any available file system can be created on
it. That approach might relieve the DBA from having to deal with some of
the idiosyncrasies of NFS, but of course the complexity of managing
remote storage then happens at other levels.
</para>
</sect3>
2007-11-08 16:21:03 +01:00
</sect2>
2007-11-04 22:48:03 +01:00
2000-06-18 23:24:54 +02:00
</sect1>
2006-06-18 17:38:37 +02:00
<sect1 id="server-start">
2002-11-11 21:14:04 +01:00
<title>Starting the Database Server</title>
2000-06-18 23:24:54 +02:00
<para>
2002-03-09 06:11:38 +01:00
Before anyone can access the database, you must start the database
2003-03-24 15:32:51 +01:00
server. The database server program is called
2017-10-09 03:44:17 +02:00
<command>postgres</command>.<indexterm><primary>postgres</primary></indexterm>
2020-09-03 17:45:26 +02:00
</para>
<para>
If you are using a pre-packaged version
of <productname>PostgreSQL</productname>, it almost certainly includes
provisions for running the server as a background task according to the
conventions of your operating system. Using the package's
infrastructure to start the server will be much less work than figuring
out how to do this yourself. Consult the package-level documentation
for details.
</para>
<para>
The bare-bones way to start the server manually is just to invoke
<command>postgres</command> directly, specifying the location of the
data directory with the <option>-D</option> option, for example:
2000-06-18 23:24:54 +02:00
<screen>
2006-06-18 17:38:37 +02:00
$ <userinput>postgres -D /usr/local/pgsql/data</userinput>
2000-06-18 23:24:54 +02:00
</screen>
2002-03-09 06:11:38 +01:00
which will leave the server running in the foreground. This must be
done while logged into the <productname>PostgreSQL</productname> user
account. Without <option>-D</option>, the server will try to use
2004-04-29 06:37:09 +02:00
the data directory named by the environment variable <envar>PGDATA</envar>.
If that variable is not provided either, it will fail.
2000-06-18 23:24:54 +02:00
</para>
<para>
2006-06-18 17:38:37 +02:00
Normally it is better to start <command>postgres</command> in the
2010-02-03 18:25:06 +01:00
background. For this, use the usual Unix shell syntax:
2000-06-18 23:24:54 +02:00
<screen>
2006-06-18 17:38:37 +02:00
$ <userinput>postgres -D /usr/local/pgsql/data >logfile 2>&1 &</userinput>
2000-06-18 23:24:54 +02:00
</screen>
2017-10-09 03:44:17 +02:00
It is important to store the server's <systemitem>stdout</systemitem> and
<systemitem>stderr</systemitem> output somewhere, as shown above. It will help
2002-03-09 06:11:38 +01:00
for auditing purposes and to diagnose problems. (See <xref
2017-11-23 15:39:47 +01:00
linkend="logfile-maintenance"/> for a more thorough discussion of log
2002-03-09 06:11:38 +01:00
file handling.)
2000-06-18 23:24:54 +02:00
</para>
<para>
2006-06-18 17:38:37 +02:00
The <command>postgres</command> program also takes a number of other
command-line options. For more information, see the
2017-11-23 15:39:47 +01:00
<xref linkend="app-postgres"/> reference page
and <xref linkend="runtime-config"/> below.
2000-06-18 23:24:54 +02:00
</para>
2001-02-10 01:50:18 +01:00
<para>
2004-12-27 00:06:56 +01:00
This shell syntax can get tedious quickly. Therefore the wrapper
program
2017-11-23 15:39:47 +01:00
<xref linkend="app-pg-ctl"/><indexterm><primary>pg_ctl</primary></indexterm>
2003-03-24 15:32:51 +01:00
is provided to simplify some tasks. For example:
2001-02-10 01:50:18 +01:00
<programlisting>
pg_ctl start -l logfile
</programlisting>
will start the server in the background and put the output into the
2002-03-09 06:11:38 +01:00
named log file. The <option>-D</option> option has the same meaning
2006-06-18 17:38:37 +02:00
here as for <command>postgres</command>. <command>pg_ctl</command>
2004-12-27 00:06:56 +01:00
is also capable of stopping the server.
2001-02-10 01:50:18 +01:00
</para>
2000-06-18 23:24:54 +02:00
<para>
Normally, you will want to start the database server when the
2011-04-08 17:36:05 +02:00
computer boots.<indexterm>
2017-10-09 03:44:17 +02:00
<primary>booting</primary>
<secondary>starting the server during</secondary>
2011-04-08 17:36:05 +02:00
</indexterm>
Autostart scripts are operating-system-specific.
2020-09-03 17:45:26 +02:00
There are a few example scripts distributed with
2002-03-09 06:11:38 +01:00
<productname>PostgreSQL</productname> in the
2017-10-09 03:44:17 +02:00
<filename>contrib/start-scripts</filename> directory. Installing one will require
2004-04-29 06:37:09 +02:00
root privileges.
2000-06-18 23:24:54 +02:00
</para>
<para>
2002-03-09 06:11:38 +01:00
Different systems have different conventions for starting up daemons
at boot time. Many systems have a file
2000-06-18 23:24:54 +02:00
<filename>/etc/rc.local</filename> or
2011-07-14 18:07:39 +02:00
<filename>/etc/rc.d/rc.local</filename>. Others use <filename>init.d</filename> or
2017-10-09 03:44:17 +02:00
<filename>rc.d</filename> directories. Whatever you do, the server must be
2002-03-09 06:11:38 +01:00
run by the <productname>PostgreSQL</productname> user account
<emphasis>and not by root</emphasis> or any other user. Therefore you
2012-09-25 19:52:53 +02:00
probably should form your commands using
<literal>su postgres -c '...'</literal>. For example:
2000-06-18 23:24:54 +02:00
<programlisting>
2012-09-25 19:52:53 +02:00
su postgres -c 'pg_ctl start -D /usr/local/pgsql/data -l serverlog'
2000-06-18 23:24:54 +02:00
</programlisting>
</para>
<para>
2004-04-29 06:37:09 +02:00
Here are a few more operating-system-specific suggestions. (In each
case be sure to use the proper installation directory and user
name where we show generic values.)
2000-06-18 23:24:54 +02:00
<itemizedlist>
<listitem>
<para>
2002-03-09 06:11:38 +01:00
For <productname>FreeBSD</productname>, look at the file
2001-02-10 01:50:18 +01:00
<filename>contrib/start-scripts/freebsd</filename> in the
<productname>PostgreSQL</productname> source distribution.
2017-10-09 03:44:17 +02:00
<indexterm><primary>FreeBSD</primary><secondary>start script</secondary></indexterm>
2001-02-10 01:50:18 +01:00
</para>
</listitem>
<listitem>
<para>
On <productname>OpenBSD</productname>, add the following lines
to the file <filename>/etc/rc.local</filename>:
2017-10-09 03:44:17 +02:00
<indexterm><primary>OpenBSD</primary><secondary>start script</secondary></indexterm>
2000-06-18 23:24:54 +02:00
<programlisting>
2006-06-18 17:38:37 +02:00
if [ -x /usr/local/pgsql/bin/pg_ctl -a -x /usr/local/pgsql/bin/postgres ]; then
2012-09-25 19:52:53 +02:00
su -l postgres -c '/usr/local/pgsql/bin/pg_ctl start -s -l /var/postgresql/log -D /usr/local/pgsql/data'
2001-02-10 01:50:18 +01:00
echo -n ' postgresql'
fi
2000-06-18 23:24:54 +02:00
</programlisting>
</para>
</listitem>
<listitem>
<para>
2001-02-10 01:50:18 +01:00
On <productname>Linux</productname> systems either add
2017-10-09 03:44:17 +02:00
<indexterm><primary>Linux</primary><secondary>start script</secondary></indexterm>
2000-06-18 23:24:54 +02:00
<programlisting>
2001-02-10 01:50:18 +01:00
/usr/local/pgsql/bin/pg_ctl start -l logfile -D /usr/local/pgsql/data
2000-06-18 23:24:54 +02:00
</programlisting>
2011-07-14 18:07:39 +02:00
to <filename>/etc/rc.d/rc.local</filename>
or <filename>/etc/rc.local</filename> or look at the file
2001-02-10 01:50:18 +01:00
<filename>contrib/start-scripts/linux</filename> in the
2002-03-09 06:11:38 +01:00
<productname>PostgreSQL</productname> source distribution.
2000-06-18 23:24:54 +02:00
</para>
2015-11-17 12:46:17 +01:00
<para>
When using <application>systemd</application>, you can use the following
service unit file (e.g.,
at <filename>/etc/systemd/system/postgresql.service</filename>):<indexterm><primary>systemd</primary></indexterm>
<programlisting>
[Unit]
Description=PostgreSQL database server
Documentation=man:postgres(1)
2022-10-12 16:51:11 +02:00
After=network-online.target
Wants=network-online.target
2015-11-17 12:46:17 +01:00
[Service]
Type=notify
User=postgres
ExecStart=/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT
2022-03-20 17:39:40 +01:00
TimeoutSec=infinity
2015-11-17 12:46:17 +01:00
[Install]
WantedBy=multi-user.target
</programlisting>
Using <literal>Type=notify</literal> requires that the server binary was
built with <literal>configure --with-systemd</literal>.
</para>
<para>
Consider carefully the timeout
setting. <application>systemd</application> has a default timeout of 90
2022-03-20 17:39:40 +01:00
seconds as of this writing and will kill a process that does not report
2015-11-17 12:46:17 +01:00
readiness within that time. But a <productname>PostgreSQL</productname>
server that might have to perform crash recovery at startup could take
2022-03-20 17:39:40 +01:00
much longer to become ready. The suggested value
of <literal>infinity</literal> disables the timeout logic.
2015-11-17 12:46:17 +01:00
</para>
2000-06-18 23:24:54 +02:00
</listitem>
<listitem>
<para>
2011-12-01 02:55:00 +01:00
On <productname>NetBSD</productname>, use either the
2001-02-10 01:50:18 +01:00
<productname>FreeBSD</productname> or
<productname>Linux</productname> start scripts, depending on
2011-12-01 02:55:00 +01:00
preference.
2017-10-09 03:44:17 +02:00
<indexterm><primary>NetBSD</primary><secondary>start script</secondary></indexterm>
2001-02-10 01:50:18 +01:00
</para>
</listitem>
<listitem>
<para>
2001-05-07 17:55:27 +02:00
On <productname>Solaris</productname>, create a file called
2003-03-24 15:32:51 +01:00
<filename>/etc/init.d/postgresql</filename> that contains
2002-03-09 06:11:38 +01:00
the following line:
2017-10-09 03:44:17 +02:00
<indexterm><primary>Solaris</primary><secondary>start script</secondary></indexterm>
2001-02-10 01:50:18 +01:00
<programlisting>
su - postgres -c "/usr/local/pgsql/bin/pg_ctl start -l logfile -D /usr/local/pgsql/data"
</programlisting>
2017-10-09 03:44:17 +02:00
Then, create a symbolic link to it in <filename>/etc/rc3.d</filename> as
<filename>S99postgresql</filename>.
2001-02-10 01:50:18 +01:00
</para>
</listitem>
</itemizedlist>
</para>
1999-05-20 07:39:29 +02:00
2000-06-18 23:24:54 +02:00
<para>
2006-06-18 17:38:37 +02:00
While the server is running, its
2003-03-24 15:32:51 +01:00
<acronym>PID</acronym> is stored in the file
2002-03-09 06:11:38 +01:00
<filename>postmaster.pid</filename> in the data directory. This is
2006-06-18 17:38:37 +02:00
used to prevent multiple server instances from
2003-03-24 15:32:51 +01:00
running in the same data directory and can also be used for
2006-06-18 17:38:37 +02:00
shutting down the server.
1999-05-20 07:39:29 +02:00
</para>
2006-06-18 17:38:37 +02:00
<sect2 id="server-start-failures">
2000-08-29 22:02:09 +02:00
<title>Server Start-up Failures</title>
2000-06-18 23:24:54 +02:00
<para>
2003-03-24 15:32:51 +01:00
There are several common reasons the server might fail to
start. Check the server's log file, or start it by hand (without
redirecting standard output or standard error) and see what error
messages appear. Below we explain some of the most common error
messages in more detail.
2000-06-18 23:24:54 +02:00
</para>
<para>
<screen>
2017-03-10 22:32:18 +01:00
LOG: could not bind IPv4 address "127.0.0.1": Address already in use
2003-09-13 00:17:24 +02:00
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2017-03-10 22:32:18 +01:00
FATAL: could not create any TCP/IP sockets
2000-06-18 23:24:54 +02:00
</screen>
2002-03-09 06:11:38 +01:00
This usually means just what it suggests: you tried to start
2006-06-18 17:38:37 +02:00
another server on the same port where one is already running.
2002-03-09 06:11:38 +01:00
However, if the kernel error message is not <computeroutput>Address
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
already in use</computeroutput> or some variant of that, there might
2006-06-18 17:38:37 +02:00
be a different problem. For example, trying to start a server
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
on a reserved port number might draw something like:
2000-06-18 23:24:54 +02:00
<screen>
2006-06-18 17:38:37 +02:00
$ <userinput>postgres -p 666</userinput>
2017-03-10 22:32:18 +01:00
LOG: could not bind IPv4 address "127.0.0.1": Permission denied
2003-09-13 00:17:24 +02:00
HINT: Is another postmaster already running on port 666? If not, wait a few seconds and retry.
2017-03-10 22:32:18 +01:00
FATAL: could not create any TCP/IP sockets
2000-06-18 23:24:54 +02:00
</screen>
</para>
<para>
2009-04-27 18:27:36 +02:00
A message like:
2000-06-18 23:24:54 +02:00
<screen>
2003-09-13 00:17:24 +02:00
FATAL: could not create shared memory segment: Invalid argument
2003-11-04 10:55:39 +01:00
DETAIL: Failed system call was shmget(key=5440001, size=4011376640, 03600).
2000-06-18 23:24:54 +02:00
</screen>
2002-03-09 06:11:38 +01:00
probably means your kernel's limit on the size of shared memory is
2003-09-13 00:17:24 +02:00
smaller than the work area <productname>PostgreSQL</productname>
2020-06-07 11:36:43 +02:00
is trying to create (4011376640 bytes in this example).
This is only likely to happen if you have set <literal>shared_memory_type</literal>
to <literal>sysv</literal>. In that case, you
2005-08-30 17:48:28 +02:00
can try starting the server with a smaller-than-normal number of
2020-06-07 11:36:43 +02:00
buffers (<xref linkend="guc-shared-buffers"/>), or
reconfigure your kernel to increase the allowed shared memory
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
size. You might also see this message when trying to start multiple
2003-09-13 00:17:24 +02:00
servers on the same machine, if their total space requested
2002-03-09 06:11:38 +01:00
exceeds the kernel limit.
2000-06-18 23:24:54 +02:00
</para>
<para>
2009-04-27 18:27:36 +02:00
An error like:
2000-06-18 23:24:54 +02:00
<screen>
2003-09-13 00:17:24 +02:00
FATAL: could not create semaphores: No space left on device
2003-11-04 10:55:39 +01:00
DETAIL: Failed system call was semget(5440126, 17, 03600).
2000-06-18 23:24:54 +02:00
</screen>
2003-01-11 06:04:14 +01:00
does <emphasis>not</emphasis> mean you've run out of disk
space. It means your kernel's limit on the number of <systemitem
2017-10-09 03:44:17 +02:00
class="osname">System V</systemitem> semaphores is smaller than the number
2003-01-11 06:04:14 +01:00
<productname>PostgreSQL</productname> wants to create. As above,
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
you might be able to work around the problem by starting the
2003-03-24 15:32:51 +01:00
server with a reduced number of allowed connections
2017-11-23 15:39:47 +01:00
(<xref linkend="guc-max-connections"/>), but you'll eventually want to
2000-06-18 23:24:54 +02:00
increase the kernel limit.
</para>
2000-07-22 16:49:01 +02:00
<para>
2017-10-09 03:44:17 +02:00
Details about configuring <systemitem class="osname">System V</systemitem>
2017-11-23 15:39:47 +01:00
<acronym>IPC</acronym> facilities are given in <xref linkend="sysvipc"/>.
2000-07-22 16:49:01 +02:00
</para>
2000-06-18 23:24:54 +02:00
</sect2>
<sect2 id="client-connection-problems">
<title>Client Connection Problems</title>
<para>
2002-03-09 06:11:38 +01:00
Although the error conditions possible on the client side are quite
varied and application-dependent, a few of them might be directly
2010-02-03 18:25:06 +01:00
related to how the server was started. Conditions other than
2002-03-09 06:11:38 +01:00
those shown below should be documented with the respective client
application.
2000-06-18 23:24:54 +02:00
</para>
<para>
<screen>
2021-01-23 21:50:51 +01:00
psql: error: connection to server at "server.joe.com" (123.123.123.123), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
2000-06-18 23:24:54 +02:00
</screen>
This is the generic <quote>I couldn't find a server to talk
to</quote> failure. It looks like the above when TCP/IP
2003-03-24 15:32:51 +01:00
communication is attempted. A common mistake is to forget to
configure the server to allow TCP/IP connections.
2000-06-18 23:24:54 +02:00
</para>
<para>
2021-01-23 21:50:51 +01:00
Alternatively, you might get this when attempting Unix-domain socket
2003-03-24 15:32:51 +01:00
communication to a local server:
2000-06-18 23:24:54 +02:00
<screen>
2021-01-23 21:50:51 +01:00
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
2000-06-18 23:24:54 +02:00
</screen>
2021-01-23 21:50:51 +01:00
If the server is indeed running, check that the client's idea of the
socket path (here <literal>/tmp</literal>) agrees with the server's
<xref linkend="guc-unix-socket-directories"/> setting.
2000-06-18 23:24:54 +02:00
</para>
<para>
2021-01-23 21:50:51 +01:00
A connection failure message always shows the server address or socket
path name, which is useful in verifying that the client is trying to
2003-03-24 15:32:51 +01:00
connect to the right place. If there is in fact no server
2021-01-23 21:50:51 +01:00
listening there, the kernel error message will typically be either
2002-03-09 06:11:38 +01:00
<computeroutput>Connection refused</computeroutput> or
2000-06-18 23:24:54 +02:00
<computeroutput>No such file or directory</computeroutput>, as
2002-03-09 06:11:38 +01:00
illustrated. (It is important to realize that
<computeroutput>Connection refused</computeroutput> in this context
2003-03-24 15:32:51 +01:00
does <emphasis>not</emphasis> mean that the server got your
connection request and rejected it. That case will produce a
2002-03-09 06:11:38 +01:00
different message, as shown in <xref
2017-11-23 15:39:47 +01:00
linkend="client-authentication-problems"/>.) Other error messages
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
such as <computeroutput>Connection timed out</computeroutput> might
2000-06-18 23:24:54 +02:00
indicate more fundamental problems, like lack of network
2021-01-23 21:50:51 +01:00
connectivity, or a firewall blocking the connection.
2000-06-18 23:24:54 +02:00
</para>
</sect2>
1999-05-20 07:39:29 +02:00
</sect1>
2000-07-22 16:49:01 +02:00
<sect1 id="kernel-resources">
<title>Managing Kernel Resources</title>
<para>
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> can sometimes exhaust various operating system
2012-07-04 21:56:12 +02:00
resource limits, especially when multiple copies of the server are running
on the same system, or in very large installations. This section explains
2017-10-09 03:44:17 +02:00
the kernel resources used by <productname>PostgreSQL</productname> and the steps you
2012-07-04 21:56:12 +02:00
can take to resolve problems related to kernel resource consumption.
2000-07-22 16:49:01 +02:00
</para>
<sect2 id="sysvipc">
<title>Shared Memory and Semaphores</title>
2001-05-13 00:51:36 +02:00
<indexterm zone="sysvipc">
<primary>shared memory</primary>
</indexterm>
<indexterm zone="sysvipc">
<primary>semaphores</primary>
</indexterm>
2000-07-22 16:49:01 +02:00
<para>
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> requires the operating system to provide
inter-process communication (<acronym>IPC</acronym>) features, specifically
2016-10-10 21:11:33 +02:00
shared memory and semaphores. Unix-derived systems typically provide
2017-10-09 03:44:17 +02:00
<quote><systemitem class="osname">System V</systemitem></quote> <acronym>IPC</acronym>,
<quote><systemitem class="osname">POSIX</systemitem></quote> <acronym>IPC</acronym>, or both.
<systemitem class="osname">Windows</systemitem> has its own implementation of
2016-10-10 21:11:33 +02:00
these features and is not discussed here.
2000-07-22 16:49:01 +02:00
</para>
<para>
2019-02-03 09:55:39 +01:00
By default, <productname>PostgreSQL</productname> allocates
2016-10-10 21:11:33 +02:00
a very small amount of System V shared memory, as well as a much larger
2019-02-03 09:55:39 +01:00
amount of anonymous <function>mmap</function> shared memory.
Alternatively, a single large System V shared memory region can be used
(see <xref linkend="guc-shared-memory-type"/>).
2019-04-08 22:27:35 +02:00
2016-10-10 21:11:33 +02:00
In addition a significant number of semaphores, which can be either
System V or POSIX style, are created at server startup. Currently,
POSIX semaphores are used on Linux and FreeBSD systems while other
platforms use System V semaphores.
</para>
<para>
2017-10-09 03:44:17 +02:00
System V <acronym>IPC</acronym> features are typically constrained by
2016-10-10 21:11:33 +02:00
system-wide allocation limits.
2017-10-09 03:44:17 +02:00
When <productname>PostgreSQL</productname> exceeds one of these limits,
2016-10-10 21:11:33 +02:00
the server will refuse to start and
2002-03-09 06:11:38 +01:00
should leave an instructive error message describing the problem
2010-02-03 18:25:06 +01:00
and what to do about it. (See also <xref
2017-11-23 15:39:47 +01:00
linkend="server-start-failures"/>.) The relevant kernel
2002-03-09 06:11:38 +01:00
parameters are named consistently across different systems; <xref
2017-11-23 15:39:47 +01:00
linkend="sysvipc-parameters"/> gives an overview. The methods to set
2002-03-09 06:11:38 +01:00
them, however, vary. Suggestions for some platforms are given below.
2000-07-22 16:49:01 +02:00
</para>
<table id="sysvipc-parameters">
2017-10-09 03:44:17 +02:00
<title><systemitem class="osname">System V</systemitem> <acronym>IPC</acronym> Parameters</title>
2000-07-22 16:49:01 +02:00
<tgroup cols="3">
2020-05-06 18:23:43 +02:00
<colspec colname="col1" colwidth="1*"/>
<colspec colname="col2" colwidth="3*"/>
<colspec colname="col3" colwidth="3*"/>
2000-07-22 16:49:01 +02:00
<thead>
<row>
2017-10-09 03:44:17 +02:00
<entry>Name</entry>
<entry>Description</entry>
<entry>Values needed to run one <productname>PostgreSQL</productname> instance</entry>
2000-07-22 16:49:01 +02:00
</row>
</thead>
<tbody>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SHMMAX</varname></entry>
<entry>Maximum size of shared memory segment (bytes)</entry>
2016-10-10 21:11:33 +02:00
<entry>at least 1kB, but the default is usually much higher</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SHMMIN</varname></entry>
<entry>Minimum size of shared memory segment (bytes)</entry>
<entry>1</entry>
2000-07-22 16:49:01 +02:00
</row>
2001-05-02 19:04:53 +02:00
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SHMALL</varname></entry>
<entry>Total amount of shared memory available (bytes or pages)</entry>
2016-10-10 21:11:33 +02:00
<entry>same as <varname>SHMMAX</varname> if bytes,
or <literal>ceil(SHMMAX/PAGE_SIZE)</literal> if pages,
2017-10-09 03:44:17 +02:00
plus room for other applications</entry>
2001-05-02 19:04:53 +02:00
</row>
2000-07-22 16:49:01 +02:00
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SHMSEG</varname></entry>
<entry>Maximum number of shared memory segments per process</entry>
<entry>only 1 segment is needed, but the default is much higher</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SHMMNI</varname></entry>
<entry>Maximum number of shared memory segments system-wide</entry>
<entry>like <varname>SHMSEG</varname> plus room for other applications</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SEMMNI</varname></entry>
<entry>Maximum number of semaphore identifiers (i.e., sets)</entry>
2019-06-09 04:25:56 +02:00
<entry>at least <literal>ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16)</literal> plus room for other applications</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SEMMNS</varname></entry>
<entry>Maximum number of semaphores system-wide</entry>
Move max_wal_senders out of max_connections for connection slot handling
Since its introduction, max_wal_senders is counted as part of
max_connections when it comes to define how many connection slots can be
used for replication connections with a WAL sender context. This can
lead to confusion for some users, as it could be possible to block a
base backup or replication from happening because other backend sessions
are already taken for other purposes by an application, and
superuser-only connection slots are not a correct solution to handle
that case.
This commit makes max_wal_senders independent of max_connections for its
handling of PGPROC entries in ProcGlobal, meaning that connection slots
for WAL senders are handled using their own free queue, like autovacuum
workers and bgworkers.
One compatibility issue that this change creates is that a standby now
requires to have a value of max_wal_senders at least equal to its
primary. So, if a standby created enforces the value of
max_wal_senders to be lower than that, then this could break failovers.
Normally this should not be an issue though, as any settings of a
standby are inherited from its primary as postgresql.conf gets normally
copied as part of a base backup, so parameters would be consistent.
Author: Alexander Kukushkin
Reviewed-by: Kyotaro Horiguchi, Petr Jelínek, Masahiko Sawada, Oleksii
Kliukin
Discussion: https://postgr.es/m/CAFh8B=nBzHQeYAu0b8fjK-AF1X4+_p6GRtwG+cCgs6Vci2uRuQ@mail.gmail.com
2019-02-12 02:07:56 +01:00
<entry><literal>ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16) * 17</literal> plus room for other applications</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SEMMSL</varname></entry>
<entry>Maximum number of semaphores per set</entry>
<entry>at least 17</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SEMMAP</varname></entry>
<entry>Number of entries in semaphore map</entry>
<entry>see text</entry>
2000-07-22 16:49:01 +02:00
</row>
<row>
2017-10-09 03:44:17 +02:00
<entry><varname>SEMVMX</varname></entry>
<entry>Maximum value of semaphore</entry>
<entry>at least 1000 (The default is often 32767; do not change unless necessary)</entry>
2000-07-22 16:49:01 +02:00
</row>
</tbody>
</tgroup>
</table>
<para>
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> requires a few bytes of System V shared memory
2012-07-04 21:56:12 +02:00
(typically 48 bytes, on 64-bit platforms) for each copy of the server.
On most modern operating systems, this amount can easily be allocated.
2019-02-03 09:55:39 +01:00
However, if you are running many copies of the server or you explicitly
configure the server to use large amounts of System V shared memory (see
<xref linkend="guc-shared-memory-type"/> and <xref
linkend="guc-dynamic-shared-memory-type"/>), it may be necessary to
2017-10-09 03:44:17 +02:00
increase <varname>SHMALL</varname>, which is the total amount of System V shared
memory system-wide. Note that <varname>SHMALL</varname> is measured in pages
2012-07-04 21:56:12 +02:00
rather than bytes on many systems.
2000-07-22 16:49:01 +02:00
</para>
<para>
Less likely to cause problems is the minimum size for shared
2017-10-09 03:44:17 +02:00
memory segments (<varname>SHMMIN</varname>), which should be at most
approximately 32 bytes for <productname>PostgreSQL</productname> (it is
2000-12-17 12:22:00 +01:00
usually just 1). The maximum number of segments system-wide
2017-10-09 03:44:17 +02:00
(<varname>SHMMNI</varname>) or per-process (<varname>SHMSEG</varname>) are unlikely
2004-12-27 00:06:56 +01:00
to cause a problem unless your system has them set to zero.
2000-07-22 16:49:01 +02:00
</para>
<para>
2016-10-10 21:11:33 +02:00
When using System V semaphores,
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> uses one semaphore per allowed connection
2017-11-23 15:39:47 +01:00
(<xref linkend="guc-max-connections"/>), allowed autovacuum worker process
(<xref linkend="guc-autovacuum-max-workers"/>) and allowed background
process (<xref linkend="guc-max-worker-processes"/>), in sets of 16.
2007-11-01 20:06:01 +01:00
Each such set will
2005-08-30 02:58:48 +02:00
also contain a 17th semaphore which contains a <quote>magic
2001-09-07 02:46:42 +02:00
number</quote>, to detect collision with semaphore sets used by
2000-12-17 12:22:00 +01:00
other applications. The maximum number of semaphores in the system
2017-10-09 03:44:17 +02:00
is set by <varname>SEMMNS</varname>, which consequently must be at least
as high as <varname>max_connections</varname> plus
Move max_wal_senders out of max_connections for connection slot handling
Since its introduction, max_wal_senders is counted as part of
max_connections when it comes to define how many connection slots can be
used for replication connections with a WAL sender context. This can
lead to confusion for some users, as it could be possible to block a
base backup or replication from happening because other backend sessions
are already taken for other purposes by an application, and
superuser-only connection slots are not a correct solution to handle
that case.
This commit makes max_wal_senders independent of max_connections for its
handling of PGPROC entries in ProcGlobal, meaning that connection slots
for WAL senders are handled using their own free queue, like autovacuum
workers and bgworkers.
One compatibility issue that this change creates is that a standby now
requires to have a value of max_wal_senders at least equal to its
primary. So, if a standby created enforces the value of
max_wal_senders to be lower than that, then this could break failovers.
Normally this should not be an issue though, as any settings of a
standby are inherited from its primary as postgresql.conf gets normally
copied as part of a base backup, so parameters would be consistent.
Author: Alexander Kukushkin
Reviewed-by: Kyotaro Horiguchi, Petr Jelínek, Masahiko Sawada, Oleksii
Kliukin
Discussion: https://postgr.es/m/CAFh8B=nBzHQeYAu0b8fjK-AF1X4+_p6GRtwG+cCgs6Vci2uRuQ@mail.gmail.com
2019-02-12 02:07:56 +01:00
<varname>autovacuum_max_workers</varname> plus <varname>max_wal_senders</varname>,
plus <varname>max_worker_processes</varname>, plus one extra for each 16
2007-11-01 20:06:01 +01:00
allowed connections plus workers (see the formula in <xref
2017-11-23 15:39:47 +01:00
linkend="sysvipc-parameters"/>). The parameter <varname>SEMMNI</varname>
2000-12-17 12:22:00 +01:00
determines the limit on the number of semaphore sets that can
exist on the system at one time. Hence this parameter must be at
Move max_wal_senders out of max_connections for connection slot handling
Since its introduction, max_wal_senders is counted as part of
max_connections when it comes to define how many connection slots can be
used for replication connections with a WAL sender context. This can
lead to confusion for some users, as it could be possible to block a
base backup or replication from happening because other backend sessions
are already taken for other purposes by an application, and
superuser-only connection slots are not a correct solution to handle
that case.
This commit makes max_wal_senders independent of max_connections for its
handling of PGPROC entries in ProcGlobal, meaning that connection slots
for WAL senders are handled using their own free queue, like autovacuum
workers and bgworkers.
One compatibility issue that this change creates is that a standby now
requires to have a value of max_wal_senders at least equal to its
primary. So, if a standby created enforces the value of
max_wal_senders to be lower than that, then this could break failovers.
Normally this should not be an issue though, as any settings of a
standby are inherited from its primary as postgresql.conf gets normally
copied as part of a base backup, so parameters would be consistent.
Author: Alexander Kukushkin
Reviewed-by: Kyotaro Horiguchi, Petr Jelínek, Masahiko Sawada, Oleksii
Kliukin
Discussion: https://postgr.es/m/CAFh8B=nBzHQeYAu0b8fjK-AF1X4+_p6GRtwG+cCgs6Vci2uRuQ@mail.gmail.com
2019-02-12 02:07:56 +01:00
least <literal>ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16)</literal>.
2007-11-01 20:06:01 +01:00
Lowering the number
2000-12-17 12:22:00 +01:00
of allowed connections is a temporary workaround for failures,
2010-02-03 18:25:06 +01:00
which are usually confusingly worded <quote>No space
2017-10-09 03:44:17 +02:00
left on device</quote>, from the function <function>semget</function>.
2000-07-22 16:49:01 +02:00
</para>
<para>
2002-03-09 06:11:38 +01:00
In some cases it might also be necessary to increase
2017-10-09 03:44:17 +02:00
<varname>SEMMAP</varname> to be at least on the order of
2019-01-08 18:03:53 +01:00
<varname>SEMMNS</varname>. If the system has this parameter
(many do not), it defines the size of the semaphore
2002-03-09 06:11:38 +01:00
resource map, in which each contiguous block of available semaphores
needs an entry. When a semaphore set is freed it is either added to
an existing entry that is adjacent to the freed block or it is
registered under a new map entry. If the map is full, the freed
semaphores get lost (until reboot). Fragmentation of the semaphore
space could over time lead to fewer available semaphores than there
should be.
2000-07-22 16:49:01 +02:00
</para>
<para>
2017-10-09 03:44:17 +02:00
Various other settings related to <quote>semaphore undo</quote>, such as
<varname>SEMMNU</varname> and <varname>SEMUME</varname>, do not affect
<productname>PostgreSQL</productname>.
2000-07-22 16:49:01 +02:00
</para>
<para>
2016-10-10 21:11:33 +02:00
When using POSIX semaphores, the number of semaphores needed is the
same as for System V, that is one semaphore per allowed connection
2017-11-23 15:39:47 +01:00
(<xref linkend="guc-max-connections"/>), allowed autovacuum worker process
(<xref linkend="guc-autovacuum-max-workers"/>) and allowed background
process (<xref linkend="guc-max-worker-processes"/>).
2016-10-10 21:11:33 +02:00
On the platforms where this option is preferred, there is no specific
kernel limit on the number of POSIX semaphores.
2000-07-22 16:49:01 +02:00
</para>
2005-08-30 02:58:48 +02:00
2000-07-22 16:49:01 +02:00
<variablelist>
2007-03-06 10:59:22 +01:00
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">AIX</systemitem>
<indexterm><primary>AIX</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2007-03-06 10:59:22 +01:00
<listitem>
<para>
2020-06-07 11:36:43 +02:00
It should not be necessary to do
2007-03-06 10:59:22 +01:00
any special configuration for such parameters as
<varname>SHMMAX</varname>, as it appears this is configured to
allow all memory to be used as shared memory. That is the
sort of configuration commonly used for other databases such
as <application>DB/2</application>.</para>
2007-11-01 20:06:01 +01:00
<para> It might, however, be necessary to modify the global
2007-03-06 10:59:22 +01:00
<command>ulimit</command> information in
<filename>/etc/security/limits</filename>, as the default hard
limits for file sizes (<varname>fsize</varname>) and numbers of
files (<varname>nofiles</varname>) might be too low.
</para>
</listitem>
2010-05-27 01:49:19 +02:00
</varlistentry>
2000-07-22 16:49:01 +02:00
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">FreeBSD</systemitem>
<indexterm><primary>FreeBSD</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2005-02-10 06:14:58 +01:00
<listitem>
2020-06-07 11:36:43 +02:00
<para>
The default shared memory settings are usually good enough, unless
you have set <literal>shared_memory_type</literal> to <literal>sysv</literal>.
System V semaphores are not used on this platform.
</para>
2005-02-10 06:14:58 +01:00
<para>
2019-01-08 18:03:53 +01:00
The default IPC settings can be changed using
2012-07-04 21:56:12 +02:00
the <command>sysctl</command> or
2005-02-10 06:14:58 +01:00
<command>loader</command> interfaces. The following
parameters can be set using <command>sysctl</command>:
<screen>
2012-08-27 05:21:45 +02:00
<prompt>#</prompt> <userinput>sysctl kern.ipc.shmall=32768</userinput>
<prompt>#</prompt> <userinput>sysctl kern.ipc.shmmax=134217728</userinput>
2005-02-10 06:14:58 +01:00
</screen>
2014-08-14 22:05:46 +02:00
To make these settings persist over reboots, modify
2005-02-10 06:14:58 +01:00
<filename>/etc/sysctl.conf</filename>.
</para>
<para>
2020-06-07 11:36:43 +02:00
If you have set <literal>shared_memory_type</literal> to
<literal>sysv</literal>, you might also want to configure your kernel
to lock System V shared memory into RAM and prevent it from being paged
out to swap. This can be accomplished using the <command>sysctl</command>
2005-02-10 06:14:58 +01:00
setting <literal>kern.ipc.shm_use_phys</literal>.
</para>
2006-04-11 21:26:42 +02:00
<para>
2020-06-07 11:36:43 +02:00
If running in a FreeBSD jail, you should set its
<literal>sysvshm</literal> parameter to <literal>new</literal>, so that
it has its own separate System V shared memory namespace.
(Before FreeBSD 11.0, it was necessary to enable shared access to
the host's IPC namespace from jails, and take measures to avoid
collisions.)
2006-04-11 21:26:42 +02:00
</para>
2005-02-10 06:14:58 +01:00
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">NetBSD</systemitem>
<indexterm><primary>NetBSD</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2011-12-01 02:55:00 +01:00
<listitem>
<para>
2020-06-07 11:36:43 +02:00
The default shared memory settings are usually good enough, unless
you have set <literal>shared_memory_type</literal> to <literal>sysv</literal>.
You will usually want to increase <literal>kern.ipc.semmni</literal>
and <literal>kern.ipc.semmns</literal>,
as <systemitem class="osname">NetBSD</systemitem>'s default settings
for these are uncomfortably small.
</para>
<para>
2011-12-01 02:55:00 +01:00
IPC parameters can be adjusted using <command>sysctl</command>,
for example:
<screen>
2019-01-08 18:03:53 +01:00
<prompt>#</prompt> <userinput>sysctl -w kern.ipc.semmni=100</userinput>
2011-12-01 02:55:00 +01:00
</screen>
2019-01-08 18:03:53 +01:00
To make these settings persist over reboots, modify
2011-12-01 02:55:00 +01:00
<filename>/etc/sysctl.conf</filename>.
</para>
2019-01-08 18:03:53 +01:00
<para>
2020-06-07 11:36:43 +02:00
If you have set <literal>shared_memory_type</literal> to
<literal>sysv</literal>, you might also want to configure your kernel
to lock System V shared memory into RAM and prevent it from being paged
out to swap. This can be accomplished using the <command>sysctl</command>
2011-12-01 02:55:00 +01:00
setting <literal>kern.ipc.shm_use_phys</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">OpenBSD</systemitem>
<indexterm><primary>OpenBSD</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2000-07-22 16:49:01 +02:00
<listitem>
<para>
2020-06-07 11:36:43 +02:00
The default shared memory settings are usually good enough, unless
you have set <literal>shared_memory_type</literal> to <literal>sysv</literal>.
2019-01-08 18:03:53 +01:00
You will usually want to
increase <literal>kern.seminfo.semmni</literal>
and <literal>kern.seminfo.semmns</literal>,
as <systemitem class="osname">OpenBSD</systemitem>'s default settings
for these are uncomfortably small.
</para>
<para>
2020-06-07 11:36:43 +02:00
IPC parameters can be adjusted using <command>sysctl</command>,
for example:
<screen>
<prompt>#</prompt> <userinput>sysctl kern.seminfo.semmni=100</userinput>
</screen>
To make these settings persist over reboots, modify
<filename>/etc/sysctl.conf</filename>.
2000-07-22 16:49:01 +02:00
</para>
2005-02-10 06:14:58 +01:00
2000-07-22 16:49:01 +02:00
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">HP-UX</systemitem>
<indexterm><primary>HP-UX</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2000-07-22 16:49:01 +02:00
<listitem>
<para>
The default settings tend to suffice for normal installations.
</para>
<para>
2017-10-09 03:44:17 +02:00
<acronym>IPC</acronym> parameters can be set in the <application>System
Administration Manager</application> (<acronym>SAM</acronym>) under
2000-07-22 16:49:01 +02:00
<menuchoice><guimenu>Kernel
2017-10-09 03:44:17 +02:00
Configuration</guimenu><guimenuitem>Configurable Parameters</guimenuitem></menuchoice>. Choose
<guibutton>Create A New Kernel</guibutton> when you're done.
2000-07-22 16:49:01 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">Linux</systemitem>
<indexterm><primary>Linux</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2000-07-22 16:49:01 +02:00
<listitem>
<para>
2020-06-07 11:36:43 +02:00
The default shared memory settings are usually good enough, unless
you have set <literal>shared_memory_type</literal> to <literal>sysv</literal>,
and even then only on older kernel versions that shipped with low defaults.
System V semaphores are not used on this platform.
2010-03-13 12:00:19 +01:00
</para>
<para>
The shared memory size settings can be changed via the
2016-08-05 20:35:09 +02:00
<command>sysctl</command> interface. For example, to allow 16 GB:
2000-12-17 12:22:00 +01:00
<screen>
2010-03-13 12:00:19 +01:00
<prompt>$</prompt> <userinput>sysctl -w kernel.shmmax=17179869184</userinput>
<prompt>$</prompt> <userinput>sysctl -w kernel.shmall=4194304</userinput>
2000-12-17 12:22:00 +01:00
</screen>
2020-06-07 11:36:43 +02:00
To make these settings persist over reboots, see
<filename>/etc/sysctl.conf</filename>.
2010-03-13 12:00:19 +01:00
</para>
2000-07-22 16:49:01 +02:00
</listitem>
</varlistentry>
2002-11-06 00:16:56 +01:00
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem class="osname">macOS</systemitem>
<indexterm><primary>macOS</primary><secondary>IPC configuration</secondary></indexterm>
2014-05-07 03:28:58 +02:00
</term>
2002-11-06 00:16:56 +01:00
<listitem>
2020-06-07 11:36:43 +02:00
<para>
The default shared memory and semaphore settings are usually good enough, unless
you have set <literal>shared_memory_type</literal> to <literal>sysv</literal>.
</para>
2002-11-06 00:16:56 +01:00
<para>
Refer to OS X as "macOS", except for the port name which is still "darwin".
We weren't terribly consistent about whether to call Apple's OS "OS X"
or "Mac OS X", and the former is probably confusing to people who aren't
Apple users. Now that Apple has rebranded it "macOS", follow their lead
to establish a consistent naming pattern. Also, avoid the use of the
ancient project name "Darwin", except as the port code name which does not
seem desirable to change. (In short, this patch touches documentation and
comments, but no actual code.)
I didn't touch contrib/start-scripts/osx/, either. I suspect those are
obsolete and due for a rewrite, anyway.
I dithered about whether to apply this edit to old release notes, but
those were responsible for quite a lot of the inconsistencies, so I ended
up changing them too. Anyway, Apple's being ahistorical about this,
so why shouldn't we be?
2016-09-25 21:40:57 +02:00
The recommended method for configuring shared memory in macOS
2017-10-09 03:44:17 +02:00
is to create a file named <filename>/etc/sysctl.conf</filename>,
2007-02-01 01:28:19 +01:00
containing variable assignments such as:
2006-03-05 04:50:44 +01:00
<programlisting>
kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=1024
</programlisting>
Refer to OS X as "macOS", except for the port name which is still "darwin".
We weren't terribly consistent about whether to call Apple's OS "OS X"
or "Mac OS X", and the former is probably confusing to people who aren't
Apple users. Now that Apple has rebranded it "macOS", follow their lead
to establish a consistent naming pattern. Also, avoid the use of the
ancient project name "Darwin", except as the port code name which does not
seem desirable to change. (In short, this patch touches documentation and
comments, but no actual code.)
I didn't touch contrib/start-scripts/osx/, either. I suspect those are
obsolete and due for a rewrite, anyway.
I dithered about whether to apply this edit to old release notes, but
those were responsible for quite a lot of the inconsistencies, so I ended
up changing them too. Anyway, Apple's being ahistorical about this,
so why shouldn't we be?
2016-09-25 21:40:57 +02:00
Note that in some macOS versions,
2017-10-09 03:44:17 +02:00
<emphasis>all five</emphasis> shared-memory parameters must be set in
<filename>/etc/sysctl.conf</filename>, else the values will be ignored.
2006-03-05 04:50:44 +01:00
</para>
<para>
2020-06-07 11:36:43 +02:00
<varname>SHMMAX</varname> can only be set to a multiple of 4096.
2002-11-06 00:16:56 +01:00
</para>
2005-10-16 23:22:12 +02:00
2006-03-02 21:30:21 +01:00
<para>
2017-10-09 03:44:17 +02:00
<varname>SHMALL</varname> is measured in 4 kB pages on this platform.
2006-03-02 21:30:21 +01:00
</para>
2005-10-16 23:22:12 +02:00
2004-12-27 00:06:56 +01:00
<para>
2020-06-07 11:36:43 +02:00
It is possible to change all but <varname>SHMMNI</varname> on the fly, using
2017-10-09 03:44:17 +02:00
<application>sysctl</application>. But it's still best to set up your preferred
values via <filename>/etc/sysctl.conf</filename>, so that the values will be
2010-05-01 00:24:50 +02:00
kept across reboots.
</para>
2011-03-11 13:53:34 +01:00
</listitem>
</varlistentry>
<varlistentry>
2020-06-07 11:36:43 +02:00
<term><systemitem class="osname">Solaris</systemitem></term>
<term><systemitem class="osname">illumos</systemitem></term>
2011-03-11 13:53:34 +01:00
<listitem>
<para>
2020-06-07 11:36:43 +02:00
The default shared memory and semaphore settings are usually good enough for most
<productname>PostgreSQL</productname> applications. Solaris defaults
2017-10-09 03:44:17 +02:00
to a <varname>SHMMAX</varname> of one-quarter of system <acronym>RAM</acronym>.
2012-07-04 21:56:12 +02:00
To further adjust this setting, use a project setting associated
2017-10-09 03:44:17 +02:00
with the <literal>postgres</literal> user. For example, run the
following as <literal>root</literal>:
2011-03-11 13:53:34 +01:00
<programlisting>
projadd -c "PostgreSQL DB User" -K "project.max-shm-memory=(privileged,8GB,deny)" -U postgres -G postgres user.postgres
</programlisting>
</para>
<para>
2017-10-09 03:44:17 +02:00
This command adds the <literal>user.postgres</literal> project and
sets the shared memory maximum for the <literal>postgres</literal>
2011-03-11 13:53:34 +01:00
user to 8GB, and takes effect the next time that user logs
2017-10-09 03:44:17 +02:00
in, or when you restart <productname>PostgreSQL</productname> (not reload).
The above assumes that <productname>PostgreSQL</productname> is run by
the <literal>postgres</literal> user in the <literal>postgres</literal>
2011-03-11 13:53:34 +01:00
group. No server reboot is required.
</para>
<para>
Other recommended kernel setting changes for database servers which will
have a large number of connections are:
<programlisting>
project.max-shm-ids=(priv,32768,deny)
project.max-sem-ids=(priv,4096,deny)
project.max-msg-ids=(priv,4096,deny)
</programlisting>
2000-07-22 16:49:01 +02:00
</para>
<para>
2017-10-09 03:44:17 +02:00
Additionally, if you are running <productname>PostgreSQL</productname>
2011-03-11 13:53:34 +01:00
inside a zone, you may need to raise the zone resource usage
limits as well. See "Chapter2: Projects and Tasks" in the
2017-10-09 03:44:17 +02:00
<citetitle>System Administrator's Guide</citetitle> for more
information on <literal>projects</literal> and <command>prctl</command>.
2000-07-22 16:49:01 +02:00
</para>
</listitem>
</varlistentry>
</variablelist>
2005-08-30 02:58:48 +02:00
2000-07-22 16:49:01 +02:00
</sect2>
2017-02-15 16:44:07 +01:00
<sect2 id="systemd-removeipc">
<title>systemd RemoveIPC</title>
<indexterm>
<primary>systemd</primary>
<secondary>RemoveIPC</secondary>
</indexterm>
<para>
If <productname>systemd</productname> is in use, some care must be taken
2020-06-08 03:20:46 +02:00
that IPC resources (including shared memory) are not prematurely
2017-02-15 16:44:07 +01:00
removed by the operating system. This is especially of concern when
installing PostgreSQL from source. Users of distribution packages of
PostgreSQL are less likely to be affected, as
the <literal>postgres</literal> user is then normally created as a system
user.
</para>
<para>
The setting <literal>RemoveIPC</literal>
in <filename>logind.conf</filename> controls whether IPC objects are
removed when a user fully logs out. System users are exempt. This
setting defaults to on in stock <productname>systemd</productname>, but
some operating system distributions default it to off.
</para>
<para>
2020-06-08 03:20:46 +02:00
A typical observed effect when this setting is on is that shared memory
objects used for parallel query execution are removed at apparently random
times, leading to errors and warnings while attempting to open and remove
them, like
2017-02-15 16:44:07 +01:00
<screen>
2020-06-08 03:20:46 +02:00
WARNING: could not remove shared memory segment "/PostgreSQL.1450751626": No such file or directory
2017-02-15 16:44:07 +01:00
</screen>
Different types of IPC objects (shared memory vs. semaphores, System V
vs. POSIX) are treated slightly differently
by <productname>systemd</productname>, so one might observe that some IPC
resources are not removed in the same way as others. But it is not
advisable to rely on these subtle differences.
</para>
<para>
A <quote>user logging out</quote> might happen as part of a maintenance
job or manually when an administrator logs in as
the <literal>postgres</literal> user or something similar, so it is hard
to prevent in general.
</para>
<para>
What is a <quote>system user</quote> is determined
at <productname>systemd</productname> compile time from
the <symbol>SYS_UID_MAX</symbol> setting
in <filename>/etc/login.defs</filename>.
</para>
<para>
Packaging and deployment scripts should be careful to create
the <literal>postgres</literal> user as a system user by
using <literal>useradd -r</literal>, <literal>adduser --system</literal>,
or equivalent.
</para>
<para>
Alternatively, if the user account was created incorrectly or cannot be
changed, it is recommended to set
<programlisting>
RemoveIPC=no
</programlisting>
in <filename>/etc/systemd/logind.conf</filename> or another appropriate
configuration file.
</para>
<caution>
<para>
At least one of these two things has to be ensured, or the PostgreSQL
server will be very unreliable.
</para>
</caution>
</sect2>
2001-01-08 22:01:54 +01:00
<sect2>
<title>Resource Limits</title>
<para>
2002-03-09 06:11:38 +01:00
Unix-like operating systems enforce various kinds of resource limits
that might interfere with the operation of your
<productname>PostgreSQL</productname> server. Of particular
importance are limits on the number of processes per user, the
number of open files per process, and the amount of memory available
to each process. Each of these have a <quote>hard</quote> and a
<quote>soft</quote> limit. The soft limit is what actually counts
but it can be changed by the user up to the hard limit. The hard
limit can only be changed by the root user. The system call
<function>setrlimit</function> is responsible for setting these
parameters. The shell's built-in command <command>ulimit</command>
2017-10-09 03:44:17 +02:00
(Bourne shells) or <command>limit</command> (<application>csh</application>) is
2002-03-09 06:11:38 +01:00
used to control the resource limits from the command line. On
BSD-derived systems the file <filename>/etc/login.conf</filename>
2003-03-24 15:32:51 +01:00
controls the various resource limits set during login. See the
operating system documentation for details. The relevant
2001-01-08 22:01:54 +01:00
parameters are <varname>maxproc</varname>,
2002-03-09 06:11:38 +01:00
<varname>openfiles</varname>, and <varname>datasize</varname>. For
example:
2001-01-08 22:01:54 +01:00
<programlisting>
default:\
...
:datasize-cur=256M:\
:maxproc-cur=256:\
:openfiles-cur=256:\
...
</programlisting>
(<literal>-cur</literal> is the soft limit. Append
<literal>-max</literal> to set the hard limit.)
</para>
<para>
2002-03-09 06:11:38 +01:00
Kernels can also have system-wide limits on some resources.
2001-02-09 21:38:15 +01:00
<itemizedlist>
<listitem>
<para>
2022-08-23 15:41:37 +02:00
On <productname>Linux</productname> the kernel parameter
<varname>fs.file-max</varname> determines the maximum number of open
files that the kernel will support. It can be changed with
<literal>sysctl -w fs.file-max=<replaceable>N</replaceable></literal>.
To make the setting persist across reboots, add an assignment
in <filename>/etc/sysctl.conf</filename>.
2001-01-08 22:01:54 +01:00
The maximum limit of files per process is fixed at the time the
kernel is compiled; see
<filename>/usr/src/linux/Documentation/proc.txt</filename> for
more information.
2001-02-09 21:38:15 +01:00
</para>
</listitem>
</itemizedlist>
2001-01-08 22:01:54 +01:00
</para>
<para>
2001-11-21 07:09:45 +01:00
The <productname>PostgreSQL</productname> server uses one process
2001-01-08 22:01:54 +01:00
per connection so you should provide for at least as many processes
as allowed connections, in addition to what you need for the rest
of your system. This is usually not a problem but if you run
several servers on one machine things might get tight.
</para>
<para>
The factory default limit on open files is often set to
<quote>socially friendly</quote> values that allow many users to
coexist on a machine without using an inappropriate fraction of
the system resources. If you run many servers on a machine this
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
is perhaps what you want, but on dedicated servers you might want to
2001-01-08 22:01:54 +01:00
raise this limit.
</para>
2001-12-27 22:37:34 +01:00
<para>
On the other side of the coin, some systems allow individual
2002-03-09 06:11:38 +01:00
processes to open large numbers of files; if more than a few
processes do so then the system-wide limit can easily be exceeded.
2004-03-09 17:57:47 +01:00
If you find this happening, and you do not want to alter the
2017-10-09 03:44:17 +02:00
system-wide limit, you can set <productname>PostgreSQL</productname>'s <xref
2017-11-23 15:39:47 +01:00
linkend="guc-max-files-per-process"/> configuration parameter to
2002-03-09 06:11:38 +01:00
limit the consumption of open files.
2001-12-27 22:37:34 +01:00
</para>
2022-08-23 15:55:37 +02:00
<para>
Another kernel limit that may be of concern when supporting large
numbers of client connections is the maximum socket connection queue
length. If more than that many connection requests arrive within a very
short period, some may get rejected before the postmaster can service
the requests, with those clients receiving unhelpful connection failure
errors such as <quote>Resource temporarily unavailable</quote> or
<quote>Connection refused</quote>. The default queue length limit is 128
on many platforms. To raise it, adjust the appropriate kernel parameter
via <application>sysctl</application>, then restart the postmaster.
The parameter is variously named <varname>net.core.somaxconn</varname>
on Linux, <varname>kern.ipc.soacceptqueue</varname> on newer FreeBSD,
and <varname>kern.ipc.somaxconn</varname> on macOS and other BSD
variants.
</para>
2001-01-08 22:01:54 +01:00
</sect2>
2000-07-22 16:49:01 +02:00
2010-03-21 01:43:40 +01:00
<sect2 id="linux-memory-overcommit">
2003-11-04 10:55:39 +01:00
<title>Linux Memory Overcommit</title>
2016-01-01 04:03:13 +01:00
<indexterm>
<primary>memory overcommit</primary>
</indexterm>
<indexterm>
<primary>OOM</primary>
</indexterm>
<indexterm>
<primary>overcommit</primary>
</indexterm>
2003-11-04 10:55:39 +01:00
<para>
2020-06-07 11:36:43 +02:00
The default virtual memory behavior on Linux is not
2003-11-16 04:38:34 +01:00
optimal for <productname>PostgreSQL</productname>. Because of the
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
way that the kernel implements memory overcommit, the kernel might
2012-06-13 21:34:57 +02:00
terminate the <productname>PostgreSQL</productname> postmaster (the
2020-06-15 19:19:32 +02:00
supervisor server process) if the memory demands of either
2012-08-30 20:14:22 +02:00
<productname>PostgreSQL</productname> or another process cause the
system to run out of virtual memory.
2003-11-04 10:55:39 +01:00
</para>
<para>
2003-11-16 04:38:34 +01:00
If this happens, you will see a kernel message that looks like
this (consult your system documentation and configuration on where
to look for such a message):
2003-11-04 10:55:39 +01:00
<programlisting>
2010-01-11 19:39:32 +01:00
Out of Memory: Killed process 12345 (postgres).
2003-11-04 10:55:39 +01:00
</programlisting>
2006-06-18 17:38:37 +02:00
This indicates that the <filename>postgres</filename> process
2003-11-16 18:01:50 +01:00
has been terminated due to memory pressure.
Although existing database connections will continue to function
normally, no new connections will be accepted. To recover,
2003-11-16 04:38:34 +01:00
<productname>PostgreSQL</productname> will need to be restarted.
2003-11-04 10:55:39 +01:00
</para>
<para>
2003-11-16 18:01:50 +01:00
One way to avoid this problem is to run
2007-03-06 10:54:23 +01:00
<productname>PostgreSQL</productname> on a machine where you can
be sure that other processes will not run the machine out of
memory. If memory is tight, increasing the swap space of the
2010-01-11 19:39:32 +01:00
operating system can help avoid the problem, because the
out-of-memory (OOM) killer is invoked only when physical memory and
2007-03-06 10:54:23 +01:00
swap space are exhausted.
2003-11-16 18:01:50 +01:00
</para>
2012-08-30 20:14:22 +02:00
<para>
If <productname>PostgreSQL</productname> itself is the cause of the
system running out of memory, you can avoid the problem by changing
your configuration. In some cases, it may help to lower memory-related
configuration parameters, particularly
2020-07-29 23:14:58 +02:00
<link linkend="guc-shared-buffers"><varname>shared_buffers</varname></link>,
<link linkend="guc-work-mem"><varname>work_mem</varname></link>, and
<link linkend="guc-hash-mem-multiplier"><varname>hash_mem_multiplier</varname></link>.
In other cases, the problem may be caused by allowing too many
connections to the database server itself. In many cases, it may
be better to reduce
2017-10-09 03:44:17 +02:00
<link linkend="guc-max-connections"><varname>max_connections</varname></link>
2012-08-30 20:14:22 +02:00
and instead make use of external connection-pooling software.
</para>
2003-11-16 18:01:50 +01:00
<para>
2020-06-07 11:36:43 +02:00
It is possible to modify the
2017-10-09 03:44:17 +02:00
kernel's behavior so that it will not <quote>overcommit</quote> memory.
2007-12-22 06:13:03 +01:00
Although this setting will not prevent the <ulink
2018-07-16 10:44:06 +02:00
url="https://lwn.net/Articles/104179/">OOM killer</ulink> from being invoked
2007-12-22 06:13:03 +01:00
altogether, it will lower the chances significantly and will therefore
lead to more robust system behavior. This is done by selecting strict
overcommit mode via <command>sysctl</command>:
2003-11-04 10:55:39 +01:00
<programlisting>
sysctl -w vm.overcommit_memory=2
</programlisting>
2017-10-09 03:44:17 +02:00
or placing an equivalent entry in <filename>/etc/sysctl.conf</filename>.
2010-01-11 19:39:32 +01:00
You might also wish to modify the related setting
2017-10-09 03:44:17 +02:00
<varname>vm.overcommit_ratio</varname>. For details see the kernel documentation
2016-01-01 04:03:13 +01:00
file <ulink url="https://www.kernel.org/doc/Documentation/vm/overcommit-accounting"></ulink>.
2003-11-04 10:55:39 +01:00
</para>
2010-01-11 19:39:32 +01:00
<para>
Another approach, which can be used with or without altering
2017-10-09 03:44:17 +02:00
<varname>vm.overcommit_memory</varname>, is to set the process-specific
<firstterm>OOM score adjustment</firstterm> value for the postmaster process to
<literal>-1000</literal>, thereby guaranteeing it will not be targeted by the OOM
2012-06-13 21:34:57 +02:00
killer. The simplest way to do this is to execute
2010-01-11 19:39:32 +01:00
<programlisting>
2012-06-13 21:34:57 +02:00
echo -1000 > /proc/self/oom_score_adj
2010-01-11 19:39:32 +01:00
</programlisting>
in the postmaster's startup script just before invoking the postmaster.
Note that this action must be done as root, or it will have no effect;
so a root-owned startup script is the easiest place to do it. If you
2014-06-19 02:12:47 +02:00
do this, you should also set these environment variables in the startup
script before invoking the postmaster:
<programlisting>
export PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
export PG_OOM_ADJUST_VALUE=0
</programlisting>
These settings will cause postmaster child processes to run with the
normal OOM score adjustment of zero, so that the OOM killer can still
target them at need. You could use some other value for
2017-10-09 03:44:17 +02:00
<envar>PG_OOM_ADJUST_VALUE</envar> if you want the child processes to run
with some other OOM score adjustment. (<envar>PG_OOM_ADJUST_VALUE</envar>
2014-06-19 02:12:47 +02:00
can also be omitted, in which case it defaults to zero.) If you do not
2017-10-09 03:44:17 +02:00
set <envar>PG_OOM_ADJUST_FILE</envar>, the child processes will run with the
2014-06-19 02:12:47 +02:00
same OOM score adjustment as the postmaster, which is unwise since the
whole point is to ensure that the postmaster has a preferential setting.
2010-01-11 19:39:32 +01:00
</para>
2003-11-04 10:55:39 +01:00
</sect2>
2014-03-03 19:52:48 +01:00
<sect2 id="linux-huge-pages">
2016-10-22 20:04:51 +02:00
<title>Linux Huge Pages</title>
2014-03-03 19:52:48 +01:00
<para>
Using huge pages reduces overhead when using large contiguous chunks of
2016-10-22 20:04:51 +02:00
memory, as <productname>PostgreSQL</productname> does, particularly when
2017-11-23 15:39:47 +01:00
using large values of <xref linkend="guc-shared-buffers"/>. To use this
2014-03-03 19:52:48 +01:00
feature in <productname>PostgreSQL</productname> you need a kernel
with <varname>CONFIG_HUGETLBFS=y</varname> and
2020-07-17 04:33:00 +02:00
<varname>CONFIG_HUGETLB_PAGE=y</varname>. You will also have to configure
the operating system to provide enough huge pages of the desired size.
2021-09-21 03:31:58 +02:00
To determine the number of huge pages needed, use the
<command>postgres</command> command to see the value of
<xref linkend="guc-shared-memory-size-in-huge-pages"/>. Note that the
server must be shut down to view this runtime-computed parameter.
2020-07-17 04:33:00 +02:00
This might look like:
2014-03-03 19:52:48 +01:00
<programlisting>
Silence extra logging when using "postgres -C" on runtime-computed GUCs
Presently, the server may emit a variety of log messages when inspecting
a runtime-computed GUC, mostly in the shape of one LOG message with the
default configuration, related to the startup sequence launched as such
GUCs require a load of the control file and of external shared
libraries.
For example, the server will always emit a "database system is shut
down" LOG (unless the user has set log_min_messages higher than LOG),
which is an annoying behavior as "postgres -C" is expected to only emit
in its output the parameter value we are looking for. The parameter
value is sent to stdout, while the logs are sent to stderr so we could
recommend to use a redirection, but there was not much love for this
workaround either.
To avoid such extra log messages, per discussion, this change sets
log_min_messages to FATAL internally when -C is used on a
runtime-computed GUC (even if set to PANIC in postgresql.conf). At
FATAL, the user will still receive messages explaining why a GUC value
cannot be inspected, and will know if the command is attempted on a
server already running, something not supported yet for a
runtime-computed GUC.
Reported-by: Magnus Hagander, Bruce Momjian
Author: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/Yni6ZHkGotUU+RSf@paquier.xyz
2022-05-11 07:21:06 +02:00
$ <userinput>postgres -D $PGDATA -C shared_memory_size_in_huge_pages</userinput>
2021-09-21 03:31:58 +02:00
3170
2016-10-22 20:04:51 +02:00
$ <userinput>grep ^Hugepagesize /proc/meminfo</userinput>
Hugepagesize: 2048 kB
2020-07-17 04:33:00 +02:00
$ <userinput>ls /sys/kernel/mm/hugepages</userinput>
hugepages-1048576kB hugepages-2048kB
2014-03-03 19:52:48 +01:00
</programlisting>
2020-07-17 04:33:00 +02:00
In this example the default is 2MB, but you can also explicitly request
2021-09-21 03:31:58 +02:00
either 2MB or 1GB with <xref linkend="guc-huge-page-size"/> to adapt
the number of pages calculated by
<varname>shared_memory_size_in_huge_pages</varname>.
2020-07-17 04:33:00 +02:00
2021-09-21 03:31:58 +02:00
While we need at least <literal>3170</literal> huge pages in this example,
a larger setting would be appropriate if other programs on the machine
also need huge pages.
2020-07-17 04:33:00 +02:00
We can set this with:
<programlisting>
# <userinput>sysctl -w vm.nr_hugepages=3170</userinput>
</programlisting>
Don't forget to add this setting to <filename>/etc/sysctl.conf</filename>
so that it is reapplied after reboots. For non-default huge page sizes,
we can instead use:
2014-03-03 19:52:48 +01:00
<programlisting>
2020-07-17 04:33:00 +02:00
# <userinput>echo 3170 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages</userinput>
2014-03-03 19:52:48 +01:00
</programlisting>
2020-07-17 04:33:00 +02:00
It is also possible to provide these settings at boot time using
kernel parameters such as <literal>hugepagesz=2M hugepages=3170</literal>.
2016-10-22 20:04:51 +02:00
</para>
<para>
2014-03-03 19:52:48 +01:00
Sometimes the kernel is not able to allocate the desired number of huge
2020-07-17 04:33:00 +02:00
pages immediately due to fragmentation, so it might be necessary
to repeat the command or to reboot. (Immediately after a reboot, most of
the machine's memory should be available to convert into huge pages.)
To verify the huge page allocation situation for a given size, use:
2016-10-22 20:04:51 +02:00
<programlisting>
2020-07-17 04:33:00 +02:00
$ <userinput>cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages</userinput>
2016-10-22 20:04:51 +02:00
</programlisting>
2014-03-03 19:52:48 +01:00
</para>
2014-10-13 23:23:36 +02:00
<para>
2016-10-22 20:04:51 +02:00
It may also be necessary to give the database server's operating system
2014-10-13 23:23:36 +02:00
user permission to use huge pages by setting
2017-10-09 03:44:17 +02:00
<varname>vm.hugetlb_shm_group</varname> via <application>sysctl</application>, and/or
give permission to lock memory with <command>ulimit -l</command>.
2014-10-13 23:23:36 +02:00
</para>
2014-03-03 19:52:48 +01:00
<para>
The default behavior for huge pages in
2020-07-17 04:33:00 +02:00
<productname>PostgreSQL</productname> is to use them when possible, with
the system's default huge page size, and
to fall back to normal pages on failure. To enforce the use of huge
2017-11-23 15:39:47 +01:00
pages, you can set <xref linkend="guc-huge-pages"/>
2017-10-09 03:44:17 +02:00
to <literal>on</literal> in <filename>postgresql.conf</filename>.
Note that with this setting <productname>PostgreSQL</productname> will fail to
2016-10-22 20:04:51 +02:00
start if not enough huge pages are available.
2014-03-03 19:52:48 +01:00
</para>
<para>
For a detailed description of the <productname>Linux</productname> huge
pages feature have a look
2016-01-01 04:26:57 +01:00
at <ulink url="https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt"></ulink>.
2014-03-03 19:52:48 +01:00
</para>
</sect2>
2000-07-22 16:49:01 +02:00
</sect1>
2006-06-18 17:38:37 +02:00
<sect1 id="server-shutdown">
2002-11-11 21:14:04 +01:00
<title>Shutting Down the Server</title>
2000-06-18 23:24:54 +02:00
2006-06-18 17:38:37 +02:00
<indexterm zone="server-shutdown">
2017-10-09 03:44:17 +02:00
<primary>shutdown</primary>
2003-08-31 19:32:24 +02:00
</indexterm>
2000-06-18 23:24:54 +02:00
<para>
2020-09-03 17:45:26 +02:00
There are several ways to shut down the database server.
Under the hood, they all reduce to sending a signal to the supervisor
2006-06-18 17:38:37 +02:00
<command>postgres</command> process.
2020-09-03 17:45:26 +02:00
</para>
<para>
If you are using a pre-packaged version
of <productname>PostgreSQL</productname>, and you used its provisions
for starting the server, then you should also use its provisions for
stopping the server. Consult the package-level documentation for
details.
</para>
<para>
When managing the server directly, you can control the type of shutdown
by sending different signals to the <command>postgres</command>
process:
2003-09-13 00:17:24 +02:00
2000-06-18 23:24:54 +02:00
<variablelist>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem>SIGTERM</systemitem><indexterm><primary>SIGTERM</primary></indexterm></term>
2000-06-18 23:24:54 +02:00
<listitem>
<para>
2008-04-27 00:47:40 +02:00
This is the <firstterm>Smart Shutdown</firstterm> mode.
2003-03-24 15:32:51 +01:00
After receiving <systemitem>SIGTERM</systemitem>, the server
disallows new connections, but lets existing sessions end their
2008-04-27 00:47:40 +02:00
work normally. It shuts down only after all of the sessions terminate.
2022-04-06 20:41:03 +02:00
If the server is in recovery when a smart
2010-04-08 03:39:37 +02:00
shutdown is requested, recovery and streaming replication will be
stopped only after all regular sessions have terminated.
2000-06-18 23:24:54 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem>SIGINT</systemitem><indexterm><primary>SIGINT</primary></indexterm></term>
2000-06-18 23:24:54 +02:00
<listitem>
<para>
2008-04-27 00:47:40 +02:00
This is the <firstterm>Fast Shutdown</firstterm> mode.
2003-03-24 15:32:51 +01:00
The server disallows new connections and sends all existing
server processes <systemitem>SIGTERM</systemitem>, which will cause them
2002-03-09 06:11:38 +01:00
to abort their current transactions and exit promptly. It then
2010-02-03 18:25:06 +01:00
waits for all server processes to exit and finally shuts down.
2000-06-18 23:24:54 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
2017-10-09 03:44:17 +02:00
<term><systemitem>SIGQUIT</systemitem><indexterm><primary>SIGQUIT</primary></indexterm></term>
2000-06-18 23:24:54 +02:00
<listitem>
2003-03-24 15:32:51 +01:00
<para>
2008-04-27 00:47:40 +02:00
This is the <firstterm>Immediate Shutdown</firstterm> mode.
Send SIGKILL to children if they don't die quickly in immediate shutdown
On immediate shutdown, or during a restart-after-crash sequence,
postmaster used to send SIGQUIT (and then abandon ship if shutdown); but
this is not a good strategy if backends don't die because of that
signal. (This might happen, for example, if a backend gets tangled
trying to malloc() due to gettext(), as in an example illustrated by
MauMau.) This causes problems when later trying to restart the server,
because some processes are still attached to the shared memory segment.
Instead of just abandoning such backends to their fates, we now have
postmaster hang around for a little while longer, send a SIGKILL after
some reasonable waiting period, and then exit. This makes immediate
shutdown more reliable.
There is disagreement on whether it's best for postmaster to exit after
sending SIGKILL, or to stick around until all children have reported
death. If this controversy is resolved differently than what this patch
implements, it's an easy change to make.
Bug reported by MauMau in message 20DAEA8949EC4E2289C6E8E58560DEC0@maumau
MauMau and Álvaro Herrera
2013-06-28 23:20:53 +02:00
The server will send <systemitem>SIGQUIT</systemitem> to all child
2015-06-19 20:23:39 +02:00
processes and wait for them to terminate. If any do not terminate
within 5 seconds, they will be sent <systemitem>SIGKILL</systemitem>.
2020-06-15 19:19:32 +02:00
The supervisor server process exits as soon as all child processes have
2015-06-19 20:23:39 +02:00
exited, without doing normal database shutdown processing.
This will lead to recovery (by
2002-03-09 06:11:38 +01:00
replaying the WAL log) upon next start-up. This is recommended
only in emergencies.
2000-06-18 23:24:54 +02:00
</para>
</listitem>
</varlistentry>
</variablelist>
2003-03-24 15:32:51 +01:00
</para>
2000-06-18 23:24:54 +02:00
2004-12-27 00:06:56 +01:00
<para>
2017-11-23 15:39:47 +01:00
The <xref linkend="app-pg-ctl"/> program provides a convenient
2004-12-27 00:06:56 +01:00
interface for sending these signals to shut down the server.
2017-10-09 03:44:17 +02:00
Alternatively, you can send the signal directly using <command>kill</command>
2007-12-17 15:00:52 +01:00
on non-Windows systems.
2017-10-09 03:44:17 +02:00
The <acronym>PID</acronym> of the <command>postgres</command> process can be
2004-12-27 00:06:56 +01:00
found using the <command>ps</command> program, or from the file
<filename>postmaster.pid</filename> in the data directory. For
2000-06-18 23:24:54 +02:00
example, to do a fast shutdown:
<screen>
2001-11-28 21:49:10 +01:00
$ <userinput>kill -INT `head -1 /usr/local/pgsql/data/postmaster.pid`</userinput>
2000-06-18 23:24:54 +02:00
</screen>
</para>
2004-12-27 00:06:56 +01:00
2007-12-23 04:10:04 +01:00
<important>
<para>
2019-06-08 19:12:26 +02:00
It is best not to use <systemitem>SIGKILL</systemitem> to shut down the
server. Doing so will prevent the server from releasing shared memory and
semaphores. Furthermore, <systemitem>SIGKILL</systemitem> kills
the <command>postgres</command> process without letting it relay the
signal to its subprocesses, so it might be necessary to kill the
individual subprocesses by hand as well.
2007-12-23 04:10:04 +01:00
</para>
</important>
2008-04-17 22:56:41 +02:00
<para>
2008-04-27 00:47:40 +02:00
To terminate an individual session while allowing other sessions to
2017-10-09 03:44:17 +02:00
continue, use <function>pg_terminate_backend()</function> (see <xref
2017-11-23 15:39:47 +01:00
linkend="functions-admin-signal-table"/>) or send a
2017-10-09 03:44:17 +02:00
<systemitem>SIGTERM</systemitem> signal to the child process associated with
2008-04-17 22:56:41 +02:00
the session.
</para>
2000-06-18 23:24:54 +02:00
</sect1>
2011-01-31 18:32:03 +01:00
<sect1 id="upgrading">
2017-10-09 03:44:17 +02:00
<title>Upgrading a <productname>PostgreSQL</productname> Cluster</title>
2011-01-31 18:32:03 +01:00
<indexterm zone="upgrading">
<primary>upgrading</primary>
</indexterm>
<indexterm zone="upgrading">
<primary>version</primary>
<secondary>compatibility</secondary>
</indexterm>
<para>
This section discusses how to upgrade your database data from one
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> release to a newer one.
2011-01-31 18:32:03 +01:00
</para>
<para>
2016-08-15 19:49:49 +02:00
Current <productname>PostgreSQL</productname> version numbers consist of a
major and a minor version number. For example, in the version number 10.1,
the 10 is the major version number and the 1 is the minor version number,
meaning this would be the first minor release of the major release 10. For
releases before <productname>PostgreSQL</productname> version 10.0, version
numbers consist of three numbers, for example, 9.5.3. In those cases, the
major version consists of the first two digit groups of the version number,
e.g., 9.5, and the minor version is the third number, e.g., 3, meaning this
would be the third minor release of the major release 9.5.
</para>
<para>
Minor releases never change the internal storage format and are always
compatible with earlier and later minor releases of the same major version
number. For example, version 10.1 is compatible with version 10.0 and
version 10.6. Similarly, for example, 9.5.3 is compatible with 9.5.0,
9.5.1, and 9.5.6. To update between compatible versions, you simply
replace the executables while the server is down and restart the server.
The data directory remains unchanged — minor upgrades are that
simple.
2011-01-31 18:32:03 +01:00
</para>
<para>
2017-10-09 03:44:17 +02:00
For <emphasis>major</emphasis> releases of <productname>PostgreSQL</productname>, the
2011-01-31 18:32:03 +01:00
internal data storage format is subject to change, thus complicating
upgrades. The traditional method for moving data to a new major version
2022-07-21 20:55:23 +02:00
is to dump and restore the database, though this can be slow. A
2017-11-23 15:39:47 +01:00
faster method is <xref linkend="pgupgrade"/>. Replication methods are
2014-08-25 21:34:51 +02:00
also available, as discussed below.
2020-09-03 17:45:26 +02:00
(If you are using a pre-packaged version
of <productname>PostgreSQL</productname>, it may provide scripts to
assist with major version upgrades. Consult the package-level
documentation for details.)
2011-01-31 18:32:03 +01:00
</para>
<para>
New major versions also typically introduce some user-visible
incompatibilities, so application programming changes might be required.
All user-visible changes are listed in the release notes (<xref
2017-11-23 15:39:47 +01:00
linkend="release"/>); pay particular attention to the section
2021-04-02 03:17:24 +02:00
labeled "Migration". Though you can upgrade from one major version
to another without upgrading to intervening versions, you should read
the major release notes of all intervening versions.
2011-01-31 18:32:03 +01:00
</para>
<para>
Cautious users will want to test their client applications on the new
version before switching over fully; therefore, it's often a good idea to
set up concurrent installations of old and new versions. When
2017-10-09 03:44:17 +02:00
testing a <productname>PostgreSQL</productname> major upgrade, consider the
2011-01-31 18:32:03 +01:00
following categories of possible changes:
</para>
<variablelist>
<varlistentry>
<term>Administration</term>
<listitem>
<para>
The capabilities available for administrators to monitor and control
the server often change and improve in each major release.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>SQL</term>
<listitem>
<para>
Typically this includes new SQL command capabilities and not changes
in behavior, unless specifically mentioned in the release notes.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Library API</term>
<listitem>
<para>
2017-10-09 03:44:17 +02:00
Typically libraries like <application>libpq</application> only add new
2011-01-31 18:32:03 +01:00
functionality, again unless mentioned in the release notes.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>System Catalogs</term>
<listitem>
<para>
System catalog changes usually only affect database management tools.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Server C-language API</term>
<listitem>
<para>
This involves changes in the backend function API, which is written
in the C programming language. Such changes affect code that
references backend functions deep inside the server.
</para>
</listitem>
</varlistentry>
</variablelist>
2014-08-25 21:34:51 +02:00
<sect2 id="upgrading-via-pgdumpall">
2017-10-09 03:44:17 +02:00
<title>Upgrading Data via <application>pg_dumpall</application></title>
2011-01-31 18:32:03 +01:00
<para>
2014-08-25 21:34:51 +02:00
One upgrade method is to dump data from one major version of
2022-07-21 20:55:23 +02:00
<productname>PostgreSQL</productname> and restore it in another — to do
2017-10-09 03:44:17 +02:00
this, you must use a <emphasis>logical</emphasis> backup tool like
<application>pg_dumpall</application>; file system
2011-01-31 18:32:03 +01:00
level backup methods will not work. (There are checks in place that prevent
you from using a data directory with an incompatible version of
<productname>PostgreSQL</productname>, so no great harm can be done by
trying to start the wrong server version on a data directory.)
</para>
<para>
2017-10-09 03:44:17 +02:00
It is recommended that you use the <application>pg_dump</application> and
<application>pg_dumpall</application> programs from the <emphasis>newer</emphasis>
2014-08-25 21:34:51 +02:00
version of
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname>, to take advantage of enhancements
2011-01-31 18:32:03 +01:00
that might have been made in these programs. Current releases of the
2022-05-31 18:14:02 +02:00
dump programs can read data from any server version back to 9.2.
2011-01-31 18:32:03 +01:00
</para>
<para>
These instructions assume that your existing installation is under the
2017-10-09 03:44:17 +02:00
<filename>/usr/local/pgsql</filename> directory, and that the data area is in
<filename>/usr/local/pgsql/data</filename>. Substitute your paths
2011-01-31 18:32:03 +01:00
appropriately.
</para>
<procedure>
<step>
<para>
If making a backup, make sure that your database is not being updated.
This does not affect the integrity of the backup, but the changed
data would of course not be included. If necessary, edit the
2017-10-09 03:44:17 +02:00
permissions in the file <filename>/usr/local/pgsql/data/pg_hba.conf</filename>
2011-01-31 18:32:03 +01:00
(or equivalent) to disallow access from everyone except you.
2017-11-23 15:39:47 +01:00
See <xref linkend="client-authentication"/> for additional information on
2011-01-31 18:32:03 +01:00
access control.
</para>
<para>
<indexterm>
<primary>pg_dumpall</primary>
<secondary>use during upgrade</secondary>
</indexterm>
To back up your database installation, type:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>pg_dumpall > <replaceable>outputfile</replaceable></userinput>
2011-01-31 18:32:03 +01:00
</screen>
</para>
<para>
To make the backup, you can use the <application>pg_dumpall</application>
2014-08-25 21:34:51 +02:00
command from the version you are currently running; see <xref
2017-11-23 15:39:47 +01:00
linkend="backup-dump-all"/> for more details. For best
2011-01-31 18:32:03 +01:00
results, however, try to use the <application>pg_dumpall</application>
command from <productname>PostgreSQL</productname> &version;,
since this version contains bug fixes and improvements over older
versions. While this advice might seem idiosyncratic since you
haven't installed the new version yet, it is advisable to follow
it if you plan to install the new version in parallel with the
old version. In that case you can complete the installation
normally and transfer the data later. This will also decrease
the downtime.
</para>
</step>
<step>
<para>
Shut down the old server:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>pg_ctl stop</userinput>
2011-01-31 18:32:03 +01:00
</screen>
2017-10-09 03:44:17 +02:00
On systems that have <productname>PostgreSQL</productname> started at boot time,
2011-01-31 18:32:03 +01:00
there is probably a start-up file that will accomplish the same thing. For
2017-10-09 03:44:17 +02:00
example, on a <systemitem class="osname">Red Hat Linux</systemitem> system one
2011-01-31 18:32:03 +01:00
might find that this works:
<screen>
<userinput>/etc/rc.d/init.d/postgresql stop</userinput>
</screen>
2017-11-23 15:39:47 +01:00
See <xref linkend="runtime"/> for details about starting and
2011-05-19 00:14:45 +02:00
stopping the server.
2011-01-31 20:58:49 +01:00
</para>
2011-01-31 18:32:03 +01:00
</step>
<step>
<para>
If restoring from backup, rename or delete the old installation
2014-08-25 21:34:51 +02:00
directory if it is not version-specific. It is a good idea to
rename the directory, rather than
2011-01-31 18:32:03 +01:00
delete it, in case you have trouble and need to revert to it. Keep
in mind the directory might consume significant disk space. To rename
the directory, use a command like this:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>mv /usr/local/pgsql /usr/local/pgsql.old</userinput>
2011-01-31 18:32:03 +01:00
</screen>
(Be sure to move the directory as a single unit so relative paths
remain unchanged.)
</para>
</step>
<step>
<para>
Install the new version of <productname>PostgreSQL</productname> as
2017-11-23 15:39:47 +01:00
outlined in <xref linkend="install-procedure"/>.
2011-01-31 18:32:03 +01:00
</para>
</step>
<step>
<para>
Create a new database cluster if needed. Remember that you must
execute these commands while logged in to the special database user
account (which you already have if you are upgrading).
<programlisting>
2017-10-09 03:44:17 +02:00
<userinput>/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data</userinput>
2011-01-31 18:32:03 +01:00
</programlisting>
</para>
</step>
<step>
<para>
2017-10-09 03:44:17 +02:00
Restore your previous <filename>pg_hba.conf</filename> and any
<filename>postgresql.conf</filename> modifications.
2011-01-31 18:32:03 +01:00
</para>
</step>
<step>
<para>
Start the database server, again using the special database user
account:
<programlisting>
2017-10-09 03:44:17 +02:00
<userinput>/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data</userinput>
2011-01-31 18:32:03 +01:00
</programlisting>
</para>
</step>
<step>
<para>
Finally, restore your data from backup with:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>/usr/local/pgsql/bin/psql -d postgres -f <replaceable>outputfile</replaceable></userinput>
2011-01-31 18:32:03 +01:00
</screen>
2017-10-09 03:44:17 +02:00
using the <emphasis>new</emphasis> <application>psql</application>.
2011-01-31 18:32:03 +01:00
</para>
</step>
</procedure>
<para>
The least downtime can be achieved by installing the new server in
a different directory and running both the old and the new servers
in parallel, on different ports. Then you can use something like:
<programlisting>
pg_dumpall -p 5432 | psql -d postgres -p 5433
</programlisting>
to transfer your data.
</para>
</sect2>
2014-08-25 21:34:51 +02:00
<sect2 id="upgrading-via-pg-upgrade">
2017-10-09 03:44:17 +02:00
<title>Upgrading Data via <application>pg_upgrade</application></title>
2011-01-31 18:32:03 +01:00
<para>
2017-11-23 15:39:47 +01:00
The <xref linkend="pgupgrade"/> module allows an installation to
2017-10-09 03:44:17 +02:00
be migrated in-place from one major <productname>PostgreSQL</productname>
2014-08-25 21:34:51 +02:00
version to another. Upgrades can be performed in minutes,
2017-10-09 03:44:17 +02:00
particularly with <option>--link</option> mode. It requires steps similar to
2020-09-01 00:33:37 +02:00
<application>pg_dumpall</application> above, e.g., starting/stopping the server,
2017-10-09 03:44:17 +02:00
running <application>initdb</application>. The <application>pg_upgrade</application> <link
linkend="pgupgrade">documentation</link> outlines the necessary steps.
2011-01-31 18:32:03 +01:00
</para>
2014-08-25 21:34:51 +02:00
</sect2>
<sect2 id="upgrading-via-replication">
<title>Upgrading Data via Replication</title>
2011-01-31 18:32:03 +01:00
<para>
2018-06-12 03:34:32 +02:00
It is also possible to use logical replication methods to create a standby
server with the updated version of <productname>PostgreSQL</productname>.
This is possible because logical replication supports
2011-02-01 21:21:22 +01:00
replication between different major versions of
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname>. The standby can be on the same computer or
2020-06-15 19:12:58 +02:00
a different computer. Once it has synced up with the primary server
2017-10-09 03:44:17 +02:00
(running the older version of <productname>PostgreSQL</productname>), you can
2020-06-15 19:12:58 +02:00
switch primaries and make the standby the primary and shut down the older
2011-01-31 18:32:03 +01:00
database instance. Such a switch-over results in only several seconds
of downtime for an upgrade.
</para>
2018-06-12 03:34:32 +02:00
<para>
This method of upgrading can be performed using the built-in logical
replication facilities as well as using external logical replication
systems such as <productname>pglogical</productname>,
<productname>Slony</productname>, <productname>Londiste</productname>, and
<productname>Bucardo</productname>.
</para>
2011-01-31 18:32:03 +01:00
</sect2>
</sect1>
2007-12-25 05:00:44 +01:00
<sect1 id="preventing-server-spoofing">
<title>Preventing Server Spoofing</title>
<indexterm zone="preventing-server-spoofing">
<primary>server spoofing</primary>
</indexterm>
<para>
While the server is running, it is not possible for a malicious user
2009-04-24 22:46:16 +02:00
to take the place of the normal database server. However, when the
2010-02-03 18:25:06 +01:00
server is down, it is possible for a local user to spoof the normal
2007-12-25 05:00:44 +01:00
server by starting their own server. The spoof server could read
passwords and queries sent by clients, but could not return any data
2017-10-09 03:44:17 +02:00
because the <varname>PGDATA</varname> directory would still be secure because
2007-12-25 05:00:44 +01:00
of directory permissions. Spoofing is possible because any user can
start a database server; a client cannot identify an invalid server
unless it is specially configured.
</para>
<para>
2017-10-09 03:44:17 +02:00
One way to prevent spoofing of <literal>local</literal>
2007-12-25 05:00:44 +01:00
connections is to use a Unix domain socket directory (<xref
2017-11-23 15:39:47 +01:00
linkend="guc-unix-socket-directories"/>) that has write permission only
2007-12-25 05:00:44 +01:00
for a trusted local user. This prevents a malicious user from creating
2008-01-31 18:22:43 +01:00
their own socket file in that directory. If you are concerned that
2017-10-09 03:44:17 +02:00
some applications might still reference <filename>/tmp</filename> for the
2008-02-01 00:03:16 +01:00
socket file and hence be vulnerable to spoofing, during operating system
2017-10-09 03:44:17 +02:00
startup create a symbolic link <filename>/tmp/.s.PGSQL.5432</filename> that points
2008-02-01 00:03:16 +01:00
to the relocated socket file. You also might need to modify your
2017-10-09 03:44:17 +02:00
<filename>/tmp</filename> cleanup script to prevent removal of the symbolic link.
2008-01-31 18:22:43 +01:00
</para>
2016-08-19 03:41:10 +02:00
<para>
2017-10-09 03:44:17 +02:00
Another option for <literal>local</literal> connections is for clients to use
<link linkend="libpq-connect-requirepeer"><literal>requirepeer</literal></link>
2016-08-19 03:41:10 +02:00
to specify the required owner of the server process connected to
the socket.
</para>
2008-01-31 18:22:43 +01:00
<para>
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
To prevent spoofing on TCP connections, either use
SSL certificates and make sure that clients check the server's certificate,
or use GSSAPI encryption (or both, if they're on separate connections).
</para>
<para>
To prevent spoofing with SSL, the server
2017-10-09 03:44:17 +02:00
must be configured to accept only <literal>hostssl</literal> connections (<xref
2017-11-23 15:39:47 +01:00
linkend="auth-pg-hba-conf"/>) and have SSL key and certificate files
(<xref linkend="ssl-tcp"/>). The TCP client must connect using
2017-10-09 03:44:17 +02:00
<literal>sslmode=verify-ca</literal> or
<literal>verify-full</literal> and have the appropriate root certificate
2017-11-23 15:39:47 +01:00
file installed (<xref linkend="libq-ssl-certificates"/>).
2007-12-25 05:00:44 +01:00
</para>
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
<para>
To prevent spoofing with GSSAPI, the server must be configured to accept
only <literal>hostgssenc</literal> connections
(<xref linkend="auth-pg-hba-conf"/>) and use <literal>gss</literal>
authentication with them. The TCP client must connect
using <literal>gssencmode=require</literal>.
</para>
2007-12-25 05:00:44 +01:00
</sect1>
2010-11-23 21:27:50 +01:00
2005-05-09 19:26:22 +02:00
<sect1 id="encryption-options">
<title>Encryption Options</title>
2005-05-09 19:13:04 +02:00
2005-05-09 19:26:22 +02:00
<indexterm zone="encryption-options">
2005-05-09 19:13:04 +02:00
<primary>encryption</primary>
</indexterm>
<para>
<productname>PostgreSQL</productname> offers encryption at several
levels, and provides flexibility in protecting data from disclosure
due to database server theft, unscrupulous administrators, and
2005-05-25 04:56:15 +02:00
insecure networks. Encryption might also be required to secure
sensitive data such as medical records or financial transactions.
2005-05-09 19:13:04 +02:00
</para>
2010-11-23 21:27:50 +01:00
2005-05-09 19:13:04 +02:00
<variablelist>
<varlistentry>
2018-02-03 17:29:23 +01:00
<term>Password Encryption</term>
2005-05-09 19:13:04 +02:00
<listitem>
<para>
2018-02-03 17:29:23 +01:00
Database user passwords are stored as hashes (determined by the setting
<xref linkend="guc-password-encryption"/>), so the administrator cannot
determine the actual password assigned to the user. If SCRAM or MD5
encryption is used for client authentication, the unencrypted password is
never even temporarily present on the server because the client encrypts
it before being sent across the network. SCRAM is preferred, because it
is an Internet standard and is more secure than the PostgreSQL-specific
MD5 authentication protocol.
2005-05-09 19:13:04 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Encryption For Specific Columns</term>
<listitem>
<para>
2017-11-23 15:39:47 +01:00
The <xref linkend="pgcrypto"/> module allows certain fields to be
2011-01-26 15:22:21 +01:00
stored encrypted.
2010-08-15 03:57:03 +02:00
This is useful if only some of the data is sensitive.
2005-05-09 19:13:04 +02:00
The client supplies the decryption key and the data is decrypted
on the server and then sent to the client.
</para>
2010-11-23 21:27:50 +01:00
2005-05-09 19:13:04 +02:00
<para>
The decrypted data and the decryption key are present on the
server for a brief time while it is being decrypted and
communicated between the client and server. This presents a brief
moment where the data and keys can be intercepted by someone with
complete access to the database server, such as the system
administrator.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Data Partition Encryption</term>
<listitem>
<para>
2013-12-06 15:47:39 +01:00
Storage encryption can be performed at the file system level or the
block level. Linux file system encryption options include eCryptfs
and EncFS, while FreeBSD uses PEFS. Block level or full disk
encryption options include dm-crypt + LUKS on Linux and GEOM
modules geli and gbde on FreeBSD. Many other operating systems
support this functionality, including Windows.
2005-05-09 19:13:04 +02:00
</para>
<para>
2005-11-05 00:14:02 +01:00
This mechanism prevents unencrypted data from being read from the
2005-05-25 04:56:15 +02:00
drives if the drives or the entire computer is stolen. This does
2005-11-05 00:14:02 +01:00
not protect against attacks while the file system is mounted,
2005-05-25 04:56:15 +02:00
because when mounted, the operating system provides an unencrypted
2005-11-05 00:14:02 +01:00
view of the data. However, to mount the file system, you need some
2005-05-25 04:56:15 +02:00
way for the encryption key to be passed to the operating system,
and sometimes the key is stored somewhere on the host that mounts
the disk.
2005-05-09 19:13:04 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Encrypting Data Across A Network</term>
<listitem>
<para>
SSL connections encrypt all data sent across the network: the
password, the queries, and the data returned. The
2017-10-09 03:44:17 +02:00
<filename>pg_hba.conf</filename> file allows administrators to specify
which hosts can use non-encrypted connections (<literal>host</literal>)
2005-05-09 19:13:04 +02:00
and which require SSL-encrypted connections
2017-10-09 03:44:17 +02:00
(<literal>hostssl</literal>). Also, clients can specify that they
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
connect to servers only via SSL.
</para>
<para>
GSSAPI-encrypted connections encrypt all data sent across the network,
including queries and data returned. (No password is sent across the
network.) The <filename>pg_hba.conf</filename> file allows
administrators to specify which hosts can use non-encrypted connections
(<literal>host</literal>) and which require GSSAPI-encrypted connections
(<literal>hostgssenc</literal>). Also, clients can specify that they
connect to servers only on GSSAPI-encrypted connections
(<literal>gssencmode=require</literal>).
</para>
<para>
<application>Stunnel</application> or
<application>SSH</application> can also be used to encrypt
transmissions.
2005-05-09 19:13:04 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>SSL Host Authentication</term>
<listitem>
2010-05-27 01:49:19 +02:00
<para>
2008-11-13 10:45:25 +01:00
It is possible for both the client and server to provide SSL
certificates to each other. It takes some extra configuration
2005-05-09 19:13:04 +02:00
on each side, but this provides stronger verification of identity
2005-05-25 04:56:15 +02:00
than the mere use of passwords. It prevents a computer from
2005-05-09 19:13:04 +02:00
pretending to be the server just long enough to read the password
2017-10-09 03:44:17 +02:00
sent by the client. It also helps prevent <quote>man in the middle</quote>
2005-05-09 19:13:04 +02:00
attacks where a computer between the client and server pretends to
be the server and reads and passes all data between the client and
server.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Client-Side Encryption</term>
<listitem>
<para>
2010-05-27 01:49:19 +02:00
If the system administrator for the server's machine cannot be trusted,
it is necessary
2005-05-09 19:13:04 +02:00
for the client to encrypt the data; this way, unencrypted data
never appears on the database server. Data is encrypted on the
client before being sent to the server, and database results have
2005-05-25 04:56:15 +02:00
to be decrypted on the client before being used.
2005-05-09 19:13:04 +02:00
</para>
</listitem>
</varlistentry>
</variablelist>
</sect1>
2000-10-20 16:00:49 +02:00
<sect1 id="ssl-tcp">
2000-09-06 21:54:52 +02:00
<title>Secure TCP/IP Connections with SSL</title>
2000-08-29 06:15:43 +02:00
2001-05-13 00:51:36 +02:00
<indexterm zone="ssl-tcp">
<primary>SSL</primary>
2022-03-30 13:07:30 +02:00
<secondary>TLS</secondary>
2001-05-13 00:51:36 +02:00
</indexterm>
2000-08-29 06:15:43 +02:00
<para>
2017-10-09 03:44:17 +02:00
<productname>PostgreSQL</productname> has native support for using
<acronym>SSL</acronym> connections to encrypt client/server communications
2003-03-24 15:32:51 +01:00
for increased security. This requires that
<productname>OpenSSL</productname> is installed on both client and
2017-10-09 03:44:17 +02:00
server systems and that support in <productname>PostgreSQL</productname> is
2017-11-23 15:39:47 +01:00
enabled at build time (see <xref linkend="installation"/>).
2000-08-29 06:15:43 +02:00
</para>
2022-03-30 13:07:30 +02:00
<para>
The terms <acronym>SSL</acronym> and <acronym>TLS</acronym> are often used
interchangeably to mean a secure encrypted connection using a
<acronym>TLS</acronym> protocol. <acronym>SSL</acronym> protocols are the
precursors to <acronym>TLS</acronym> protocols, and the term
<acronym>SSL</acronym> is still used for encrypted connections even though
2022-04-16 09:05:07 +02:00
<acronym>SSL</acronym> protocols are no longer supported.
2022-03-30 13:07:30 +02:00
<acronym>SSL</acronym> is used interchangeably with <acronym>TLS</acronym>
in <productname>PostgreSQL</productname>.
</para>
2018-03-07 17:32:51 +01:00
<sect2 id="ssl-setup">
<title>Basic Setup</title>
2000-08-29 06:15:43 +02:00
<para>
2017-10-09 03:44:17 +02:00
With <acronym>SSL</acronym> support compiled in, the
<productname>PostgreSQL</productname> server can be started with
2022-03-30 13:07:30 +02:00
support for encrypted connections using <acronym>TLS</acronym> protocols
2022-09-26 05:56:47 +02:00
enabled by setting the parameter
2017-11-23 15:39:47 +01:00
<xref linkend="guc-ssl"/> to <literal>on</literal> in
2017-10-09 03:44:17 +02:00
<filename>postgresql.conf</filename>. The server will listen for both normal
and <acronym>SSL</acronym> connections on the same TCP port, and will negotiate
with any connecting client on whether to use <acronym>SSL</acronym>. By
2007-12-25 05:00:44 +01:00
default, this is at the client's option; see <xref
2017-11-23 15:39:47 +01:00
linkend="auth-pg-hba-conf"/> about how to set up the server to require
2017-10-09 03:44:17 +02:00
use of <acronym>SSL</acronym> for some or all connections.
2000-08-29 06:15:43 +02:00
</para>
2007-02-16 03:59:41 +01:00
<para>
2017-10-09 03:44:17 +02:00
To start in <acronym>SSL</acronym> mode, files containing the server certificate
2012-02-22 22:40:46 +01:00
and private key must exist. By default, these files are expected to be
2017-10-09 03:44:17 +02:00
named <filename>server.crt</filename> and <filename>server.key</filename>, respectively, in
2012-02-22 22:40:46 +01:00
the server's data directory, but other names and locations can be specified
2017-11-23 15:39:47 +01:00
using the configuration parameters <xref linkend="guc-ssl-cert-file"/>
and <xref linkend="guc-ssl-key-file"/>.
2016-03-19 11:03:22 +01:00
</para>
<para>
2008-03-31 04:43:14 +02:00
On Unix systems, the permissions on <filename>server.key</filename> must
disallow any access to world or group; achieve this by the command
2016-03-19 11:03:22 +01:00
<command>chmod 0600 server.key</command>. Alternatively, the file can be
owned by root and have group read access (that is, <literal>0640</literal>
permissions). That setup is intended for installations where certificate
and key files are managed by the operating system. The user under which
the <productname>PostgreSQL</productname> server runs should then be made a
member of the group that has access to those certificate and key files.
</para>
2018-04-07 23:45:39 +02:00
<para>
If the data directory allows group read access then certificate files may
need to be located outside of the data directory in order to conform to the
security requirements outlined above. Generally, group access is enabled
to allow an unprivileged user to backup the database, and in that case the
backup software will not be able to read the certificate files and will
likely error.
</para>
2016-03-19 11:03:22 +01:00
<para>
2017-01-04 18:43:52 +01:00
If the private key is protected with a passphrase, the
server will prompt for the passphrase and will not start until it has
been entered.
2019-11-09 10:13:14 +01:00
Using a passphrase by default disables the ability to change the server's
SSL configuration without a server restart, but see <xref
linkend="guc-ssl-passphrase-command-supports-reload"/>.
2017-01-04 18:43:52 +01:00
Furthermore, passphrase-protected private keys cannot be used at all
on Windows.
2007-12-25 05:00:44 +01:00
</para>
2010-05-27 01:49:19 +02:00
<para>
2018-01-21 03:47:02 +01:00
The first certificate in <filename>server.crt</filename> must be the
server's certificate because it must match the server's private key.
The certificates of <quote>intermediate</quote> certificate authorities
can also be appended to the file. Doing this avoids the necessity of
storing intermediate certificates on clients, assuming the root and
2021-04-24 03:44:13 +02:00
intermediate certificates were created with <literal>v3_ca</literal>
2020-08-31 22:21:03 +02:00
extensions. (This sets the certificate's basic constraint of
<literal>CA</literal> to <literal>true</literal>.)
This allows easier expiration of intermediate certificates.
2018-01-21 03:47:02 +01:00
</para>
<para>
It is not necessary to add the root certificate to
<filename>server.crt</filename>. Instead, clients must have the root
certificate of the server's certificate chain.
2010-05-27 01:49:19 +02:00
</para>
2018-03-07 17:32:51 +01:00
</sect2>
<sect2 id="ssl-openssl-config">
<title>OpenSSL Configuration</title>
<para>
<productname>PostgreSQL</productname> reads the system-wide
<productname>OpenSSL</productname> configuration file. By default, this
file is named <filename>openssl.cnf</filename> and is located in the
directory reported by <literal>openssl version -d</literal>.
This default can be overridden by setting environment variable
<envar>OPENSSL_CONF</envar> to the name of the desired configuration file.
</para>
<para>
<productname>OpenSSL</productname> supports a wide range of ciphers
and authentication algorithms, of varying strength. While a list of
ciphers can be specified in the <productname>OpenSSL</productname>
configuration file, you can specify ciphers specifically for use by
the database server by modifying <xref linkend="guc-ssl-ciphers"/> in
<filename>postgresql.conf</filename>.
</para>
<note>
<para>
It is possible to have authentication without encryption overhead by
using <literal>NULL-SHA</literal> or <literal>NULL-MD5</literal> ciphers. However,
a man-in-the-middle could read and pass communications between client
and server. Also, encryption overhead is minimal compared to the
overhead of authentication. For these reasons NULL ciphers are not
recommended.
</para>
</note>
</sect2>
2010-05-27 01:49:19 +02:00
2008-11-20 10:29:36 +01:00
<sect2 id="ssl-client-certificates">
2011-01-29 19:00:18 +01:00
<title>Using Client Certificates</title>
2010-05-27 01:49:19 +02:00
2016-07-16 20:12:44 +02:00
<para>
2018-01-21 03:47:02 +01:00
To require the client to supply a trusted certificate,
place certificates of the root certificate authorities
(<acronym>CA</acronym>s) you trust in a file in the data
2017-11-23 15:39:47 +01:00
directory, set the parameter <xref linkend="guc-ssl-ca-file"/> in
2018-01-21 03:47:02 +01:00
<filename>postgresql.conf</filename> to the new file name, and add the
2019-03-09 21:09:10 +01:00
authentication option <literal>clientcert=verify-ca</literal> or
<literal>clientcert=verify-full</literal> to the appropriate
2018-01-21 03:47:02 +01:00
<literal>hostssl</literal> line(s) in <filename>pg_hba.conf</filename>.
A certificate will then be requested from the client during SSL
connection startup. (See <xref linkend="libpq-ssl"/> for a description
2019-03-09 21:09:10 +01:00
of how to set up certificates on the client.)
</para>
<para>
For a <literal>hostssl</literal> entry with
<literal>clientcert=verify-ca</literal>, the server will verify
that the client's certificate is signed by one of the trusted
certificate authorities. If <literal>clientcert=verify-full</literal>
is specified, the server will not only verify the certificate
chain, but it will also check whether the username or its mapping
matches the <literal>cn</literal> (Common Name) of the provided certificate.
Note that certificate chain validation is always ensured when the
<literal>cert</literal> authentication method is used
(see <xref linkend="auth-cert"/>).
2016-07-16 20:12:44 +02:00
</para>
<para>
2018-01-21 03:47:02 +01:00
Intermediate certificates that chain up to existing root certificates
can also appear in the <xref linkend="guc-ssl-ca-file"/> file if
you wish to avoid storing them on clients (assuming the root and
intermediate certificates were created with <literal>v3_ca</literal>
extensions). Certificate Revocation List (CRL) entries are also
2021-12-03 14:15:50 +01:00
checked if the parameter <xref linkend="guc-ssl-crl-file"/> or
<xref linkend="guc-ssl-crl-dir"/> is set.
2007-12-25 05:00:44 +01:00
</para>
<para>
2016-07-16 20:12:44 +02:00
The <literal>clientcert</literal> authentication option is available for
2017-10-09 03:44:17 +02:00
all authentication methods, but only in <filename>pg_hba.conf</filename> lines
specified as <literal>hostssl</literal>. When <literal>clientcert</literal> is
2020-10-05 21:48:40 +02:00
not specified, the server verifies the client certificate against its CA
file only if a client certificate is presented and the CA is configured.
2019-03-09 21:09:10 +01:00
</para>
<para>
There are two approaches to enforce that users provide a certificate during login.
</para>
<para>
The first approach makes use of the <literal>cert</literal> authentication
method for <literal>hostssl</literal> entries in <filename>pg_hba.conf</filename>,
such that the certificate itself is used for authentication while also
providing ssl connection security. See <xref linkend="auth-cert"/> for details.
(It is not necessary to specify any <literal>clientcert</literal> options
explicitly when using the <literal>cert</literal> authentication method.)
In this case, the <literal>cn</literal> (Common Name) provided in
the certificate is checked against the user name or an applicable mapping.
2010-05-27 01:49:19 +02:00
</para>
2008-11-20 10:29:36 +01:00
<para>
2019-03-09 21:09:10 +01:00
The second approach combines any authentication method for <literal>hostssl</literal>
entries with the verification of client certificates by setting the
<literal>clientcert</literal> authentication option to <literal>verify-ca</literal>
or <literal>verify-full</literal>. The former option only enforces that
the certificate is valid, while the latter also ensures that the
<literal>cn</literal> (Common Name) in the certificate matches
the user name or an applicable mapping.
2007-02-16 03:59:41 +01:00
</para>
2008-11-20 10:29:36 +01:00
</sect2>
2007-02-16 03:59:41 +01:00
2008-11-20 10:29:36 +01:00
<sect2 id="ssl-server-files">
<title>SSL Server File Usage</title>
2011-05-04 19:24:07 +02:00
<para>
2017-11-23 15:39:47 +01:00
<xref linkend="ssl-file-usage"/> summarizes the files that are
2012-02-22 22:40:46 +01:00
relevant to the SSL setup on the server. (The shown file names are default
2017-09-01 20:18:45 +02:00
names. The locally configured names could be different.)
2011-05-04 19:24:07 +02:00
</para>
2007-12-25 05:00:44 +01:00
2007-12-25 07:15:34 +01:00
<table id="ssl-file-usage">
<title>SSL Server File Usage</title>
<tgroup cols="3">
<thead>
<row>
<entry>File</entry>
<entry>Contents</entry>
<entry>Effect</entry>
</row>
</thead>
<tbody>
<row>
2017-11-23 15:39:47 +01:00
<entry><xref linkend="guc-ssl-cert-file"/> (<filename>$PGDATA/server.crt</filename>)</entry>
2007-12-25 07:15:34 +01:00
<entry>server certificate</entry>
2010-05-27 01:49:19 +02:00
<entry>sent to client to indicate server's identity</entry>
2007-12-25 07:15:34 +01:00
</row>
<row>
2017-11-23 15:39:47 +01:00
<entry><xref linkend="guc-ssl-key-file"/> (<filename>$PGDATA/server.key</filename>)</entry>
2007-12-25 07:15:34 +01:00
<entry>server private key</entry>
2010-05-27 01:49:19 +02:00
<entry>proves server certificate was sent by the owner; does not indicate
2007-12-25 18:06:52 +01:00
certificate owner is trustworthy</entry>
2007-12-25 07:15:34 +01:00
</row>
<row>
2017-11-23 15:39:47 +01:00
<entry><xref linkend="guc-ssl-ca-file"/></entry>
2007-12-25 07:15:34 +01:00
<entry>trusted certificate authorities</entry>
2008-11-20 10:29:36 +01:00
<entry>checks that client certificate is
2007-12-25 07:15:34 +01:00
signed by a trusted certificate authority</entry>
</row>
<row>
2017-11-23 15:39:47 +01:00
<entry><xref linkend="guc-ssl-crl-file"/></entry>
2007-12-25 07:15:34 +01:00
<entry>certificates revoked by certificate authorities</entry>
<entry>client certificate must not be on this list</entry>
</row>
</tbody>
</tgroup>
</table>
2011-05-04 19:24:07 +02:00
<para>
2017-01-03 03:37:12 +01:00
The server reads these files at server start and whenever the server
2017-10-09 03:44:17 +02:00
configuration is reloaded. On <systemitem class="osname">Windows</systemitem>
2017-01-03 03:37:12 +01:00
systems, they are also re-read whenever a new backend process is spawned
for a new client connection.
</para>
<para>
If an error in these files is detected at server start, the server will
refuse to start. But if an error is detected during a configuration
2017-01-04 18:43:52 +01:00
reload, the files are ignored and the old SSL configuration continues to
2017-10-09 03:44:17 +02:00
be used. On <systemitem class="osname">Windows</systemitem> systems, if an error in
2017-01-04 18:43:52 +01:00
these files is detected at backend start, that backend will be unable to
2017-01-03 03:37:12 +01:00
establish an SSL connection. In all these cases, the error condition is
reported in the server log.
2011-05-04 19:24:07 +02:00
</para>
2008-11-20 10:29:36 +01:00
</sect2>
2007-12-25 07:15:34 +01:00
<sect2 id="ssl-certificate-creation">
2018-01-21 03:47:02 +01:00
<title>Creating Certificates</title>
2007-12-25 05:00:44 +01:00
<para>
2018-01-21 03:47:02 +01:00
To create a simple self-signed certificate for the server, valid for 365
2017-04-18 20:21:57 +02:00
days, use the following <productname>OpenSSL</productname> command,
2018-01-21 03:47:02 +01:00
replacing <replaceable>dbhost.yourdomain.com</replaceable> with the
server's host name:
2001-01-08 22:01:54 +01:00
<programlisting>
2017-04-18 14:50:15 +02:00
openssl req -new -x509 -days 365 -nodes -text -out server.crt \
2018-01-21 03:47:02 +01:00
-keyout server.key -subj "/CN=<replaceable>dbhost.yourdomain.com</replaceable>"
2001-01-08 22:01:54 +01:00
</programlisting>
2017-04-18 14:50:15 +02:00
Then do:
2008-03-31 04:43:14 +02:00
<programlisting>
chmod og-rwx server.key
</programlisting>
because the server will reject the file if its permissions are more
liberal than this.
2007-12-25 05:00:44 +01:00
For more details on how to create your server private key and
2017-10-09 03:44:17 +02:00
certificate, refer to the <productname>OpenSSL</productname> documentation.
2007-12-25 05:00:44 +01:00
</para>
2004-09-23 15:16:02 +02:00
2007-12-25 05:00:44 +01:00
<para>
2018-01-21 03:47:02 +01:00
While a self-signed certificate can be used for testing, a certificate
signed by a certificate authority (<acronym>CA</acronym>) (usually an
enterprise-wide root <acronym>CA</acronym>) should be used in production.
2007-12-25 05:00:44 +01:00
</para>
2004-09-23 15:16:02 +02:00
2018-01-21 03:47:02 +01:00
<para>
To create a server certificate whose identity can be validated
by clients, first create a certificate signing request
(<acronym>CSR</acronym>) and a public/private key file:
<programlisting>
openssl req -new -nodes -text -out root.csr \
-keyout root.key -subj "/CN=<replaceable>root.yourdomain.com</replaceable>"
chmod og-rwx root.key
</programlisting>
Then, sign the request with the key to create a root certificate
authority (using the default <productname>OpenSSL</productname>
configuration file location on <productname>Linux</productname>):
<programlisting>
openssl x509 -req -in root.csr -text -days 3650 \
-extfile /etc/ssl/openssl.cnf -extensions v3_ca \
-signkey root.key -out root.crt
</programlisting>
Finally, create a server certificate signed by the new root certificate
authority:
<programlisting>
openssl req -new -nodes -text -out server.csr \
-keyout server.key -subj "/CN=<replaceable>dbhost.yourdomain.com</replaceable>"
chmod og-rwx server.key
openssl x509 -req -in server.csr -text -days 365 \
-CA root.crt -CAkey root.key -CAcreateserial \
-out server.crt
</programlisting>
<filename>server.crt</filename> and <filename>server.key</filename>
should be stored on the server, and <filename>root.crt</filename> should
be stored on the client so the client can verify that the server's leaf
2018-05-21 20:49:53 +02:00
certificate was signed by its trusted root certificate.
2018-01-21 03:47:02 +01:00
<filename>root.key</filename> should be stored offline for use in
creating future certificates.
</para>
<para>
It is also possible to create a chain of trust that includes
intermediate certificates:
<programlisting>
# root
openssl req -new -nodes -text -out root.csr \
-keyout root.key -subj "/CN=<replaceable>root.yourdomain.com</replaceable>"
chmod og-rwx root.key
openssl x509 -req -in root.csr -text -days 3650 \
-extfile /etc/ssl/openssl.cnf -extensions v3_ca \
-signkey root.key -out root.crt
# intermediate
openssl req -new -nodes -text -out intermediate.csr \
-keyout intermediate.key -subj "/CN=<replaceable>intermediate.yourdomain.com</replaceable>"
chmod og-rwx intermediate.key
openssl x509 -req -in intermediate.csr -text -days 1825 \
-extfile /etc/ssl/openssl.cnf -extensions v3_ca \
-CA root.crt -CAkey root.key -CAcreateserial \
-out intermediate.crt
# leaf
openssl req -new -nodes -text -out server.csr \
-keyout server.key -subj "/CN=<replaceable>dbhost.yourdomain.com</replaceable>"
chmod og-rwx server.key
openssl x509 -req -in server.csr -text -days 365 \
-CA intermediate.crt -CAkey intermediate.key -CAcreateserial \
-out server.crt
</programlisting>
<filename>server.crt</filename> and
<filename>intermediate.crt</filename> should be concatenated
into a certificate file bundle and stored on the server.
<filename>server.key</filename> should also be stored on the server.
<filename>root.crt</filename> should be stored on the client so
the client can verify that the server's leaf certificate was signed
by a chain of certificates linked to its trusted root certificate.
<filename>root.key</filename> and <filename>intermediate.key</filename>
should be stored offline for use in creating future certificates.
</para>
2007-12-25 05:00:44 +01:00
</sect2>
2004-09-27 00:51:49 +02:00
2000-08-29 06:15:43 +02:00
</sect1>
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
<sect1 id="gssapi-enc">
2019-09-08 10:26:35 +02:00
<title>Secure TCP/IP Connections with GSSAPI Encryption</title>
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
<indexterm zone="gssapi-enc">
<primary>gssapi</primary>
</indexterm>
<para>
<productname>PostgreSQL</productname> also has native support for
using <acronym>GSSAPI</acronym> to encrypt client/server communications for
increased security. Support requires that a <acronym>GSSAPI</acronym>
2020-12-28 23:44:17 +01:00
implementation (such as MIT Kerberos) is installed on both client and server
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
systems, and that support in <productname>PostgreSQL</productname> is
enabled at build time (see <xref linkend="installation"/>).
</para>
<sect2 id="gssapi-setup">
<title>Basic Setup</title>
<para>
The <productname>PostgreSQL</productname> server will listen for both
normal and <acronym>GSSAPI</acronym>-encrypted connections on the same TCP
2020-09-21 18:43:42 +02:00
port, and will negotiate with any connecting client whether to
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
use <acronym>GSSAPI</acronym> for encryption (and for authentication). By
default, this decision is up to the client (which means it can be
downgraded by an attacker); see <xref linkend="auth-pg-hba-conf"/> about
setting up the server to require the use of <acronym>GSSAPI</acronym> for
2019-05-17 01:22:28 +02:00
some or all connections.
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
</para>
2020-12-28 23:44:17 +01:00
<para>
When using <acronym>GSSAPI</acronym> for encryption, it is common to
use <acronym>GSSAPI</acronym> for authentication as well, since the
underlying mechanism will determine both client and server identities
(according to the <acronym>GSSAPI</acronym> implementation) in any
case. But this is not required;
another <productname>PostgreSQL</productname> authentication method
can be chosen to perform additional verification.
</para>
GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.
Add frontend and backend encryption support functions. Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.
In postmaster, pull GSSAPI authorization checking into a shared
function. Also share the initiator name between the encryption and
non-encryption codepaths.
For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts. "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition
into its own function to support this behavior.
Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security.
Thanks to Michael Paquier for the Windows fixes.
Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 21:02:33 +02:00
<para>
Other than configuration of the negotiation
behavior, <acronym>GSSAPI</acronym> encryption requires no setup beyond
that which is necessary for GSSAPI authentication. (For more information
on configuring that, see <xref linkend="gssapi-auth"/>.)
</para>
</sect2>
</sect1>
2000-10-20 16:00:49 +02:00
<sect1 id="ssh-tunnels">
2002-11-11 21:14:04 +01:00
<title>Secure TCP/IP Connections with <application>SSH</application> Tunnels</title>
2000-06-18 23:24:54 +02:00
2001-05-13 00:51:36 +02:00
<indexterm zone="ssh-tunnels">
<primary>ssh</primary>
</indexterm>
2000-06-18 23:24:54 +02:00
<para>
2010-02-03 18:25:06 +01:00
It is possible to use <application>SSH</application> to encrypt the network
2000-06-18 23:24:54 +02:00
connection between clients and a
2001-11-21 07:09:45 +01:00
<productname>PostgreSQL</productname> server. Done properly, this
2004-12-27 00:06:56 +01:00
provides an adequately secure network connection, even for non-SSL-capable
clients.
2000-06-18 23:24:54 +02:00
</para>
<para>
2002-11-11 21:14:04 +01:00
First make sure that an <application>SSH</application> server is
2003-03-24 15:32:51 +01:00
running properly on the same machine as the
<productname>PostgreSQL</productname> server and that you can log in using
2020-10-03 03:39:33 +02:00
<command>ssh</command> as some user; you then can establish a
secure tunnel to the remote server. A secure tunnel listens on a
local port and forwards all traffic to a port on the remote machine.
Traffic sent to the remote port can arrive on its
<literal>localhost</literal> address, or different bind
address if desired; it does not appear as coming from your
local machine. This command creates a secure tunnel from the client
machine to the remote machine <literal>foo.com</literal>:
2000-06-18 23:24:54 +02:00
<programlisting>
2008-02-26 17:07:16 +01:00
ssh -L 63333:localhost:5432 joe@foo.com
2000-06-18 23:24:54 +02:00
</programlisting>
2008-02-26 17:07:16 +01:00
The first number in the <option>-L</option> argument, 63333, is the
2020-10-03 03:39:33 +02:00
local port number of the tunnel; it can be any unused port. (IANA
reserves ports 49152 through 65535 for private use.) The name or IP
address after this is the remote bind address you are connecting to,
i.e., <literal>localhost</literal>, which is the default. The second
number, 5432, is the remote end of the tunnel, e.g., the port number
your database server is using. In order to connect to the database
server using this tunnel, you connect to port 63333 on the local
machine:
2000-06-18 23:24:54 +02:00
<programlisting>
2008-02-26 17:07:16 +01:00
psql -h localhost -p 63333 postgres
2000-06-18 23:24:54 +02:00
</programlisting>
2020-10-03 03:39:33 +02:00
To the database server it will then look as though you are
2008-02-26 17:07:16 +01:00
user <literal>joe</literal> on host <literal>foo.com</literal>
2020-10-03 03:39:33 +02:00
connecting to the <literal>localhost</literal> bind address, and it
2008-02-26 17:07:16 +01:00
will use whatever authentication procedure was configured for
2020-10-03 03:39:33 +02:00
connections by that user to that bind address. Note that the server will not
2008-02-26 17:07:16 +01:00
think the connection is SSL-encrypted, since in fact it is not
encrypted between the
2004-12-27 00:06:56 +01:00
<application>SSH</application> server and the
<productname>PostgreSQL</productname> server. This should not pose any
2020-10-03 03:39:33 +02:00
extra security risk because they are on the same machine.
2004-12-27 00:06:56 +01:00
</para>
2008-02-26 17:07:16 +01:00
2004-12-27 00:06:56 +01:00
<para>
In order for the
2002-03-09 06:11:38 +01:00
tunnel setup to succeed you must be allowed to connect via
2003-03-24 15:32:51 +01:00
<command>ssh</command> as <literal>joe@foo.com</literal>, just
2010-02-03 18:25:06 +01:00
as if you had attempted to use <command>ssh</command> to create a
2000-06-18 23:24:54 +02:00
terminal session.
</para>
2008-02-26 17:07:16 +01:00
<para>
2020-10-03 03:39:33 +02:00
You could also have set up port forwarding as
2008-02-26 17:07:16 +01:00
<programlisting>
ssh -L 63333:foo.com:5432 joe@foo.com
</programlisting>
but then the database server will see the connection as coming in
2020-10-03 03:39:33 +02:00
on its <literal>foo.com</literal> bind address, which is not opened by
2008-02-26 17:07:16 +01:00
the default setting <literal>listen_addresses =
'localhost'</literal>. This is usually not what you want.
</para>
<para>
If you have to <quote>hop</quote> to the database server via some
login host, one possible setup could look like this:
<programlisting>
ssh -L 63333:db.foo.com:5432 joe@shell.foo.com
</programlisting>
2008-02-26 19:01:26 +01:00
Note that this way the connection
from <literal>shell.foo.com</literal>
to <literal>db.foo.com</literal> will not be encrypted by the SSH
tunnel.
2008-02-26 17:07:16 +01:00
SSH offers quite a few configuration possibilities when the network
is restricted in various ways. Please refer to the SSH
documentation for details.
</para>
2000-09-06 21:54:52 +02:00
<tip>
<para>
2002-11-11 21:14:04 +01:00
Several other applications exist that can provide secure tunnels using
2000-09-06 21:54:52 +02:00
a procedure similar in concept to the one just described.
</para>
</tip>
2000-06-18 23:24:54 +02:00
</sect1>
1998-09-30 07:41:54 +02:00
2011-10-25 20:02:55 +02:00
<sect1 id="event-log-registration">
2017-10-09 03:44:17 +02:00
<title>Registering <application>Event Log</application> on <systemitem
class="osname">Windows</systemitem></title>
2011-10-25 20:02:55 +02:00
<indexterm zone="event-log-registration">
<primary>event log</primary>
<secondary>event log</secondary>
</indexterm>
<para>
2017-10-09 03:44:17 +02:00
To register a <systemitem class="osname">Windows</systemitem>
<application>event log</application> library with the operating system,
2011-10-25 20:02:55 +02:00
issue this command:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>regsvr32 <replaceable>pgsql_library_directory</replaceable>/pgevent.dll</userinput>
2011-10-25 20:02:55 +02:00
</screen>
This creates registry entries used by the event viewer, under the default
event source named <literal>PostgreSQL</literal>.
</para>
<para>
To specify a different event source name (see
2017-11-23 15:39:47 +01:00
<xref linkend="guc-event-source"/>), use the <literal>/n</literal>
2011-10-25 20:02:55 +02:00
and <literal>/i</literal> options:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>regsvr32 /n /i:<replaceable>event_source_name</replaceable> <replaceable>pgsql_library_directory</replaceable>/pgevent.dll</userinput>
2011-10-25 20:02:55 +02:00
</screen>
</para>
<para>
2017-10-09 03:44:17 +02:00
To unregister the <application>event log</application> library from
2011-10-25 20:02:55 +02:00
the operating system, issue this command:
<screen>
2017-10-09 03:44:17 +02:00
<userinput>regsvr32 /u [/i:<replaceable>event_source_name</replaceable>] <replaceable>pgsql_library_directory</replaceable>/pgevent.dll</userinput>
2011-10-25 20:02:55 +02:00
</screen>
</para>
<note>
<para>
To enable event logging in the database server, modify
2017-11-23 15:39:47 +01:00
<xref linkend="guc-log-destination"/> to include
2011-10-25 20:02:55 +02:00
<literal>eventlog</literal> in <filename>postgresql.conf</filename>.
</para>
</note>
</sect1>
2005-01-23 01:30:59 +01:00
</chapter>