This chapter discusses how to set up and run the database server and its interactions with the operating system. postgres user As with any server daemon that is accessible to the outside world, it is advisable to run PostgreSQL under a separate user account. This user account should only own the data that is managed by the server, and should not be shared with other daemons. (For example, using the user nobody is a bad idea.) It is not advisable to install executables owned by this user because compromised systems could then modify their own binaries. To add a Unix user account to your system, look for a command useradd or adduser. The user name postgres is often used, and is assumed throughout this book, but you can use another name if you like. database cluster data area database cluster Before you can do anything, you must initialize a database storage area on disk. We call this a database cluster. (SQL uses the term catalog cluster.) A database cluster is a collection of databases that is managed by a single instance of a running database server. After initialization, a database cluster will contain a database named postgres, which is meant as a default database for use by utilities, users and third party applications. The database server itself does not require the postgres database to exist, but many external utility programs assume it exists. Another database created within each cluster during initialization is called template1. As the name suggests, this will be used as a template for subsequently created databases; it should not be used for actual work. (See for information about creating new databases within a cluster.) In file system terms, a database cluster will be a single directory under which all data will be stored. We call this the data directory or data area. It is completely up to you where you choose to store your data. There is no default, although locations such as /usr/local/pgsql/data or /var/lib/pgsql/data are popular. To initialize a database cluster, use the command ,initdb which is installed with PostgreSQL. The desired file system location of your database cluster is indicated by the -D option, for example: $ initdb -D /usr/local/pgsql/data Note that you must execute this command while logged into the PostgreSQL user account, which is described in the previous section. As an alternative to the -D option, you can set the environment variable PGDATA. PGDATA Alternatively, you can run initdb via the programpg_ctl like so: $ pg_ctl -D /usr/local/pgsql/data initdb This may be more intuitive if you are using pg_ctl for starting and stopping the server (see ), so that pg_ctl would be the sole command you use for managing the database server instance. initdb will attempt to create the directory you specify if it does not already exist. It is likely that it will not have the permission to do so (if you followed our advice and created an unprivileged account). In that case you should create the directory yourself (as root) and change the owner to be the PostgreSQL user. Here is how this might be done: root# mkdir /usr/local/pgsql/data root# chown postgres /usr/local/pgsql/data root# su postgres postgres$ initdb -D /usr/local/pgsql/data initdb will refuse to run if the data directory looks like it has already been initialized. Because the data directory contains all the data stored in the database, it is essential that it be secured from unauthorized access. initdb therefore revokes access permissions from everyone but the PostgreSQL user. However, while the directory contents are secure, the default client authentication setup allows any local user to connect to the database and even become the database superuser. If you do not trust other local users, we recommend you use one of initdb's -W, --pwprompt or --pwfile options to assign a password to the database superuser. password of the superuser Also, specify -A md5 or -A password so that the default trust authentication mode is not used; or modify the generated pg_hba.conf file after running initdb, but before you start the server for the first time. (Other reasonable approaches include using peer authentication or file system permissions to restrict connections. See for more information.) initdb also initializes the default localelocale for the database cluster. Normally, it will just take the locale settings in the environment and apply them to the initialized database. It is possible to specify a different locale for the database; more information about that can be found in . The default sort order used within the particular database cluster is set by initdb, and while you can create new databases using different sort order, the order used in the template databases that initdb creates cannot be changed without dropping and recreating them. There is also a performance impact for using locales other than C or POSIX. Therefore, it is important to make this choice correctly the first time. initdb also sets the default character set encoding for the database cluster. Normally this should be chosen to match the locale setting. For details see . Network File Systems NFSNetwork File Systems Network Attached Storage (NAS)Network File Systems Many installations create database clusters on network file systems. Sometimes this is done directly via NFS, or by using a Network Attached Storage (NAS) device that uses NFS internally. PostgreSQL does nothing special for NFS file systems, meaning it assumes NFS behaves exactly like locally-connected drives (DAS, Direct Attached Storage). If client and server NFS implementations have non-standard semantics, this can cause reliability problems (see ). Specifically, delayed (asynchronous) writes to the NFS server can cause reliability problems; if possible, mount NFS file systems synchronously (without caching) to avoid this. Also, soft-mounting NFS is not recommended. (Storage Area Networks (SAN) use a low-level communication protocol rather than NFS.) Before anyone can access the database, you must start the database server. The database server program is called postgres.postgres The postgres program must know where to find the data it is supposed to use. This is done with the -D option. Thus, the simplest way to start the server is: $ postgres -D /usr/local/pgsql/data which will leave the server running in the foreground. This must be done while logged into the PostgreSQL user account. Without -D, the server will try to use the data directory named by the environment variable PGDATA. If that variable is not provided either, it will fail. Normally it is better to start postgres in the background. For this, use the usual Unix shell syntax: $ postgres -D /usr/local/pgsql/data >logfile 2>&1 & It is important to store the server's stdout and stderr output somewhere, as shown above. It will help for auditing purposes and to diagnose problems. (See for a more thorough discussion of log file handling.) The postgres program also takes a number of other command-line options. For more information, see the reference page and below. This shell syntax can get tedious quickly. Therefore the wrapper program pg_ctl is provided to simplify some tasks. For example: pg_ctl start -l logfile will start the server in the background and put the output into the named log file. The -D option has the same meaning here as for postgres. pg_ctl is also capable of stopping the server. Normally, you will want to start the database server when the computer boots. booting starting the server during Autostart scripts are operating-system-specific. There are a few distributed with PostgreSQL in the contrib/start-scripts directory. Installing one will require root privileges. Different systems have different conventions for starting up daemons at boot time. Many systems have a file /etc/rc.local or /etc/rc.d/rc.local. Others use init.d or rc.d directories. Whatever you do, the server must be run by the PostgreSQL user account and not by root or any other user. Therefore you probably should form your commands using su postgres -c '...'. For example: su postgres -c 'pg_ctl start -D /usr/local/pgsql/data -l serverlog' Here are a few more operating-system-specific suggestions. (In each case be sure to use the proper installation directory and user name where we show generic values.) For FreeBSD, look at the file contrib/start-scripts/freebsd in the PostgreSQL source distribution. FreeBSDstart script On OpenBSD, add the following lines to the file /etc/rc.local: OpenBSDstart script if [ -x /usr/local/pgsql/bin/pg_ctl -a -x /usr/local/pgsql/bin/postgres ]; then su -l postgres -c '/usr/local/pgsql/bin/pg_ctl start -s -l /var/postgresql/log -D /usr/local/pgsql/data' echo -n ' postgresql' fi On Linux systems either add Linuxstart script /usr/local/pgsql/bin/pg_ctl start -l logfile -D /usr/local/pgsql/data to /etc/rc.d/rc.local or /etc/rc.local or look at the file contrib/start-scripts/linux in the PostgreSQL source distribution. On NetBSD, use either the FreeBSD or Linux start scripts, depending on preference. NetBSDstart script On Solaris, create a file called /etc/init.d/postgresql that contains the following line: Solarisstart script su - postgres -c "/usr/local/pgsql/bin/pg_ctl start -l logfile -D /usr/local/pgsql/data" Then, create a symbolic link to it in /etc/rc3.d as S99postgresql. While the server is running, its PID is stored in the file postmaster.pid in the data directory. This is used to prevent multiple server instances from running in the same data directory and can also be used for shutting down the server. There are several common reasons the server might fail to start. Check the server's log file, or start it by hand (without redirecting standard output or standard error) and see what error messages appear. Below we explain some of the most common error messages in more detail. LOG: could not bind IPv4 socket: Address already in use HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry. FATAL: could not create TCP/IP listen socket This usually means just what it suggests: you tried to start another server on the same port where one is already running. However, if the kernel error message is not Address already in use or some variant of that, there might be a different problem. For example, trying to start a server on a reserved port number might draw something like: $ postgres -p 666 LOG: could not bind IPv4 socket: Permission denied HINT: Is another postmaster already running on port 666? If not, wait a few seconds and retry. FATAL: could not create TCP/IP listen socket A message like: FATAL: could not create shared memory segment: Invalid argument DETAIL: Failed system call was shmget(key=5440001, size=4011376640, 03600). probably means your kernel's limit on the size of shared memory is smaller than the work area PostgreSQL is trying to create (4011376640 bytes in this example). Or it could mean that you do not have System-V-style shared memory support configured into your kernel at all. As a temporary workaround, you can try starting the server with a smaller-than-normal number of buffers (). You will eventually want to reconfigure your kernel to increase the allowed shared memory size. You might also see this message when trying to start multiple servers on the same machine, if their total space requested exceeds the kernel limit. An error like: FATAL: could not create semaphores: No space left on device DETAIL: Failed system call was semget(5440126, 17, 03600). does not mean you've run out of disk space. It means your kernel's limit on the number of System V semaphores is smaller than the number PostgreSQL wants to create. As above, you might be able to work around the problem by starting the server with a reduced number of allowed connections (), but you'll eventually want to increase the kernel limit. If you get an illegal system call error, it is likely that shared memory or semaphores are not supported in your kernel at all. In that case your only option is to reconfigure the kernel to enable these features. Details about configuring System V IPC facilities are given in . Although the error conditions possible on the client side are quite varied and application-dependent, a few of them might be directly related to how the server was started. Conditions other than those shown below should be documented with the respective client application. psql: could not connect to server: Connection refused Is the server running on host "server.joe.com" and accepting TCP/IP connections on port 5432? This is the generic I couldn't find a server to talk to failure. It looks like the above when TCP/IP communication is attempted. A common mistake is to forget to configure the server to allow TCP/IP connections. Alternatively, you'll get this when attempting Unix-domain socket communication to a local server: psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? The last line is useful in verifying that the client is trying to connect to the right place. If there is in fact no server running there, the kernel error message will typically be either Connection refused or No such file or directory, as illustrated. (It is important to realize that Connection refused in this context does not mean that the server got your connection request and rejected it. That case will produce a different message, as shown in .) Other error messages such as Connection timed out might indicate more fundamental problems, like lack of network connectivity. PostgreSQL can sometimes exhaust various operating system resource limits, especially when multiple copies of the server are running on the same system, or in very large installations. This section explains the kernel resources used by PostgreSQL and the steps you can take to resolve problems related to kernel resource consumption. shared memory semaphores Shared memory and semaphores are collectively referred to as System V IPC (together with message queues, which are not relevant for PostgreSQL). Except on Windows, where PostgreSQL provides its own replacement implementation of these facilities, these facilities are required in order to run PostgreSQL. The complete lack of these facilities is usually manifested by an Illegal system call error upon server start. In that case there is no alternative but to reconfigure your kernel. PostgreSQL won't work without them. This situation is rare, however, among modern operating systems. When PostgreSQL exceeds one of the various hard IPC limits, the server will refuse to start and should leave an instructive error message describing the problem and what to do about it. (See also .) The relevant kernel parameters are named consistently across different systems; gives an overview. The methods to set them, however, vary. Suggestions for some platforms are given below. Prior to PostgreSQL 9.3, the amount of System V shared memory required to start the server was much larger. If you are running an older version of the server, please consult the documentation for your server version. Name Description Reasonable values SHMMAX Maximum size of shared memory segment (bytes) at least 1kB (more if running many copies of the server) SHMMIN Minimum size of shared memory segment (bytes) 1 SHMALL Total amount of shared memory available (bytes or pages) if bytes, same as SHMMAX; if pages, ceil(SHMMAX/PAGE_SIZE) SHMSEG Maximum number of shared memory segments per process only 1 segment is needed, but the default is much higher SHMMNI Maximum number of shared memory segments system-wide like SHMSEG plus room for other applications SEMMNI Maximum number of semaphore identifiers (i.e., sets) at least ceil((max_connections + autovacuum_max_workers + 4) / 16) SEMMNS Maximum number of semaphores system-wide ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications SEMMSL Maximum number of semaphores per set at least 17 SEMMAP Number of entries in semaphore map see text SEMVMX Maximum value of semaphore at least 1000 (The default is often 32767; do not change unless necessary) PostgreSQL requires a few bytes of System V shared memory (typically 48 bytes, on 64-bit platforms) for each copy of the server. On most modern operating systems, this amount can easily be allocated. However, if you are running many copies of the server, or if other applications are also using System V shared memory, it may be necessary to increase SHMMAX, the maximum size in bytes of a shared memory segment, or SHMALL, the total amount of System V shared memory system-wide. Note that SHMALL is measured in pages rather than bytes on many systems. Less likely to cause problems is the minimum size for shared memory segments (SHMMIN), which should be at most approximately 32 bytes for PostgreSQL (it is usually just 1). The maximum number of segments system-wide (SHMMNI) or per-process (SHMSEG) are unlikely to cause a problem unless your system has them set to zero. PostgreSQL uses one semaphore per allowed connection () and allowed autovacuum worker process (), in sets of 16. Each such set will also contain a 17th semaphore which contains a magic number, to detect collision with semaphore sets used by other applications. The maximum number of semaphores in the system is set by SEMMNS, which consequently must be at least as high as max_connections plus autovacuum_max_workers, plus one extra for each 16 allowed connections plus workers (see the formula in ). The parameter SEMMNI determines the limit on the number of semaphore sets that can exist on the system at one time. Hence this parameter must be at least ceil((max_connections + autovacuum_max_workers + 4) / 16). Lowering the number of allowed connections is a temporary workaround for failures, which are usually confusingly worded No space left on device, from the function semget. In some cases it might also be necessary to increase SEMMAP to be at least on the order of SEMMNS. This parameter defines the size of the semaphore resource map, in which each contiguous block of available semaphores needs an entry. When a semaphore set is freed it is either added to an existing entry that is adjacent to the freed block or it is registered under a new map entry. If the map is full, the freed semaphores get lost (until reboot). Fragmentation of the semaphore space could over time lead to fewer available semaphores than there should be. The SEMMSL parameter, which determines how many semaphores can be in a set, must be at least 17 for PostgreSQL. Various other settings related to semaphore undo, such as SEMMNU and SEMUME, do not affect PostgreSQL. AIX AIXIPC configuration At least as of version 5.1, it should not be necessary to do any special configuration for such parameters as SHMMAX, as it appears this is configured to allow all memory to be used as shared memory. That is the sort of configuration commonly used for other databases such as DB/2. It might, however, be necessary to modify the global ulimit information in /etc/security/limits, as the default hard limits for file sizes (fsize) and numbers of files (nofiles) might be too low. FreeBSD FreeBSDIPC configuration The default settings can be changed using the sysctl or loader interfaces. The following parameters can be set using sysctl: # sysctl kern.ipc.shmall=32768 # sysctl kern.ipc.shmmax=134217728 # sysctl kern.ipc.semmap=256 To have these settings persist over reboots, modify /etc/sysctl.conf. The remaining semaphore settings are read-only as far as sysctl is concerned, but can be set in /boot/loader.conf: kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 After modifying these values a reboot is required for the new settings to take affect. You might also want to configure your kernel to lock shared memory into RAM and prevent it from being paged out to swap. This can be accomplished using the sysctl setting kern.ipc.shm_use_phys. If running in FreeBSD jails by enabling sysctl's security.jail.sysvipc_allowed, postmasters running in different jails should be run by different operating system users. This improves security because it prevents non-root users from interfering with shared memory or semaphores in different jails, and it allows the PostgreSQL IPC cleanup code to function properly. (In FreeBSD 6.0 and later the IPC cleanup code does not properly detect processes in other jails, preventing the running of postmasters on the same port in different jails.) FreeBSD versions before 4.0 work like OpenBSD (see below). NetBSD NetBSDIPC configuration In NetBSD 5.0 and later, IPC parameters can be adjusted using sysctl, for example: $ sysctl -w kern.ipc.shmmax=16777216 To have these settings persist over reboots, modify /etc/sysctl.conf. You might also want to configure your kernel to lock shared memory into RAM and prevent it from being paged out to swap. This can be accomplished using the sysctl setting kern.ipc.shm_use_phys. NetBSD versions before 5.0 work like OpenBSD (see below), except that parameters should be set with the keyword options not option. OpenBSD OpenBSDIPC configuration The options SYSVSHM and SYSVSEM need to be enabled when the kernel is compiled. (They are by default.) The maximum size of shared memory is determined by the option SHMMAXPGS (in pages). The following shows an example of how to set the various parameters: option SYSVSHM option SHMMAXPGS=4096 option SHMSEG=256 option SYSVSEM option SEMMNI=256 option SEMMNS=512 option SEMMNU=256 option SEMMAP=256 You might also want to configure your kernel to lock shared memory into RAM and prevent it from being paged out to swap. This can be accomplished using the sysctl setting kern.ipc.shm_use_phys. HP-UX HP-UXIPC configuration The default settings tend to suffice for normal installations. On HP-UX 10, the factory default for SEMMNS is 128, which might be too low for larger database sites. IPC parameters can be set in the System Administration Manager (SAM) under Kernel ConfigurationConfigurable Parameters. Choose Create A New Kernel when you're done. Linux LinuxIPC configuration The default maximum segment size is 32 MB, and the default maximum total size is 2097152 pages. A page is almost always 4096 bytes except in unusual kernel configurations with huge pages (use getconf PAGE_SIZE to verify). The shared memory size settings can be changed via the sysctl interface. For example, to allow 16 GB: $ sysctl -w kernel.shmmax=17179869184 $ sysctl -w kernel.shmall=4194304 In addition these settings can be preserved between reboots in the file /etc/sysctl.conf. Doing that is highly recommended. Ancient distributions might not have the sysctl program, but equivalent changes can be made by manipulating the /proc file system: $ echo 17179869184 >/proc/sys/kernel/shmmax $ echo 4194304 >/proc/sys/kernel/shmall The remaining defaults are quite generously sized, and usually do not require changes. Mac OS X Mac OS XIPC configuration The recommended method for configuring shared memory in OS X is to create a file named /etc/sysctl.conf, containing variable assignments such as: kern.sysv.shmmax=4194304 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=1024 Note that in some OS X versions, all five shared-memory parameters must be set in /etc/sysctl.conf, else the values will be ignored. Beware that recent releases of OS X ignore attempts to set SHMMAX to a value that isn't an exact multiple of 4096. SHMALL is measured in 4 kB pages on this platform. In older OS X versions, you will need to reboot to have changes in the shared memory parameters take effect. As of 10.5 it is possible to change all but SHMMNI on the fly, using sysctl. But it's still best to set up your preferred values via /etc/sysctl.conf, so that the values will be kept across reboots. The file /etc/sysctl.conf is only honored in OS X 10.3.9 and later. If you are running a previous 10.3.x release, you must edit the file /etc/rc and change the values in the following commands: sysctl -w kern.sysv.shmmax sysctl -w kern.sysv.shmmin sysctl -w kern.sysv.shmmni sysctl -w kern.sysv.shmseg sysctl -w kern.sysv.shmall Note that /etc/rc is usually overwritten by OS X system updates, so you should expect to have to redo these edits after each update. In OS X 10.2 and earlier, instead edit these commands in the file /System/Library/StartupItems/SystemTuning/SystemTuning. SCO OpenServer SCO OpenServerIPC configuration In the default configuration, only 512 kB of shared memory per segment is allowed. To increase the setting, first change to the directory /etc/conf/cf.d. To display the current value of SHMMAX, run: ./configure -y SHMMAX To set a new value for SHMMAX, run: ./configure SHMMAX=value where value is the new value you want to use (in bytes). After setting SHMMAX, rebuild the kernel: ./link_unix and reboot. Solaris 2.6 to 2.9 (Solaris 6 to Solaris 9) SolarisIPC configuration The relevant settings can be changed in /etc/system, for example: set shmsys:shminfo_shmmax=0x2000000 set shmsys:shminfo_shmmin=1 set shmsys:shminfo_shmmni=256 set shmsys:shminfo_shmseg=256 set semsys:seminfo_semmap=256 set semsys:seminfo_semmni=512 set semsys:seminfo_semmns=512 set semsys:seminfo_semmsl=32 You need to reboot for the changes to take effect. See also for information on shared memory under older versions of Solaris. Solaris 2.10 (Solaris 10) OpenSolaris In Solaris 10 and OpenSolaris, the default shared memory and semaphore settings are good enough for most PostgreSQL applications. Solaris now defaults to a SHMMAX of one-quarter of system RAM. To further adjust this setting, use a project setting associated with the postgres user. For example, run the following as root: projadd -c "PostgreSQL DB User" -K "project.max-shm-memory=(privileged,8GB,deny)" -U postgres -G postgres user.postgres This command adds the user.postgres project and sets the shared memory maximum for the postgres user to 8GB, and takes effect the next time that user logs in, or when you restart PostgreSQL (not reload). The above assumes that PostgreSQL is run by the postgres user in the postgres group. No server reboot is required. Other recommended kernel setting changes for database servers which will have a large number of connections are: project.max-shm-ids=(priv,32768,deny) project.max-sem-ids=(priv,4096,deny) project.max-msg-ids=(priv,4096,deny) Additionally, if you are running PostgreSQL inside a zone, you may need to raise the zone resource usage limits as well. See "Chapter2: Projects and Tasks" in the Solaris 10 System Administrator's Guide for more information on projects and prctl. UnixWare UnixWareIPC configuration On UnixWare 7, the maximum size for shared memory segments is 512 kB in the default configuration. To display the current value of SHMMAX, run: /etc/conf/bin/idtune -g SHMMAX which displays the current, default, minimum, and maximum values. To set a new value for SHMMAX, run: /etc/conf/bin/idtune SHMMAX value where value is the new value you want to use (in bytes). After setting SHMMAX, rebuild the kernel: /etc/conf/bin/idbuild -B and reboot. Unix-like operating systems enforce various kinds of resource limits that might interfere with the operation of your PostgreSQL server. Of particular importance are limits on the number of processes per user, the number of open files per process, and the amount of memory available to each process. Each of these have a hard and a soft limit. The soft limit is what actually counts but it can be changed by the user up to the hard limit. The hard limit can only be changed by the root user. The system call setrlimit is responsible for setting these parameters. The shell's built-in command ulimit (Bourne shells) or limit (csh) is used to control the resource limits from the command line. On BSD-derived systems the file /etc/login.conf controls the various resource limits set during login. See the operating system documentation for details. The relevant parameters are maxproc, openfiles, and datasize. For example: default:\ ... :datasize-cur=256M:\ :maxproc-cur=256:\ :openfiles-cur=256:\ ... (-cur is the soft limit. Append -max to set the hard limit.) Kernels can also have system-wide limits on some resources. On Linux /proc/sys/fs/file-max determines the maximum number of open files that the kernel will support. It can be changed by writing a different number into the file or by adding an assignment in /etc/sysctl.conf. The maximum limit of files per process is fixed at the time the kernel is compiled; see /usr/src/linux/Documentation/proc.txt for more information. The PostgreSQL server uses one process per connection so you should provide for at least as many processes as allowed connections, in addition to what you need for the rest of your system. This is usually not a problem but if you run several servers on one machine things might get tight. The factory default limit on open files is often set to socially friendly values that allow many users to coexist on a machine without using an inappropriate fraction of the system resources. If you run many servers on a machine this is perhaps what you want, but on dedicated servers you might want to raise this limit. On the other side of the coin, some systems allow individual processes to open large numbers of files; if more than a few processes do so then the system-wide limit can easily be exceeded. If you find this happening, and you do not want to alter the system-wide limit, you can set PostgreSQL's configuration parameter to limit the consumption of open files. In Linux 2.4 and later, the default virtual memory behavior is not optimal for PostgreSQL. Because of the way that the kernel implements memory overcommit, the kernel might terminate the PostgreSQL postmaster (the master server process) if the memory demands of either PostgreSQL or another process cause the system to run out of virtual memory. If this happens, you will see a kernel message that looks like this (consult your system documentation and configuration on where to look for such a message): Out of Memory: Killed process 12345 (postgres). This indicates that the postgres process has been terminated due to memory pressure. Although existing database connections will continue to function normally, no new connections will be accepted. To recover, PostgreSQL will need to be restarted. One way to avoid this problem is to run PostgreSQL on a machine where you can be sure that other processes will not run the machine out of memory. If memory is tight, increasing the swap space of the operating system can help avoid the problem, because the out-of-memory (OOM) killer is invoked only when physical memory and swap space are exhausted. If PostgreSQL itself is the cause of the system running out of memory, you can avoid the problem by changing your configuration. In some cases, it may help to lower memory-related configuration parameters, particularly shared_buffers and work_mem. In other cases, the problem may be caused by allowing too many connections to the database server itself. In many cases, it may be better to reduce max_connections and instead make use of external connection-pooling software. On Linux 2.6 and later, it is possible to modify the kernel's behavior so that it will not overcommit memory. Although this setting will not prevent the OOM killer from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via sysctl: sysctl -w vm.overcommit_memory=2 or placing an equivalent entry in /etc/sysctl.conf. You might also wish to modify the related setting vm.overcommit_ratio. For details see the kernel documentation file Documentation/vm/overcommit-accounting. Another approach, which can be used with or without altering vm.overcommit_memory, is to set the process-specific oom_score_adj value for the postmaster process to -1000, thereby guaranteeing it will not be targeted by the OOM killer. The simplest way to do this is to execute echo -1000 > /proc/self/oom_score_adj in the postmaster's startup script just before invoking the postmaster. Note that this action must be done as root, or it will have no effect; so a root-owned startup script is the easiest place to do it. If you do this, you may also wish to build PostgreSQL with -DLINUX_OOM_SCORE_ADJ=0 added to CPPFLAGS. That will cause postmaster child processes to run with the normal oom_score_adj value of zero, so that the OOM killer can still target them at need. Older Linux kernels do not offer /proc/self/oom_score_adj, but may have a previous version of the same functionality called /proc/self/oom_adj. This works the same except the disable value is -17 not -1000. The corresponding build flag for PostgreSQL is -DLINUX_OOM_ADJ=0. Some vendors' Linux 2.4 kernels are reported to have early versions of the 2.6 overcommit sysctl parameter. However, setting vm.overcommit_memory to 2 on a 2.4 kernel that does not have the relevant code will make things worse, not better. It is recommended that you inspect the actual kernel source code (see the function vm_enough_memory in the file mm/mmap.c) to verify what is supported in your kernel before you try this in a 2.4 installation. The presence of the overcommit-accounting documentation file should not be taken as evidence that the feature is there. If in any doubt, consult a kernel expert or your kernel vendor. shutdown There are several ways to shut down the database server. You control the type of shutdown by sending different signals to the master postgres process. SIGTERMSIGTERM This is the Smart Shutdown mode. After receiving SIGTERM, the server disallows new connections, but lets existing sessions end their work normally. It shuts down only after all of the sessions terminate. If the server is in online backup mode, it additionally waits until online backup mode is no longer active. While backup mode is active, new connections will still be allowed, but only to superusers (this exception allows a superuser to connect to terminate online backup mode). If the server is in recovery when a smart shutdown is requested, recovery and streaming replication will be stopped only after all regular sessions have terminated. SIGINTSIGINT This is the Fast Shutdown mode. The server disallows new connections and sends all existing server processes SIGTERM, which will cause them to abort their current transactions and exit promptly. It then waits for all server processes to exit and finally shuts down. If the server is in online backup mode, backup mode will be terminated, rendering the backup useless. SIGQUITSIGQUIT This is the Immediate Shutdown mode. The master postgres process will send a SIGQUIT to all child processes and exit immediately, without properly shutting itself down. The child processes likewise exit immediately upon receiving SIGQUIT. This will lead to recovery (by replaying the WAL log) upon next start-up. This is recommended only in emergencies. The program provides a convenient interface for sending these signals to shut down the server. Alternatively, you can send the signal directly using kill on non-Windows systems. The PID of the postgres process can be found using the ps program, or from the file postmaster.pid in the data directory. For example, to do a fast shutdown: $ kill -INT `head -1 /usr/local/pgsql/data/postmaster.pid` It is best not to use SIGKILL to shut down the server. Doing so will prevent the server from releasing shared memory and semaphores, which might then have to be done manually before a new server can be started. Furthermore, SIGKILL kills the postgres process without letting it relay the signal to its subprocesses, so it will be necessary to kill the individual subprocesses by hand as well. To terminate an individual session while allowing other sessions to continue, use pg_terminate_backend() (see ) or send a SIGTERM signal to the child process associated with the session. upgrading version compatibility This section discusses how to upgrade your database data from one PostgreSQL release to a newer one. PostgreSQL major versions are represented by the first two digit groups of the version number, e.g., 8.4. PostgreSQL minor versions are represented by the third group of version digits, e.g., 8.4.2 is the second minor release of 8.4. Minor releases never change the internal storage format and are always compatible with earlier and later minor releases of the same major version number, e.g., 8.4.2 is compatible with 8.4, 8.4.1 and 8.4.6. To update between compatible versions, you simply replace the executables while the server is down and restart the server. The data directory remains unchanged — minor upgrades are that simple. For major releases of PostgreSQL, the internal data storage format is subject to change, thus complicating upgrades. The traditional method for moving data to a new major version is to dump and reload the database. Other methods are available, as discussed below. New major versions also typically introduce some user-visible incompatibilities, so application programming changes might be required. All user-visible changes are listed in the release notes (); pay particular attention to the section labeled "Migration". If you are upgrading across several major versions, be sure to read the release notes for each intervening version. Cautious users will want to test their client applications on the new version before switching over fully; therefore, it's often a good idea to set up concurrent installations of old and new versions. When testing a PostgreSQL major upgrade, consider the following categories of possible changes: Administration The capabilities available for administrators to monitor and control the server often change and improve in each major release. SQL Typically this includes new SQL command capabilities and not changes in behavior, unless specifically mentioned in the release notes. Library API Typically libraries like libpq only add new functionality, again unless mentioned in the release notes. System Catalogs System catalog changes usually only affect database management tools. Server C-language API This involves changes in the backend function API, which is written in the C programming language. Such changes affect code that references backend functions deep inside the server. To dump data from one major version of PostgreSQL and reload it in another, you must use pg_dump; file system level backup methods will not work. (There are checks in place that prevent you from using a data directory with an incompatible version of PostgreSQL, so no great harm can be done by trying to start the wrong server version on a data directory.) It is recommended that you use the pg_dump and pg_dumpall programs from the newer version of PostgreSQL, to take advantage of enhancements that might have been made in these programs. Current releases of the dump programs can read data from any server version back to 7.0. These instructions assume that your existing installation is under the /usr/local/pgsql directory, and that the data area is in /usr/local/pgsql/data. Substitute your paths appropriately. If making a backup, make sure that your database is not being updated. This does not affect the integrity of the backup, but the changed data would of course not be included. If necessary, edit the permissions in the file /usr/local/pgsql/data/pg_hba.conf (or equivalent) to disallow access from everyone except you. See for additional information on access control. pg_dumpall use during upgrade To back up your database installation, type: pg_dumpall > outputfile If you need to preserve OIDs (such as when using them as foreign keys), then use the -o option when running pg_dumpall. To make the backup, you can use the pg_dumpall command from the version you are currently running. For best results, however, try to use the pg_dumpall command from PostgreSQL &version;, since this version contains bug fixes and improvements over older versions. While this advice might seem idiosyncratic since you haven't installed the new version yet, it is advisable to follow it if you plan to install the new version in parallel with the old version. In that case you can complete the installation normally and transfer the data later. This will also decrease the downtime. Shut down the old server: pg_ctl stop On systems that have PostgreSQL started at boot time, there is probably a start-up file that will accomplish the same thing. For example, on a Red Hat Linux system one might find that this works: /etc/rc.d/init.d/postgresql stop See for details about starting and stopping the server. If restoring from backup, rename or delete the old installation directory. It is a good idea to rename the directory, rather than delete it, in case you have trouble and need to revert to it. Keep in mind the directory might consume significant disk space. To rename the directory, use a command like this: mv /usr/local/pgsql /usr/local/pgsql.old (Be sure to move the directory as a single unit so relative paths remain unchanged.) Install the new version of PostgreSQL as outlined in .]]> Create a new database cluster if needed. Remember that you must execute these commands while logged in to the special database user account (which you already have if you are upgrading). /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data Restore your previous pg_hba.conf and any postgresql.conf modifications. Start the database server, again using the special database user account: /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data Finally, restore your data from backup with: /usr/local/pgsql/bin/psql -d postgres -f outputfile using the new psql. The least downtime can be achieved by installing the new server in a different directory and running both the old and the new servers in parallel, on different ports. Then you can use something like: pg_dumpall -p 5432 | psql -d postgres -p 5433 to transfer your data. The pg_upgrade module allows an installation to be migrated in-place from one major PostgreSQL version to the next. Upgrades can be performed in minutes. It is also possible to use certain replication methods, such as Slony, to create a standby server with the updated version of PostgreSQL. This is possible because Slony supports replication between different major versions of PostgreSQL. The standby can be on the same computer or a different computer. Once it has synced up with the master server (running the older version of PostgreSQL), you can switch masters and make the standby the master and shut down the older database instance. Such a switch-over results in only several seconds of downtime for an upgrade. server spoofing While the server is running, it is not possible for a malicious user to take the place of the normal database server. However, when the server is down, it is possible for a local user to spoof the normal server by starting their own server. The spoof server could read passwords and queries sent by clients, but could not return any data because the PGDATA directory would still be secure because of directory permissions. Spoofing is possible because any user can start a database server; a client cannot identify an invalid server unless it is specially configured. The simplest way to prevent spoofing for local connections is to use a Unix domain socket directory () that has write permission only for a trusted local user. This prevents a malicious user from creating their own socket file in that directory. If you are concerned that some applications might still reference /tmp for the socket file and hence be vulnerable to spoofing, during operating system startup create a symbolic link /tmp/.s.PGSQL.5432 that points to the relocated socket file. You also might need to modify your /tmp cleanup script to prevent removal of the symbolic link. To prevent spoofing on TCP connections, the best solution is to use SSL certificates and make sure that clients check the server's certificate. To do that, the server must be configured to accept only hostssl connections () and have SSL key and certificate files (). The TCP client must connect using sslmode=verify-ca or verify-full and have the appropriate root certificate file installed (). encryption PostgreSQL offers encryption at several levels, and provides flexibility in protecting data from disclosure due to database server theft, unscrupulous administrators, and insecure networks. Encryption might also be required to secure sensitive data such as medical records or financial transactions. Password Storage Encryption By default, database user passwords are stored as MD5 hashes, so the administrator cannot determine the actual password assigned to the user. If MD5 encryption is used for client authentication, the unencrypted password is never even temporarily present on the server because the client MD5-encrypts it before being sent across the network. Encryption For Specific Columns The module allows certain fields to be stored encrypted. This is useful if only some of the data is sensitive. The client supplies the decryption key and the data is decrypted on the server and then sent to the client. The decrypted data and the decryption key are present on the server for a brief time while it is being decrypted and communicated between the client and server. This presents a brief moment where the data and keys can be intercepted by someone with complete access to the database server, such as the system administrator. Data Partition Encryption On Linux, encryption can be layered on top of a file system using a loopback device. This allows an entire file system partition to be encrypted on disk, and decrypted by the operating system. On FreeBSD, the equivalent facility is called GEOM Based Disk Encryption (gbde), and many other operating systems support this functionality, including Windows. This mechanism prevents unencrypted data from being read from the drives if the drives or the entire computer is stolen. This does not protect against attacks while the file system is mounted, because when mounted, the operating system provides an unencrypted view of the data. However, to mount the file system, you need some way for the encryption key to be passed to the operating system, and sometimes the key is stored somewhere on the host that mounts the disk. Encrypting Passwords Across A Network The MD5 authentication method double-encrypts the password on the client before sending it to the server. It first MD5-encrypts it based on the user name, and then encrypts it based on a random salt sent by the server when the database connection was made. It is this double-encrypted value that is sent over the network to the server. Double-encryption not only prevents the password from being discovered, it also prevents another connection from using the same encrypted password to connect to the database server at a later time. Encrypting Data Across A Network SSL connections encrypt all data sent across the network: the password, the queries, and the data returned. The pg_hba.conf file allows administrators to specify which hosts can use non-encrypted connections (host) and which require SSL-encrypted connections (hostssl). Also, clients can specify that they connect to servers only via SSL. Stunnel or SSH can also be used to encrypt transmissions. SSL Host Authentication It is possible for both the client and server to provide SSL certificates to each other. It takes some extra configuration on each side, but this provides stronger verification of identity than the mere use of passwords. It prevents a computer from pretending to be the server just long enough to read the password sent by the client. It also helps prevent man in the middle attacks where a computer between the client and server pretends to be the server and reads and passes all data between the client and server. Client-Side Encryption If the system administrator for the server's machine cannot be trusted, it is necessary for the client to encrypt the data; this way, unencrypted data never appears on the database server. Data is encrypted on the client before being sent to the server, and database results have to be decrypted on the client before being used. SSL PostgreSQL has native support for using SSL connections to encrypt client/server communications for increased security. This requires that OpenSSL is installed on both client and server systems and that support in PostgreSQL is enabled at build time (see ). With SSL support compiled in, the PostgreSQL server can be started with SSL enabled by setting the parameter to on in postgresql.conf. The server will listen for both normal and SSL connections on the same TCP port, and will negotiate with any connecting client on whether to use SSL. By default, this is at the client's option; see about how to set up the server to require use of SSL for some or all connections. PostgreSQL reads the system-wide OpenSSL configuration file. By default, this file is named openssl.cnf and is located in the directory reported by openssl version -d. This default can be overridden by setting environment variable OPENSSL_CONF to the name of the desired configuration file. OpenSSL supports a wide range of ciphers and authentication algorithms, of varying strength. While a list of ciphers can be specified in the OpenSSL configuration file, you can specify ciphers specifically for use by the database server by modifying in postgresql.conf. It is possible to have authentication without encryption overhead by using NULL-SHA or NULL-MD5 ciphers. However, a man-in-the-middle could read and pass communications between client and server. Also, encryption overhead is minimal compared to the overhead of authentication. For these reasons NULL ciphers are not recommended. To start in SSL mode, files containing the server certificate and private key must exist. By default, these files are expected to be named server.crt and server.key, respectively, in the server's data directory, but other names and locations can be specified using the configuration parameters and . On Unix systems, the permissions on server.key must disallow any access to world or group; achieve this by the command chmod 0600 server.key. If the private key is protected with a passphrase, the server will prompt for the passphrase and will not start until it has been entered. In some cases, the server certificate might be signed by an intermediate certificate authority, rather than one that is directly trusted by clients. To use such a certificate, append the certificate of the signing authority to the server.crt file, then its parent authority's certificate, and so on up to a root authority that is trusted by the clients. The root certificate should be included in every case where server.crt contains more than one certificate. To require the client to supply a trusted certificate, place certificates of the certificate authorities (CAs) you trust in the file root.crt in the data directory, set the parameter in postgresql.conf to root.crt, and set the clientcert parameter to 1 on the appropriate hostssl line(s) in pg_hba.conf. A certificate will then be requested from the client during SSL connection startup. (See for a description of how to set up certificates on the client.) The server will verify that the client's certificate is signed by one of the trusted certificate authorities. Certificate Revocation List (CRL) entries are also checked if the parameter is set. (See for diagrams showing SSL certificate usage.) The clientcert option in pg_hba.conf is available for all authentication methods, but only for rows specified as hostssl. When clientcert is not specified or is set to 0, the server will still verify presented client certificates against its CA list, if one is configured, — but it will not insist that a client certificate be presented. Note that root.crt lists the top-level CAs that are considered trusted for signing client certificates. In principle it need not list the CA that signed the server's certificate, though in most cases that CA would also be trusted for client certificates. If you are setting up client certificates, you may wish to use the cert authentication method, so that the certificates control user authentication as well as providing connection security. See for details. summarizes the files that are relevant to the SSL setup on the server. (The shown file names are default or typical names. The locally configured names could be different.) File Contents Effect ($PGDATA/server.crt) server certificate sent to client to indicate server's identity ($PGDATA/server.key) server private key proves server certificate was sent by the owner; does not indicate certificate owner is trustworthy ($PGDATA/root.crt) trusted certificate authorities checks that client certificate is signed by a trusted certificate authority ($PGDATA/root.crl) certificates revoked by certificate authorities client certificate must not be on this list The files server.key, server.crt, root.crt, and root.crl (or their configured alternative names) are only examined during server start; so you must restart the server for changes in them to take effect. To create a quick self-signed certificate for the server, use the following OpenSSL command: openssl req -new -text -out server.req Fill out the information that openssl asks for. Make sure you enter the local host name as Common Name; the challenge password can be left blank. The program will generate a key that is passphrase protected; it will not accept a passphrase that is less than four characters long. To remove the passphrase (as you must if you want automatic start-up of the server), run the commands: openssl rsa -in privkey.pem -out server.key rm privkey.pem Enter the old passphrase to unlock the existing key. Now do: openssl req -x509 -in server.req -text -key server.key -out server.crt to turn the certificate into a self-signed certificate and to copy the key and certificate to where the server will look for them. Finally do: chmod og-rwx server.key because the server will reject the file if its permissions are more liberal than this. For more details on how to create your server private key and certificate, refer to the OpenSSL documentation. A self-signed certificate can be used for testing, but a certificate signed by a certificate authority (CA) (either one of the global CAs or a local one) should be used in production so that clients can verify the server's identity. If all the clients are local to the organization, using a local CA is recommended. ssh It is possible to use SSH to encrypt the network connection between clients and a PostgreSQL server. Done properly, this provides an adequately secure network connection, even for non-SSL-capable clients. First make sure that an SSH server is running properly on the same machine as the PostgreSQL server and that you can log in using ssh as some user. Then you can establish a secure tunnel with a command like this from the client machine: ssh -L 63333:localhost:5432 joe@foo.com The first number in the -L argument, 63333, is the port number of your end of the tunnel; it can be any unused port. (IANA reserves ports 49152 through 65535 for private use.) The second number, 5432, is the remote end of the tunnel: the port number your server is using. The name or IP address between the port numbers is the host with the database server you are going to connect to, as seen from the host you are logging in to, which is foo.com in this example. In order to connect to the database server using this tunnel, you connect to port 63333 on the local machine: psql -h localhost -p 63333 postgres To the database server it will then look as though you are really user joe on host foo.com connecting to localhost in that context, and it will use whatever authentication procedure was configured for connections from this user and host. Note that the server will not think the connection is SSL-encrypted, since in fact it is not encrypted between the SSH server and the PostgreSQL server. This should not pose any extra security risk as long as they are on the same machine. In order for the tunnel setup to succeed you must be allowed to connect via ssh as joe@foo.com, just as if you had attempted to use ssh to create a terminal session. You could also have set up the port forwarding as ssh -L 63333:foo.com:5432 joe@foo.com but then the database server will see the connection as coming in on its foo.com interface, which is not opened by the default setting listen_addresses = 'localhost'. This is usually not what you want. If you have to hop to the database server via some login host, one possible setup could look like this: ssh -L 63333:db.foo.com:5432 joe@shell.foo.com Note that this way the connection from shell.foo.com to db.foo.com will not be encrypted by the SSH tunnel. SSH offers quite a few configuration possibilities when the network is restricted in various ways. Please refer to the SSH documentation for details. Several other applications exist that can provide secure tunnels using a procedure similar in concept to the one just described. event log event log To register a Windows event log library with the operating system, issue this command: regsvr32 pgsql_library_directory/pgevent.dll This creates registry entries used by the event viewer, under the default event source named PostgreSQL. To specify a different event source name (see ), use the /n and /i options: regsvr32 /n /i:event_source_name pgsql_library_directory/pgevent.dll To unregister the event log library from the operating system, issue this command: regsvr32 /u [/i:event_source_name] pgsql_library_directory/pgevent.dll To enable event logging in the database server, modify to include eventlog in postgresql.conf.