postgresql/doc/src/sgml/backup.sgml

<!--
$PostgreSQL: pgsql/doc/src/sgml/backup.sgml,v 2.48 2004/09/30 10:30:10 neilc Exp $
-->
<chapter id="backup">
 <title>Backup and Restore</title>

 <indexterm zone="backup"><primary>backup</></>

 <para>
  As everything that contains valuable data, <productname>PostgreSQL</>
  databases should be backed up regularly. While the procedure is
  essentially simple, it is important to have a basic understanding of
  the underlying techniques and assumptions.
 </para>

 <para>
  There are three fundamentally different approaches to backing up
  <productname>PostgreSQL</> data:
  <itemizedlist>
   <listitem><para><acronym>SQL</> dump</para></listitem>
   <listitem><para>File system level backup</para></listitem>
   <listitem><para>On-line backup</para></listitem>
  </itemizedlist>
  Each has its own strengths and weaknesses.
 </para>

 <sect1 id="backup-dump">
  <title><acronym>SQL</> Dump</title>

  <para>
   The idea behind the SQL-dump method is to generate a text file with SQL
   commands that, when fed back to the server, will recreate the
   database in the same state as it was at the time of the dump.
   <productname>PostgreSQL</> provides the utility program
   <xref linkend="app-pgdump"> for this purpose. The basic usage of this
   command is:
<synopsis>
pg_dump <replaceable class="parameter">dbname</replaceable> &gt; <replaceable class="parameter">outfile</replaceable>
</synopsis>
   As you see, <application>pg_dump</> writes its results to the
   standard output. We will see below how this can be useful.
  </para>

  <para>
   <application>pg_dump</> is a regular <productname>PostgreSQL</>
   client application (albeit a particularly clever one). This means
   that you can do this backup procedure from any remote host that has
   access to the database. But remember that <application>pg_dump</>
   does not operate with special permissions. In particular, you must
   have read access to all tables that you want to back up, so in
   practice you almost always have to be a database superuser.
  </para>

  <para>
   To specify which database server <application>pg_dump</> should
   contact, use the command line options <option>-h
   <replaceable>host</></> and <option>-p <replaceable>port</></>. The
   default host is the local host or whatever your
   <envar>PGHOST</envar> environment variable specifies. Similarly,
   the default port is indicated by the <envar>PGPORT</envar>
   environment variable or, failing that, by the compiled-in default.
   (Conveniently, the server will normally have the same compiled-in
   default.)
  </para>

  <para>
   As any other <productname>PostgreSQL</> client application,
   <application>pg_dump</> will by default connect with the database
   user name that is equal to the current operating system user name. To override
   this, either specify the <option>-U</option> option or set the
   environment variable <envar>PGUSER</envar>. Remember that
   <application>pg_dump</> connections are subject to the normal
   client authentication mechanisms (which are described in <xref
   linkend="client-authentication">).
  </para>

  <para>
   Dumps created by <application>pg_dump</> are internally consistent,
   that is, updates to the database while <application>pg_dump</> is
   running will not be in the dump. <application>pg_dump</> does not
   block other operations on the database while it is working.
   (Exceptions are those operations that need to operate with an
   exclusive lock, such as <command>VACUUM FULL</command>.)
  </para>

  <important>
   <para>
    When your database schema relies on OIDs (for instance as foreign
    keys) you must instruct <application>pg_dump</> to dump the OIDs
    as well. To do this, use the <option>-o</option> command line
    option.  <quote>Large objects</> are not dumped by default,
    either.  See <xref linkend="app-pgdump">'s reference page if you
    use large objects.
   </para>
  </important>

  <sect2 id="backup-dump-restore">
   <title>Restoring the dump</title>

   <para>
    The text files created by <application>pg_dump</> are intended to
    be read in by the <application>psql</application> program. The
    general command form to restore a dump is
<synopsis>
psql <replaceable class="parameter">dbname</replaceable> &lt; <replaceable class="parameter">infile</replaceable>
</synopsis>
    where <replaceable class="parameter">infile</replaceable> is what
    you used as <replaceable class="parameter">outfile</replaceable>
    for the <application>pg_dump</> command. The database <replaceable
    class="parameter">dbname</replaceable> will not be created by this
    command, you must create it yourself from <literal>template0</> before executing
    <application>psql</> (e.g., with <literal>createdb -T template0
    <replaceable class="parameter">dbname</></literal>).
    <application>psql</> supports similar options to <application>pg_dump</> 
    for controlling the database server location and the user name. See
    its reference page for more information.
   </para>

   <para>
    If the objects in the original database were owned by different
    users, then the dump will instruct <application>psql</> to connect
    as each affected user in turn and then create the relevant
    objects. This way the original ownership is preserved. This also
    means, however, that all these users must already exist, and
    furthermore that you must be allowed to connect as each of them.
    It might therefore be necessary to temporarily relax the client
    authentication settings.
   </para>

   <para>
    Once restored, it is wise to run <xref linkend="sql-analyze"
    endterm="sql-analyze-title"> on each database so the optimizer has
    useful statistics. You can also run <command>vacuumdb -a -z</> to
    <command>VACUUM ANALYZE</> all databases; this is equivalent to
    running <command>VACUUM ANALYZE</command> manually.
   </para>

   <para>
    The ability of <application>pg_dump</> and <application>psql</> to
    write to or read from pipes makes it possible to dump a database
    directly from one server to another; for example:
<programlisting>
pg_dump -h <replaceable>host1</> <replaceable>dbname</> | psql -h <replaceable>host2</> <replaceable>dbname</>
</programlisting>
   </para>

   <important>
    <para>
     The dumps produced by <application>pg_dump</> are relative to
     <literal>template0</>. This means that any languages, procedures,
     etc. added to <literal>template1</> will also be dumped by
     <application>pg_dump</>. As a result, when restoring, if you are
     using a customized <literal>template1</>, you must create the
     empty database from <literal>template0</>, as in the example
     above.
    </para>
   </important>

   <para>
    For advice on how to load large amounts of data into
    <productname>PostgreSQL</productname> efficiently, refer to <xref
    linkend="populate">.
   </para>
  </sect2>

  <sect2 id="backup-dump-all">
   <title>Using <application>pg_dumpall</></title>

   <para>
    The above mechanism is cumbersome and inappropriate when backing
    up an entire database cluster. For this reason the <xref
    linkend="app-pg-dumpall"> program is provided.
    <application>pg_dumpall</> backs up each database in a given
    cluster, and also preserves cluster-wide data such as users and
    groups. The basic usage of this command is:
<synopsis>
pg_dumpall &gt; <replaceable>outfile</>
</synopsis>
    The resulting dump can be restored with <application>psql</>:
<synopsis>
psql template1 &lt; <replaceable class="parameter">infile</replaceable>
</synopsis>
    (Actually, you can specify any existing database name to start from,
    but if you are reloading in an empty cluster then <literal>template1</>
    is the only available choice.)  It is always necessary to have
    database superuser access when restoring a <application>pg_dumpall</>
    dump, as that is required to restore the user and group information.
   </para>
  </sect2>

  <sect2 id="backup-dump-large">
   <title>Large Databases</title>

   <para>
    Since <productname>PostgreSQL</productname> allows tables larger
    than the maximum file size on your system, it can be problematic
    to dump such a table to a file, since the resulting file will likely
    be larger than the maximum size allowed by your system. Since
    <application>pg_dump</> can write to the standard output, you can
    just use standard Unix tools to work around this possible problem.
   </para>

   <formalpara>
    <title>Use compressed dumps.</title>
    <para>
     You can use your favorite compression program, for example
     <application>gzip</application>.

<programlisting>
pg_dump <replaceable class="parameter">dbname</replaceable> | gzip &gt; <replaceable class="parameter">filename</replaceable>.gz
</programlisting>

     Reload with

<programlisting>
createdb <replaceable class="parameter">dbname</replaceable>
gunzip -c <replaceable class="parameter">filename</replaceable>.gz | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>

     or

<programlisting>
cat <replaceable class="parameter">filename</replaceable>.gz | gunzip | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>
    </para>
   </formalpara>

   <formalpara>
    <title>Use <command>split</>.</title>
    <para>
     The <command>split</command> command
     allows you to split the output into pieces that are
     acceptable in size to the underlying file system. For example, to
     make chunks of 1 megabyte:

<programlisting>
pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <replaceable class="parameter">filename</replaceable>
</programlisting>

     Reload with

<programlisting>
createdb <replaceable class="parameter">dbname</replaceable>
cat <replaceable class="parameter">filename</replaceable>* | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>
    </para>
   </formalpara>

   <formalpara>
    <title>Use the custom dump format.</title>
    <para>
     If <productname>PostgreSQL</productname> was built on a system with the <application>zlib</> compression library
     installed, the custom dump format will compress data as it writes it
     to the output file. For large databases, this will produce similar dump
     sizes to using <command>gzip</command>, but has the added advantage that the tables can be
     restored selectively. The following command dumps a database using the
     custom dump format:

<programlisting>
pg_dump -Fc <replaceable class="parameter">dbname</replaceable> > <replaceable class="parameter">filename</replaceable>
</programlisting>

     See the <xref linkend="app-pgdump"> and <xref
     linkend="app-pgrestore"> reference pages for details.
    </para>
   </formalpara>

  </sect2>

  <sect2 id="backup-dump-caveats">
   <title>Caveats</title>

   <para>
    For reasons of backward compatibility, <application>pg_dump</>
    does not dump large objects by default.<indexterm><primary>large
    object</primary><secondary>backup</secondary></indexterm> To dump
    large objects you must use either the custom or the tar output
    format, and use the <option>-b</> option in
    <application>pg_dump</>. See the reference pages for details.  The
    directory <filename>contrib/pg_dumplo</> of the
    <productname>PostgreSQL</> source tree also contains a program
    that can dump large objects.
   </para>

   <para>
    Please familiarize yourself with the <xref linkend="app-pgdump">
    reference page.
   </para>
  </sect2>
 </sect1>

 <sect1 id="backup-file">
  <title>File system level backup</title>

  <para>
   An alternative backup strategy is to directly copy the files that
   <productname>PostgreSQL</> uses to store the data in the database. In
   <xref linkend="creating-cluster"> it is explained where these files
   are located, but you have probably found them already if you are
   interested in this method. You can use whatever method you prefer
   for doing usual file system backups, for example

<programlisting>
tar -cf backup.tar /usr/local/pgsql/data
</programlisting>
  </para>

  <para>
   There are two restrictions, however, which make this method
   impractical, or at least inferior to the <application>pg_dump</>
   method:

   <orderedlist>
    <listitem>
     <para>
      The database server <emphasis>must</> be shut down in order to
      get a usable backup. Half-way measures such as disallowing all
      connections will <emphasis>not</emphasis> work
      (mainly because <command>tar</command> and similar tools do not take an
      atomic snapshot of the state of the filesystem at a point in
      time). Information about stopping the server can be found in
      <xref linkend="postmaster-shutdown">.  Needless to say that you
      also need to shut down the server before restoring the data.
     </para>
    </listitem>

    <listitem>
     <para>
      If you have dug into the details of the file system layout of the
      database, you may be tempted to try to back up or restore only certain
      individual tables or databases from their respective files or
      directories. This will <emphasis>not</> work because the
      information contained in these files contains only half the
      truth. The other half is in the commit log files
      <filename>pg_clog/*</filename>, which contain the commit status of
      all transactions. A table file is only usable with this
      information. Of course it is also impossible to restore only a
      table and the associated <filename>pg_clog</filename> data
      because that would render all other tables in the database
      cluster useless.  So file system backups only work for complete
      restoration of an entire database cluster.
     </para>
    </listitem>
   </orderedlist>
  </para>

  <para>
   An alternative file-system backup approach is to make a
   <quote>consistent snapshot</quote> of the data directory, if the
   file system supports that functionality (and you are willing to
   trust that it is implemented correctly).  The typical procedure is
   to make a <quote>frozen snapshot</> of the volume containing the
   database, then copy the whole data directory (not just parts, see
   above) from the snapshot to a backup device, then release the frozen
   snapshot.  This will work even while the database server is running.
   However, a backup created in this way saves
   the database files in a state where the database server was not
   properly shut down; therefore, when you start the database server
   on the backed-up data, it will think the server had crashed
   and replay the WAL log.  This is not a problem, just be aware of
   it (and be sure to include the WAL files in your dump).
  </para>

  <para>
   If your database is spread across multiple volumes (for example,
   data files and WAL log on different disks) there may not be any way
   to obtain exactly-simultaneous frozen snapshots of all the volumes.
   Read your filesystem documentation very carefully before trusting
   to the consistent-snapshot technique in such situations.
  </para>

  <para>
   Note that a file system backup will not necessarily be
   smaller than an SQL dump. On the contrary, it will most likely be
   larger. (<application>pg_dump</application> does not need to dump
   the contents of indexes for example, just the commands to recreate
   them.)
  </para>
 </sect1>

 <sect1 id="backup-online">
  <title>On-line backup and point-in-time recovery</title>

  <indexterm zone="backup">
   <primary>on-line backup</primary>
  </indexterm>

  <indexterm zone="backup">
   <primary>point-in-time recovery</primary>
  </indexterm>

  <para>
   At all times, <productname>PostgreSQL</> maintains a <firstterm>write ahead
   log</> (WAL) that shows details of every change made to the database's data
   files.  This log exists primarily for crash-safety purposes: if the system
   crashes, the database can be restored to consistency by <quote>replaying</>
   the log entries made since the last checkpoint.  However, the existence
   of the log makes it possible to use a third strategy for backing up
   databases: we can combine a filesystem-level backup with backup of the WAL
   files.  If recovery is needed, we restore the backup and then replay from
   the backed-up WAL files to bring the backup up to current time.  This
   approach is notably more complex to administer than either of the previous
   approaches, but it has some significant benefits to offer:
  <itemizedlist>
   <listitem>
    <para>
     We do not need a perfectly consistent backup as the starting point.
     Any internal inconsistency in the backup will be corrected by log
     replay (this is not significantly different from what happens during
     crash recovery).  So we don't need filesystem snapshot capability,
     just <application>tar</> or a similar archiving tool.
    </para>
   </listitem>
   <listitem>
    <para>
     Since we can string together an indefinitely long sequence of WAL files
     for replay, continuous backup can be had simply by continuing to archive
     the WAL files.  This is particularly valuable for large databases, where
     making a full backup may take an unreasonable amount of time.
    </para>
   </listitem>
   <listitem>
    <para>
     There is nothing that says we have to replay the WAL entries all the
     way to the end.  We could stop the replay at any point and have a
     consistent snapshot of the database as it was at that time.  Thus,
     this technique supports <firstterm>point-in-time recovery</>: it is
     possible to restore the database to its state at any time since your base
     backup was taken.
    </para>
   </listitem>
   <listitem>
    <para>
     If we continuously feed the series of WAL files to another machine
     that's been loaded with the same base backup file, we have a <quote>hot
     standby</> system: at any point we can bring up the second machine
     and it will have a nearly-current copy of the database.
    </para>
   </listitem>
  </itemizedlist>
  </para>

  <para>
   As with the plain filesystem-backup technique, this method can only
   support restoration of an entire database cluster, not a subset.
   Also, it requires a lot of archival storage: the base backup is bulky,
   and a busy system will generate many megabytes of WAL traffic that
   have to be archived.  Still, it is the preferred backup technique in
   many situations where high reliability is needed.
  </para>

  <para>
   To recover successfully using an on-line backup, you need a continuous
   sequence of archived WAL files that extends back at least as far as the
   start time of your backup.  So to get started, you should set up and test
   your procedure for archiving WAL files <emphasis>before</> you take your
   first base backup.  Accordingly, we first discuss the mechanics of
   archiving WAL files.
  </para>

  <sect2 id="backup-archiving-wal">
   <title>Setting up WAL archiving</title>

   <para>
    In an abstract sense, a running <productname>PostgreSQL</> system
    produces an indefinitely long sequence of WAL records.  The system
    physically divides this sequence into WAL <firstterm>segment files</>,
    which are normally 16Mb apiece (although the size can be altered when
    building the server).  The segment files are given numeric names that
    reflect their position in the abstract WAL sequence.  When not using WAL
    archiving, the system normally creates just a few segment files and then
    <quote>recycles</> them by renaming no-longer-needed segment files to
    higher segment numbers.  It's assumed that a segment file whose contents
    precede the checkpoint-before-last is no longer of interest and can be
    recycled.
   </para>

   <para>
    When archiving WAL data, we want to capture the contents of each segment
    file once it is filled, and save that data somewhere before the segment
    file is recycled for reuse.  Depending on the application and the
    available hardware, there could be many different ways of <quote>saving
    the data somewhere</>: we could copy the segment files to an NFS-mounted
    directory on another machine, or write them onto a tape drive, or batch
    them together and burn them onto CDs, or something else entirely.  To
    provide the database administrator with as much flexibility as possible,
    <productname>PostgreSQL</> tries not to make any assumptions about how 
    the archiving will be done.  Instead, <productname>PostgreSQL</> lets
    the administrator specify a shell command to be executed to copy a
    completed segment file to wherever it needs to go.  The command could be
    as simple as a <application>cp</>, or it could invoke a complex shell
    script --- it's all up to you.
   </para>

   <para>
    The shell command to use is specified by the <xref
    linkend="guc-archive-command"> configuration parameter, which in practice
    will always be placed in the <filename>postgresql.conf</filename> file.
    In this string,
    any <literal>%p</> is replaced by the absolute path of the file to
    archive, while any <literal>%f</> is replaced by the file name only.
    Write <literal>%%</> if you need to embed an actual <literal>%</>
    character in the command.  The simplest useful command is something
    like
<programlisting>
archive_command = 'cp %p /mnt/server/archivedir/%f'	
</programlisting>
    which will copy archivable WAL segments to the directory
    <literal>/mnt/server/archivedir</>.
   </para>

   <para>
    The archive command will be executed under the ownership of the same
    user that the <productname>PostgreSQL</> server is running as.  Since
    the series of WAL files being archived contains effectively everything
    in your database, you will want to be sure that the archived data is
    protected from prying eyes; for example, archive into a directory that
    does not have group or world read access.
   </para>

   <para>
    It is important that the archive command return zero exit status if and
    only if it succeeded.  Upon getting a zero result,
    <productname>PostgreSQL</> will assume that the WAL segment file has been
    successfully archived, and it may be overwritten with new data very
    soon thereafter.  However, a nonzero status tells
    <productname>PostgreSQL</> that the file was not archived; it will try
    again periodically until it succeeds.
   </para>

   <para>
    Speed of the archiving command is not important, so long as it can keep up
    with the average rate at which your server generates WAL data.  It is okay
    if the archiving process falls a little behind (or even a lot behind, if
    you don't mind the <literal>pg_xlog/</> directory filling up with
    not-yet-archived segment files).
   </para>

   <para>
    If you are concerned about being able to recover right up to the current
    instant, you may want to take additional steps to ensure that the current,
    partially-filled WAL segment is also copied someplace.  This is
    particularly important if your server generates only little WAL traffic
    (or has slack periods where it does so), since it could take a long time
    before a WAL segment file is completely filled and ready to archive.
    One possible way to handle this is to set up a <application>cron</> job
    that periodically (once a minute, perhaps) identifies the current WAL
    segment file and saves it someplace safe.  The combination of the archived
    WAL segments and the saved current segment will then be enough to ensure
    you can always restore to within a minute of current time.  This behavior
    is not presently built into <productname>PostgreSQL</> because we did not
    want to complicate the definition of the <xref
    linkend="guc-archive-command"> by requiring it to keep track of
    successively archived, but different, copies of the same WAL file.
    The <xref linkend="guc-archive-command"> is only invoked on finished
    WAL segments that will not change anymore; and except in the case of
    retrying a failure, it will be called only once for any given file name.
   </para>

   <para>
    In writing your archive command, you should assume that the filenames to
    be archived may be up to 64 characters long and may contain any
    combination of ASCII letters, digits, and dots.  It is not necessary to
    remember the original full path (<literal>%p</>) but it is necessary to
    remember the file name (<literal>%f</>).
   </para>
  </sect2>

  <sect2 id="backup-base-backup">
   <title>Making a Base Backup</title>

   <para>
    The procedure for making a base backup is relatively simple:
  <orderedlist>
   <listitem>
    <para>
     Ensure that WAL archiving is enabled and working.
    </para>
   </listitem>
   <listitem>
    <para>
     Connect to the database as a superuser, and issue the command
<programlisting>
SELECT pg_start_backup('label');
</programlisting>
     where <literal>label</> is any string you want to use to uniquely
     identify this backup operation.  (One good practice is to use the
     full path where you intend to put the backup dump file.)  It does
     not matter which database within the cluster you connect to to issue
     this command.  You can ignore the result returned by the function;
     but if it reports an error, deal with that before proceeding.
    </para>
   </listitem>
   <listitem>
    <para>
     Perform the backup, using any convenient filesystem-backup tool
     such as <application>tar</> or <application>cpio</>.  It is neither
     necessary nor desirable to stop normal operation of the database
     while you do this.
    </para>
   </listitem>
   <listitem>
    <para>
     Again connect to the database as a superuser, and issue the command
<programlisting>
SELECT pg_stop_backup();
</programlisting>
     If this returns successfully, you're done.
    </para>
   </listitem>
  </orderedlist>
   </para>

   <para>
    It is not necessary to be very concerned about the amount of time elapsed
    between <function>pg_start_backup</> and the start of the actual backup,
    nor between the end of the backup and <function>pg_stop_backup</>; a
    few minutes' delay won't hurt anything.  You
    must however be quite sure that these operations are carried out in
    sequence and don't overlap.
   </para>

   <para>
    Be certain that your backup dump includes all of the files underneath
    the database cluster directory (e.g., <literal>/usr/local/pgsql/data</>).
    If you are using tablespaces that do not reside underneath this directory,
    be careful to include them as well (and be sure that your backup dump
    archives symbolic links as links, otherwise the restore will mess up
    your tablespaces).
   </para>

   <para>
    You may, however, omit from the backup dump the files within the
    <literal>pg_xlog/</> subdirectory of the cluster directory.  This
    slight complication is worthwhile because it reduces the risk
    of mistakes when restoring.  This is easy to arrange if
    <literal>pg_xlog/</> is a symbolic link pointing to someplace outside
    the cluster directory, which is a common setup anyway for performance
    reasons.
   </para>

   <para>
    To make use of this backup, you will need to keep around all the WAL
    segment files generated at or after the starting time of the backup.
    To aid you in doing this, the <function>pg_stop_backup</> function
    creates a <firstterm>backup history file</> that is immediately stored
    into the WAL archive area.  This file is named after the first WAL
    segment file that you need to have to make use of the backup.  For
    example, if the starting WAL file is <literal>0000000100001234000055CD</>
    the backup history file will be named something like
    <literal>0000000100001234000055CD.007C9330.backup</>.  (The second part of
    this file name stands for an exact position within the WAL file, and can
    ordinarily be ignored.)  Once you have safely archived the backup dump
    file, you can delete all archived WAL segments with names numerically
    preceding this one.  The backup history file is just a small text file.
    It contains the label string you gave to <function>pg_start_backup</>, as
    well as the starting and ending times of the backup.  If you used the
    label to identify where the associated dump file is kept, then the
    archived history file is enough to tell you which dump file to restore,
    should you need to do so.
   </para>

   <para>
    Since you have to keep around all the archived WAL files back to your
    last full dump, your interval between full dumps would usually be chosen
    based on how much storage you want to expend on archived WAL files.
    You should also consider how long you are prepared to spend recovering,
    if recovery should be necessary --- the system will have to replay all
    those segments, and that could take awhile if it's been a long time
    since the full dump.
   </para>

   <para>
    It's also worth noting that the <function>pg_start_backup</> function
    makes a file named <literal>backup_label</> in the database cluster
    directory, which is then removed again by <function>pg_stop_backup</>.
    This file will of course be archived as a part of your backup dump file.
    The backup label file includes the label string you gave to
    <function>pg_start_backup</>, as well as the time at which
    <function>pg_start_backup</> was run, and the name of the starting WAL
    file.  In case of confusion it will
    therefore be possible to look inside a backup dump file and determine
    exactly which backup session the dump file came from.
   </para>

   <para>
    It is also possible to make a backup dump while the postmaster is
    stopped.  In this case, obviously you can't use
    <function>pg_start_backup</> or <function>pg_stop_backup</>, and
    you will therefore be left to your own devices to keep track of which
    backup dump is which and how far back the associated WAL files go.
    It's generally better to follow the on-line backup procedure above.
   </para>
  </sect2>

  <sect2 id="backup-pitr-recovery">
   <title>Recovering with an On-line Backup</title>

   <para>
    Okay, the worst has happened and you need to recover from your backup.
    Here is the procedure:
  <orderedlist>
   <listitem>
    <para>
     Stop the postmaster, if it's running, and clean out all existing files
     under the cluster data directory and under the root directories of any
     tablespaces you are using.
     (If there are recent, unarchived WAL segment files in
     <literal>pg_xlog/</> that you want to use during restore, move these aside
     instead of removing them.)
    </para>
   </listitem>
   <listitem>
    <para>
     Restore the database files from your backup dump.  Be careful that they
     are restored with the right ownership (the database system user, not
     root!) and with the right permissions.  If you are using tablespaces,
     you may want to verify that the symbolic links in <literal>pg_tblspc/</>
     were correctly restored.
    </para>
   </listitem>
   <listitem>
    <para>
     Remove any files present in <literal>pg_xlog/</>; these came from the
     backup dump and are therefore probably obsolete rather than current.
     If you didn't archive <literal>pg_xlog/</> at all, then re-create it,
     and be sure to re-create the subdirectory
    <literal>pg_xlog/archive_status/</> as well.
    </para>
   </listitem>
   <listitem>
    <para>
     If you had unarchived WAL segment files that you saved aside in step 1,
     copy them into <literal>pg_xlog/</>.  (It's best to copy them, not move
     them back in, so that you still have the unmodified files if the worst
     happens and you have to start over.)
    </para>
   </listitem>
   <listitem>
    <para>
     Create a recovery command file <literal>recovery.conf</> in the cluster
     data directory, as discussed below.  You may also want to temporarily
     modify <literal>pg_hba.conf</> to prevent ordinary users from connecting
     until you are sure the recovery has worked.
    </para>
   </listitem>
   <listitem>
    <para>
     Start the postmaster.  The postmaster will go into recovery mode and
     proceed to read through the archived WAL files it needs.  Upon completion
     of the recovery process, the postmaster will rename
     <literal>recovery.conf</> to <literal>recovery.done</> (to prevent
     accidentally re-entering recovery mode in case of a crash later) and then
     commence normal database operations.
    </para>
   </listitem>
   <listitem>
    <para>
     Inspect the contents of the database to ensure you have recovered to
     where you want to be.  If not, return to step 1.  If all is well,
     let in your users by restoring <literal>pg_hba.conf</> to normal.
    </para>
   </listitem>
  </orderedlist>
   </para>

   <para>
    The key part of all this is to set up a recovery command file
    that describes how you want to recover and how far the recovery
    should run.  You can use <literal>recovery.conf.sample</> (normally
    installed in the installation <literal>share/</> directory) as a
    prototype.  The one thing that you absolutely must specify in
    <literal>recovery.conf</> is the <literal>restore_command</>,
    which tells how to get back archived WAL file segments.  Like
    the <literal>archive_command</>, this is a shell command string.
    It may contain <literal>%f</>,
    which is replaced by the name of the desired log file, and <literal>%p</>,
    which is replaced by the absolute path to copy the log file to.
    Write <literal>%%</> if you need to embed an actual <literal>%</>
    character in the command.  The simplest useful command is something
    like
<programlisting>
restore_command = 'cp /mnt/server/archivedir/%f %p'
</programlisting>
    which will copy previously archived WAL segments from the directory
    <literal>/mnt/server/archivedir</>.  You could of course use something
    much more complicated, perhaps even a shell script that requests the
    operator to mount an appropriate tape.
   </para>

   <para>
    It is important that the command return nonzero exit status on failure.
    The command <emphasis>will</> be asked for log files that are not present
    in the archive; it must return nonzero when so asked.  This is not an
    error condition.  Be aware also that the basename of the <literal>%p</>
    path will be different from <literal>%f</>; do not expect them to be
    interchangeable.
   </para>

   <para>
    WAL segments that cannot be found in the archive will be sought in
    <literal>pg_xlog/</>; this allows use of recent un-archived segments.
    However segments that are available from the archive will be used in
    preference to files in <literal>pg_xlog/</>.  The system will not
    overwrite the existing contents of <literal>pg_xlog/</> when retrieving
    archived files.
   </para>

   <para>
    Normally, recovery will proceed through all available WAL segments,
    thereby restoring the database to current time (or as close as we can
    get given the available WAL segments).  But if you want to recover to
    some previous point in time (say, right before the junior DBA dropped your
    main transaction table), just specify the required stopping point in
    <literal>recovery.conf</>.  You can specify the stop point either by
    date/time or completion of a specific transaction ID.  The stop
    specification can be inclusive or exclusive.  As of this writing 
    only the date/time option is very usable, since there are no tools 
    to help you identify which transaction ID to use.  Keep in mind 
    that while transaction IDs are asigned sequentially at transaction 
    start, transactions can complete in a different numeric order.
   </para>
   <para>
    Note that the stop point must be after the ending time of the backup
    (ie, the time of <function>pg_stop_backup</>).  You cannot use a base
    backup to recover to a time when that backup was still going on.  (To
    recover to such a time, you must go back to your previous base backup
    and roll forward from there.)
   </para>
  </sect2>

  <sect2 id="backup-timelines">
   <title>Timelines</title>

  <indexterm zone="backup">
   <primary>timelines</primary>
  </indexterm>

   <para>
    The ability to restore the database to a previous point in time creates
    some complexities that are akin to science-fiction stories about time
    travel and parallel universes.  In the original history of the database,
    perhaps you dropped a critical table at 5:15PM on Tuesday evening.
    Unfazed, you get out your backup, restore to the point-in-time 5:14PM
    Tuesday evening, and are up and running.  In <emphasis>this</> history of
    the database universe, you never dropped the table at all.  But suppose
    you later realize this wasn't such a great idea after all, and would like
    to return to some later point in the original history?  You won't be able
    to if, while your database was up-and-running, it overwrote some of the
    sequence of WAL segment files that led up to the time you now wish you
    could get back to.  So you really want to distinguish the series of
    WAL records generated after you've done a point-in-time recovery from
    those that were generated in the original database history.
   </para>

   <para>
    To deal with these problems, <productname>PostgreSQL</> has a notion
    of <firstterm>timelines</>.  Each time you recover to a point-in-time
    earlier than the end of the WAL sequence, a new timeline is created
    to identify the series of WAL records generated after that recovery.
    (If recovery proceeds all the way to the end of WAL, however, we do not
    start a new timeline: we just extend the existing one.)  The timeline
    ID number is part of WAL segment file names, and so a new timeline does
    not overwrite the WAL data generated by previous timelines.  It is
    in fact possible to archive many different timelines.  While that might
    seem like a useless feature, it's often a lifesaver.  Consider the
    situation where you aren't quite sure what point-in-time to recover to,
    and so have to do several point-in-time recoveries by trial and error
    until you find the best place to branch off from the old history.  Without
    timelines this process would soon generate an unmanageable mess.  With
    timelines, you can recover to <emphasis>any</> prior state, including
    states in timeline branches that you later abandoned.
   </para>

   <para>
    Each time a new timeline is created, <productname>PostgreSQL</> creates
    a <quote>timeline history</> file that shows which timeline it branched
    off from and when.  These history files are necessary to allow the system
    to pick the right WAL segment files when recovering from an archive that
    contains multiple timelines.  Therefore, they are archived into the WAL
    archive area just like WAL segment files.  The history files are just
    small text files, so it's cheap and appropriate to keep them around
    indefinitely (unlike the segment files which are large).  You can, if
    you like, add comments to a history file to make your own notes about
    how and why this particular timeline came to be.  Such comments will be
    especially valuable when you have a thicket of different timelines as
    a result of experimentation.
   </para>

   <para>
    The default behavior of recovery is to recover along the same timeline
    that was current when the base backup was taken.  If you want to recover
    into some child timeline (that is, you want to return to some state that
    was itself generated after a recovery attempt), you need to specify the
    target timeline in <literal>recovery.conf</>.  You cannot recover into
    timelines that branched off earlier than the base backup.
   </para>
  </sect2>

  <sect2 id="backup-online-caveats">
   <title>Caveats</title>

   <para>
    At this writing, there are several limitations of the on-line backup
    technique.  These will probably be fixed in future releases:

  <itemizedlist>
   <listitem>
    <para>
     Operations on non-btree indexes (hash, R-tree, and GiST indexes) are
     not presently WAL-logged, so replay will not update these index types.
     The recommended workaround, if you use any non-btree indexes, is to
     manually <command>REINDEX</> each such index after completing a
     recovery operation.
    </para>
   </listitem>
  </itemizedlist>
   </para>

   <para>
    It should also be noted that the present <acronym>WAL</acronym>
    format is extremely bulky since it includes many disk page
    snapshots.  This is appropriate for crash recovery purposes,
    since we may need to fix partially-written disk pages.  It is not
    necessary to store so many page copies for PITR operations, however.
    An area for future development is to compress archived WAL data by
    removing unnecessary page copies.
   </para>
  </sect2>
 </sect1>

 <sect1 id="migration">
  <title>Migration Between Releases</title>

  <indexterm zone="migration">
   <primary>upgrading</primary>
  </indexterm>

  <indexterm zone="migration">
   <primary>version</primary>
   <secondary>compatibility</secondary>
  </indexterm>

  <para>
   As a general rule, the internal data storage format is subject to
   change between major releases of <productname>PostgreSQL</> (where
   the number after the first dot changes). This does not apply to
   different minor releases under the same major release (where the
   number after the second dot changes); these always have compatible
   storage formats. For example, releases 7.0.1, 7.1.2, and 7.2 are
   not compatible, whereas 7.1.1 and 7.1.2 are. When you update
   between compatible versions, you can simply replace the executables
   and reuse the data area on disk. Otherwise you need to <quote>back
   up</> your data and <quote>restore</> it on the new server, using
   <application>pg_dump</>. There are checks in place that prevent you
   from using a data area with an incompatible version of
   <productname>PostgreSQL</productname>, so no harm can be done by
   confusing these things. It is recommended that you use the
   <application>pg_dump</> program from the newer version of
   <productname>PostgreSQL</> to take advantage of any enhancements in
   <application>pg_dump</> that may have been made. The precise
   installation procedure is not the subject of this section; those
   details are in <xref linkend="installation">.
  </para>

  <para>
   The least downtime can be achieved by installing the new server in
   a different directory and running both the old and the new servers
   in parallel, on different ports. Then you can use something like

<programlisting>
pg_dumpall -p 5432 | psql -d template1 -p 6543
</programlisting>

   to transfer your data.  Or use an intermediate file if you want.
   Then you can shut down the old server and start the new server at
   the port the old one was running at. You should make sure that the
   database is not updated after you run <application>pg_dumpall</>,
   otherwise you will obviously lose that data. See <xref
   linkend="client-authentication"> for information on how to prohibit
   access. In practice you probably want to test your client
   applications on the new setup before switching over.
  </para>

  <para>
   If you cannot or do not want to run two servers in parallel you can
   do the backup step before installing the new version, bring down
   the server, move the old version out of the way, install the new
   version, start the new server, restore the data. For example:

<programlisting>
pg_dumpall > backup
pg_ctl stop
mv /usr/local/pgsql /usr/local/pgsql.old
cd ~/postgresql-&version;
gmake install
initdb -D /usr/local/pgsql/data
postmaster -D /usr/local/pgsql/data
psql template1 < backup
</programlisting>

   See <xref linkend="runtime"> about ways to start and stop the
   server and other details. The installation instructions will advise
   you of strategic places to perform these steps.
  </para>

  <para>
   You will always need a SQL dump (<application>pg_dump</> dump) for
   migrating to a new release.  Filesystem-level backups (including
   on-line backups) will not work, for the same reason that you can't
   just do the update in-place: the file formats won't necessarily be
   compatible across major releases.
  </para>

  <note>
   <para>
    When you <quote>move the old installation out of the way</quote>
    it is no longer perfectly usable. Some parts of the installation
    contain information about where the other parts are located. This
    is usually not a big problem but if you plan on using two
    installations in parallel for a while you should assign them
    different installation directories at build time.
   </para>
  </note>
 </sect1>
</chapter>

<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-tabs-mode:nil
sgml-indent-data:t
sgml-parent-document:nil
sgml-default-dtd-file:"./reference.ced"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/share/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->