rdelaage/postgresql
1
0
Fork 0
mirror of https://git.postgresql.org/git/postgresql.git synced 2024-09-01 19:37:22 +02:00
Code Issues Packages Projects Releases Wiki Activity
c6f6646bb0
postgresql/doc/src/sgml/ref/pg_receivewal.sgml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

533 lines
19 KiB
Plaintext
Raw Normal View History

Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<!--
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
doc/src/sgml/ref/pg_receivewal.sgml
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
PostgreSQL documentation
-->
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<refentry id="app-pgreceivewal">
<indexterm zone="app-pgreceivewal">
<primary>pg_receivewal</primary>
doc: Improve DocBook XML validity DocBook XML is superficially compatible with DocBook SGML but has a slightly stricter DTD that we have been violating in a few cases. Although XSLT doesn't care whether the document is valid, the style sheets don't necessarily process invalid documents correctly, so we need to work toward fixing this. This first commit moves the indexterms in refentry elements to an allowed position. It has no impact on the output.
2014-02-24 03:25:35 +01:00
</indexterm>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<refmeta>
Add missing <application> tags in application doc <refentrytitle>s Most of them already have this, but some were missing. Author: Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> Discussion: https://www.postgresql.org/message-id/flat/87o8pco34z.fsf%40wibble.ilmari.org
2020-07-10 16:51:29 +02:00
<refentrytitle><application>pg_receivewal</application></refentrytitle>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<manvolnum>1</manvolnum>
<refmiscinfo>Application</refmiscinfo>
</refmeta>
<refnamediv>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<refname>pg_receivewal</refname>
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
<refpurpose>stream write-ahead logs from a <productname>PostgreSQL</productname> server</refpurpose>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<command>pg_receivewal</command>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
<arg rep="repeat"><replaceable>option</replaceable></arg>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</cmdsynopsis>
</refsynopsisdiv>
<refsect1>
doc: Remove line breaks after <title> This creates unnecessary rendering problem risks, and it's inconsistent and gets copied around.
2020-06-07 13:10:18 +02:00
<title>Description</title>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<para>
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
<application>pg_receivewal</application> is used to stream the write-ahead log
from a running <productname>PostgreSQL</productname> cluster. The write-ahead
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
log is streamed using the streaming replication protocol, and is written
to a local directory of files. This directory can be used as the archive
location for doing a restore using point-in-time recovery (see
Convert documentation to DocBook XML Since some preparation work had already been done, the only source changes left were changing empty-element tags like <xref linkend="foo"> to <xref linkend="foo"/>, and changing the DOCTYPE. The source files are still named *.sgml, but they are actually XML files now. Renaming could be considered later. In the build system, the intermediate step to convert from SGML to XML is removed. Everything is build straight from the source files again. The OpenSP (or the old SP) package is no longer needed. The documentation toolchain instructions are updated and are much simpler now. Peter Eisentraut, Alexander Lakhin, Jürgen Purtz
2017-11-23 15:39:47 +01:00
<xref linkend="continuous-archiving"/>).
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
<para>
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
<application>pg_receivewal</application> streams the write-ahead
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
log in real time as it's being generated on the server, and does not wait
Restore archive_command documentation Commit 5ef1eefd76f404ddc59b885d50340e602b70f05f, which added archive_library, purged most mentions of archive_command from the documentation. This is inappropriate, since archive_command is still a feature in use and users will want to see information about it. This restores all the removed mentions and rephrases things so that archive_command and archive_library are presented as alternatives of each other. Reviewed-by: Nathan Bossart <nathandbossart@gmail.com> Discussion: https://www.postgresql.org/message-id/9366d634-a917-85a9-4991-b2a4859edaf9@enterprisedb.com
2022-09-17 11:34:20 +02:00
for segments to complete like <xref linkend="guc-archive-command"/> and
<xref linkend="guc-archive-library"/> do.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
For this reason, it is not necessary to set
Convert documentation to DocBook XML Since some preparation work had already been done, the only source changes left were changing empty-element tags like <xref linkend="foo"> to <xref linkend="foo"/>, and changing the DOCTYPE. The source files are still named *.sgml, but they are actually XML files now. Renaming could be considered later. In the build system, the intermediate step to convert from SGML to XML is removed. Everything is build straight from the source files again. The OpenSP (or the old SP) package is no longer needed. The documentation toolchain instructions are updated and are much simpler now. Peter Eisentraut, Alexander Lakhin, Jürgen Purtz
2017-11-23 15:39:47 +01:00
<xref linkend="guc-archive-timeout"/> when using
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<application>pg_receivewal</application>.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko.
2014-11-17 18:32:48 +01:00
<para>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
Unlike the WAL receiver of a PostgreSQL standby server, <application>pg_receivewal</application>
Order some new options on man pages more sensibly, minor improvements
2015-09-18 02:56:58 +02:00
by default flushes WAL data only when a WAL file is closed.
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
The option <option>--synchronous</option> must be specified to flush WAL data
Doc: Clarify interactions of pg_receivewal with remote_apply Using pg_receivewal with synchronous_commit = remote_apply set in the backend is incompatible if pg_receivewal is a synchronous standby as it never applies WAL, so document this problem and solutions to it. Backpatch to 9.6, where remote_apply has been added. Author: Robert Haas, Jesper Pedersen Reviewed-by: Laurenz Albe, Álvaro Herrera, Michael Paquier Discussion: https://postgr.es/m/1427a2d3-1e51-9335-1931-4f8853d90d5e@redhat.com Backpatch-through: 9.6
2019-07-24 04:25:43 +02:00
in real time. Since <application>pg_receivewal</application> does not
apply WAL, you should not allow it to become a synchronous standby when
<xref linkend="guc-synchronous-commit"/> equals
<literal>remote_apply</literal>. If it does, it will appear to be a
standby that never catches up, and will cause transaction commits to
block. To avoid this, you should either configure an appropriate value
for <xref linkend="guc-synchronous-standby-names"/>, or specify
<varname>application_name</varname> for
<application>pg_receivewal</application> that does not match it, or
change the value of <varname>synchronous_commit</varname> to
something other than <literal>remote_apply</literal>.
Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko.
2014-11-17 18:32:48 +01:00
</para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<para>
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
The write-ahead log is streamed over a regular
doc: Clarify some things on pg_receivexlog reference page
2015-11-19 20:19:04 +01:00
<productname>PostgreSQL</productname> connection and uses the replication
doc: Trim trailing whitespace
2020-06-07 13:24:40 +02:00
protocol. The connection must be made with a user having
Doc: Outline REPLICATION before SUPERUSER privilege The following docs are updated: - High-availaility section - pg_basebackup - pg_receivewal Per the principle of least privilege, we want to encourage users to interact with those areas using roles that have replication rights, but superusers were mentioned first. Author: Daniel Gustafsson Reviewed-by: Fujii Masao, Michael Paquier Discussion: https://postgr.es/m/ECEBD212-7101-41EB-84F3-2F356E4B6401@yesql.se
2020-05-05 07:16:01 +02:00
<literal>REPLICATION</literal> permissions (see
<xref linkend="role-attributes"/>) or a superuser, and
<filename>pg_hba.conf</filename> must permit the replication connection.
The server must also be configured with
<xref linkend="guc-max-wal-senders"/> set high enough to leave at least
one session available for the stream.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
doc: Describe calculation method of streaming start for pg_receivewal The documentation was imprecise about the starting LSN used for WAL streaming if nothing can be found in the local archive directory defined with the pg_receivewal command, so be more talkative on this matter. Extracted from a larger patch by the same author. Author: Ronan Dunklau, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan Backpatch-through: 10
2021-10-23 07:43:03 +02:00
<para>
The starting point of the write-ahead log streaming is calculated when
<application>pg_receivewal</application> starts:
<orderedlist>
<listitem>
<para>
First, scan the directory where the WAL segment files are written and
Fix various typos, grammar and code style in comments and docs This fixes a set of issues that have accumulated over the past months (or years) in various code areas. Most fixes are related to some recent additions, as of the development of v15. Author: Justin Pryzby Discussion: https://postgr.es/m/20220124030001.GQ23027@telsasoft.com
2022-01-25 01:40:04 +01:00
find the newest completed segment file, using as the starting point the
beginning of the next WAL segment file.
doc: Describe calculation method of streaming start for pg_receivewal The documentation was imprecise about the starting LSN used for WAL streaming if nothing can be found in the local archive directory defined with the pg_receivewal command, so be more talkative on this matter. Extracted from a larger patch by the same author. Author: Ronan Dunklau, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan Backpatch-through: 10
2021-10-23 07:43:03 +02:00
</para>
</listitem>
Allow pg_receivewal to stream from a slot's restart LSN Prior to this patch, when running pg_receivewal, the streaming start point would be the current location of the archives if anything is found in the local directory where WAL segments are written, and pg_receivewal would fall back to the current WAL flush location if there are no archives, as of the result of an IDENTIFY_SYSTEM command. If for some reason the WAL files from pg_receivewal were moved, it is better to try a restart where we left at, which is the replication slot's restart_lsn instead of skipping right to the current flush location, to avoid holes in the WAL backed up. This commit changes pg_receivewal to use the following sequence of methods to determine the starting streaming LSN: - Scan the local archives. - Use the slot's restart_lsn, if supported by the backend and if a slot is defined. - Fallback to the current flush LSN as reported by IDENTIFY_SYSTEM. To keep compatibility with older server versions, we only attempt to use READ_REPLICATION_SLOT if the backend version is at least 15, and fallback to the older behavior of streaming from the current flush LSN if the command is not supported. Some TAP tests are added to cover this feature. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan
2021-10-26 02:30:37 +02:00
<listitem>
<para>
doc: Fix grammar in page of pg_receivewal Introduced by f61e1dd. Author: Kyotaro Horiguchi Discussion: https://postgr.es/m/20211026.112304.1962954080884317968.horikyota.ntt@gmail.com
2021-10-26 06:57:18 +02:00
If a starting point cannot be calculated with the previous method,
Allow pg_receivewal to stream from a slot's restart LSN Prior to this patch, when running pg_receivewal, the streaming start point would be the current location of the archives if anything is found in the local directory where WAL segments are written, and pg_receivewal would fall back to the current WAL flush location if there are no archives, as of the result of an IDENTIFY_SYSTEM command. If for some reason the WAL files from pg_receivewal were moved, it is better to try a restart where we left at, which is the replication slot's restart_lsn instead of skipping right to the current flush location, to avoid holes in the WAL backed up. This commit changes pg_receivewal to use the following sequence of methods to determine the starting streaming LSN: - Scan the local archives. - Use the slot's restart_lsn, if supported by the backend and if a slot is defined. - Fallback to the current flush LSN as reported by IDENTIFY_SYSTEM. To keep compatibility with older server versions, we only attempt to use READ_REPLICATION_SLOT if the backend version is at least 15, and fallback to the older behavior of streaming from the current flush LSN if the command is not supported. Some TAP tests are added to cover this feature. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan
2021-10-26 02:30:37 +02:00
and if a replication slot is used, an extra
<command>READ_REPLICATION_SLOT</command> command is issued to retrieve
Fix various typos, grammar and code style in comments and docs This fixes a set of issues that have accumulated over the past months (or years) in various code areas. Most fixes are related to some recent additions, as of the development of v15. Author: Justin Pryzby Discussion: https://postgr.es/m/20220124030001.GQ23027@telsasoft.com
2022-01-25 01:40:04 +01:00
the slot's <literal>restart_lsn</literal> to use as the starting point.
Allow pg_receivewal to stream from a slot's restart LSN Prior to this patch, when running pg_receivewal, the streaming start point would be the current location of the archives if anything is found in the local directory where WAL segments are written, and pg_receivewal would fall back to the current WAL flush location if there are no archives, as of the result of an IDENTIFY_SYSTEM command. If for some reason the WAL files from pg_receivewal were moved, it is better to try a restart where we left at, which is the replication slot's restart_lsn instead of skipping right to the current flush location, to avoid holes in the WAL backed up. This commit changes pg_receivewal to use the following sequence of methods to determine the starting streaming LSN: - Scan the local archives. - Use the slot's restart_lsn, if supported by the backend and if a slot is defined. - Fallback to the current flush LSN as reported by IDENTIFY_SYSTEM. To keep compatibility with older server versions, we only attempt to use READ_REPLICATION_SLOT if the backend version is at least 15, and fallback to the older behavior of streaming from the current flush LSN if the command is not supported. Some TAP tests are added to cover this feature. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan
2021-10-26 02:30:37 +02:00
This option is only available when streaming write-ahead logs from
<productname>PostgreSQL</productname> 15 and up.
</para>
</listitem>
doc: Describe calculation method of streaming start for pg_receivewal The documentation was imprecise about the starting LSN used for WAL streaming if nothing can be found in the local archive directory defined with the pg_receivewal command, so be more talkative on this matter. Extracted from a larger patch by the same author. Author: Ronan Dunklau, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan Backpatch-through: 10
2021-10-23 07:43:03 +02:00
<listitem>
<para>
If a starting point cannot be calculated with the previous method,
the latest WAL flush location is used as reported by the server from
Fix various typos, grammar and code style in comments and docs This fixes a set of issues that have accumulated over the past months (or years) in various code areas. Most fixes are related to some recent additions, as of the development of v15. Author: Justin Pryzby Discussion: https://postgr.es/m/20220124030001.GQ23027@telsasoft.com
2022-01-25 01:40:04 +01:00
an <literal>IDENTIFY_SYSTEM</literal> command.
doc: Describe calculation method of streaming start for pg_receivewal The documentation was imprecise about the starting LSN used for WAL streaming if nothing can be found in the local archive directory defined with the pg_receivewal command, so be more talkative on this matter. Extracted from a larger patch by the same author. Author: Ronan Dunklau, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan Backpatch-through: 10
2021-10-23 07:43:03 +02:00
</para>
</listitem>
</orderedlist>
</para>
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
<para>
If the connection is lost, or if it cannot be initially established,
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
with a non-fatal error, <application>pg_receivewal</application> will
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
retry the connection indefinitely, and reestablish streaming as soon
as possible. To avoid this behavior, use the <literal>-n</literal>
parameter.
</para>
doc: Document pg_receivewal exit behavior and status
2017-08-16 02:26:58 +02:00
<para>
In the absence of fatal errors, <application>pg_receivewal</application>
Handle SIGTERM in pg_receivewal and pg_recvlogical In pg_receivewal, compressed output is only flushed on clean exits. The reason to support SIGTERM as well as SIGINT (which is currently handled) is that pg_receivewal might well be running as a daemon, and systemd's default KillSignal is SIGTERM. Since pg_recvlogical is also supposed to run as a daemon, teach it about SIGTERM as well and update the documentation to match. While in there, change pg_receivewal's time_to_stop to be sig_atomic_t like it is in pg_recvlogical. Author: Christoph Berg <myon@debian.org> Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/Yvo/5No5S0c4EFMj@msg.df7cb.de
2022-09-14 16:32:24 +02:00
will run until terminated by the <systemitem>SIGINT</systemitem>
(<keycombo action="simul"><keycap>Control</keycap><keycap>C</keycap></keycombo>)
or <systemitem>SIGTERM</systemitem> signal.
doc: Document pg_receivewal exit behavior and status
2017-08-16 02:26:58 +02:00
</para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</refsect1>
<refsect1>
<title>Options</title>
<variablelist>
<varlistentry>
<term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
pg_receivexlog: Rename option --dir to --directory getopt_long() allows abbreviating long options, so we might as well give the option the full name, and users can abbreviate it how they like. Do some general polishing of the --help output at the same time.
2012-06-11 23:55:27 +02:00
<term><option>--directory=<replaceable class="parameter">directory</replaceable></option></term>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<listitem>
<para>
Directory to write the output to.
</para>
<para>
This parameter is required.
</para>
</listitem>
</varlistentry>
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
pg_receivewal: Add --endpos option This is primarily useful for making tests of this utility more deterministic, to avoid the complexity of starting pg_receivewal as a deamon in TAP tests. While this is less useful than the equivalent pg_recvlogical option, users can as well use it for example to enforce WAL streaming up to a end-of-backup position, to save only a minimal amount of WAL. Use this new option to stream WAL data in a deterministic way within a new set of TAP tests. Author: Michael Paquier <michael.paquier@gmail.com>
2017-09-11 22:30:50 +02:00
<varlistentry>
<term><option>-E <replaceable>lsn</replaceable></option></term>
<term><option>--endpos=<replaceable>lsn</replaceable></option></term>
<listitem>
<para>
Automatically stop replication and exit with normal exit status 0 when
receiving reaches the specified LSN.
</para>
<para>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
If there is a record with LSN exactly equal to <replaceable>lsn</replaceable>,
pg_receivewal: Add --endpos option This is primarily useful for making tests of this utility more deterministic, to avoid the complexity of starting pg_receivewal as a deamon in TAP tests. While this is less useful than the equivalent pg_recvlogical option, users can as well use it for example to enforce WAL streaming up to a end-of-backup position, to save only a minimal amount of WAL. Use this new option to stream WAL data in a deterministic way within a new set of TAP tests. Author: Michael Paquier <michael.paquier@gmail.com>
2017-09-11 22:30:50 +02:00
the record will be processed.
</para>
</listitem>
</varlistentry>
Optionally don't error out due to preexisting slots in commandline utilities. pg_receivexlog and pg_recvlogical error out when --create-slot is specified and a slot with the same name already exists. In some cases, especially with pg_receivexlog, that's rather annoying and requires additional scripting. Backpatch to 9.5 as slot control functions have newly been added to pg_receivexlog, and there doesn't seem much point leaving it in a less useful state. Discussion: 20150619144755.GG29350@alap3.anarazel.de
2015-07-12 22:06:27 +02:00
<varlistentry>
<term><option>--if-not-exists</option></term>
<listitem>
<para>
Fix typos Alexander Law
2016-07-20 10:39:18 +02:00
Do not error out when <option>--create-slot</option> is specified
Optionally don't error out due to preexisting slots in commandline utilities. pg_receivexlog and pg_recvlogical error out when --create-slot is specified and a slot with the same name already exists. In some cases, especially with pg_receivexlog, that's rather annoying and requires additional scripting. Backpatch to 9.5 as slot control functions have newly been added to pg_receivexlog, and there doesn't seem much point leaving it in a less useful state. Discussion: 20150619144755.GG29350@alap3.anarazel.de
2015-07-12 22:06:27 +02:00
and a slot with the specified name already exists.
</para>
</listitem>
</varlistentry>
doc: Clean up pg_recvlogical reference page This needed a general cleanup of wording, typos, outdated terminology, formatting, and hard-to-understand and borderline incorrect information. Also tweak the pg_receivexlog page a bit to make the two more consistent.
2014-10-18 15:10:12 +02:00
<varlistentry>
<term><option>-n</option></term>
<term><option>--no-loop</option></term>
<listitem>
<para>
Don't loop on connection errors. Instead, exit right away with
an error.
</para>
</listitem>
</varlistentry>
pg_receivewal: Add --no-sync option. Michael Paquier, reviewed by Kuntal Ghosh and by me. I did a little wordsmithing on the documentation, too. Discussion: http://postgr.es/m/CAB7nPqTuXuyEoVKcWcExh_b0uAjgWd_14KfGLrCTccBZ=VA0KA@mail.gmail.com
2017-10-29 08:16:55 +01:00
<varlistentry>
<term><option>--no-sync</option></term>
<listitem>
<para>
This option causes <command>pg_receivewal</command> to not force WAL
data to be flushed to disk. This is faster, but means that a
subsequent operating system crash can leave the WAL segments corrupt.
Generally, this option is useful for testing but should not be used
when doing WAL archiving on a production deployment.
</para>
<para>
This option is incompatible with <literal>--synchronous</literal>.
</para>
</listitem>
</varlistentry>
doc: Clean up pg_recvlogical reference page This needed a general cleanup of wording, typos, outdated terminology, formatting, and hard-to-understand and borderline incorrect information. Also tweak the pg_receivexlog page a bit to make the two more consistent.
2014-10-18 15:10:12 +02:00
<varlistentry>
<term><option>-s <replaceable class="parameter">interval</replaceable></option></term>
<term><option>--status-interval=<replaceable class="parameter">interval</replaceable></option></term>
<listitem>
<para>
Specifies the number of seconds between status packets sent back to the
server. This allows for easier monitoring of the progress from server.
A value of zero disables the periodic status updates completely,
although an update will still be sent when requested by the server, to
avoid timeout disconnect. The default value is 10 seconds.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-S <replaceable>slotname</replaceable></option></term>
<term><option>--slot=<replaceable class="parameter">slotname</replaceable></option></term>
<listitem>
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
Require <application>pg_receivewal</application> to use an existing
Convert documentation to DocBook XML Since some preparation work had already been done, the only source changes left were changing empty-element tags like <xref linkend="foo"> to <xref linkend="foo"/>, and changing the DOCTYPE. The source files are still named *.sgml, but they are actually XML files now. Renaming could be considered later. In the build system, the intermediate step to convert from SGML to XML is removed. Everything is build straight from the source files again. The OpenSP (or the old SP) package is no longer needed. The documentation toolchain instructions are updated and are much simpler now. Peter Eisentraut, Alexander Lakhin, Jürgen Purtz
2017-11-23 15:39:47 +01:00
replication slot (see <xref linkend="streaming-replication-slots"/>).
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
When this option is used, <application>pg_receivewal</application> will report
doc: Clean up pg_recvlogical reference page This needed a general cleanup of wording, typos, outdated terminology, formatting, and hard-to-understand and borderline incorrect information. Also tweak the pg_receivexlog page a bit to make the two more consistent.
2014-10-18 15:10:12 +02:00
a flush position to the server, indicating when each segment has been
synchronized to disk so that the server can remove that segment if it
doc: Clarify some things on pg_receivexlog reference page
2015-11-19 20:19:04 +01:00
is not otherwise needed.
</para>
<para>
When the replication client
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
of <application>pg_receivewal</application> is configured on the
doc: Clarify some things on pg_receivexlog reference page
2015-11-19 20:19:04 +01:00
server as a synchronous standby, then using a replication slot will
report the flush position to the server, but only when a WAL file is
closed. Therefore, that configuration will cause transactions on the
primary to wait for a long time and effectively not work
satisfactorily. The option <literal>--synchronous</literal> (see
below) must be specified in addition to make this work correctly.
doc: Clean up pg_recvlogical reference page This needed a general cleanup of wording, typos, outdated terminology, formatting, and hard-to-understand and borderline incorrect information. Also tweak the pg_receivexlog page a bit to make the two more consistent.
2014-10-18 15:10:12 +02:00
</para>
</listitem>
</varlistentry>
Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko.
2014-11-17 18:32:48 +01:00
<varlistentry>
<term><option>--synchronous</option></term>
<listitem>
<para>
Rephrase the documentation on pg_receivexlog --synchronous option. The old wording talked about a "sync command", meaining fsync(), but it was not very clear.
2015-02-03 09:35:46 +01:00
Flush the WAL data to disk immediately after it has been received. Also
send a status packet back to the server immediately after flushing,
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
regardless of <literal>--status-interval</literal>.
Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko.
2014-11-17 18:32:48 +01:00
</para>
doc: Clarify some things on pg_receivexlog reference page
2015-11-19 20:19:04 +01:00
<para>
This option should be specified if the replication client
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
of <application>pg_receivewal</application> is configured on the
doc: Clarify some things on pg_receivexlog reference page
2015-11-19 20:19:04 +01:00
server as a synchronous standby, to ensure that timely feedback is
sent to the server.
</para>
Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko.
2014-11-17 18:32:48 +01:00
</listitem>
</varlistentry>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<varlistentry>
<term><option>-v</option></term>
<term><option>--verbose</option></term>
<listitem>
<para>
Enables verbose mode.
</para>
</listitem>
</varlistentry>
Add compression support to pg_receivexlog Author: Michael Paquier, review and small changes by me
2017-01-17 12:10:26 +01:00
Rework compression options of pg_receivewal pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04 03:10:31 +01:00
<varlistentry>
Rework compression options of pg_receivewal Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-13 04:09:51 +02:00
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>-Z <replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
Rework compression options of pg_receivewal pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04 03:10:31 +01:00
<listitem>
<para>
Rework compression options of pg_receivewal Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-13 04:09:51 +02:00
Enables compression of write-ahead logs.
Rework compression options of pg_receivewal pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04 03:10:31 +01:00
</para>
<para>
Rework compression options of pg_receivewal Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-13 04:09:51 +02:00
The compression method can be set to <literal>gzip</literal>,
<literal>lz4</literal> (if <productname>PostgreSQL</productname>
was compiled with <option>--with-lz4</option>) or
<literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
form <literal>keyword</literal> or <literal>keyword=value</literal>.
Currently, the only supported keyword is <literal>level</literal>.
Rework compression options of pg_receivewal pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04 03:10:31 +01:00
</para>
Add compression support to pg_receivexlog Author: Michael Paquier, review and small changes by me
2017-01-17 12:10:26 +01:00
<para>
Rework compression options of pg_receivewal Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-13 04:09:51 +02:00
If no compression level is specified, the default compression level
will be used. If only a level is specified without mentioning an
algorithm, <literal>gzip</literal> compression will be used if the
level is greater than 0, and no compression will be used if the level
is 0.
Rework compression options of pg_receivewal pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04 03:10:31 +01:00
</para>
<para>
Rework compression options of pg_receivewal Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-13 04:09:51 +02:00
The suffix <filename>.gz</filename> will automatically be added to
all filenames when using <literal>gzip</literal>, and the suffix
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
Add compression support to pg_receivexlog Author: Michael Paquier, review and small changes by me
2017-01-17 12:10:26 +01:00
</para>
</listitem>
</varlistentry>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</variablelist>
<para>
The following command-line options control the database connection parameters.
<variablelist>
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
<varlistentry>
<term><option>-d <replaceable class="parameter">connstr</replaceable></option></term>
<term><option>--dbname=<replaceable class="parameter">connstr</replaceable></option></term>
<listitem>
<para>
doc: libpq connection options can override command-line flags Reported-by: Alexander Lakhin Discussion: https://postgr.es/m/16486-b9c93d71c02c4907@postgresql.org Backpatch-through: 9.5
2020-10-03 04:19:31 +02:00
Specifies parameters used to connect to the server, as a <link
doc: fix spelling "connction" to "connection" Was wrong in commit 1a9388bd0f. Reported-by: Tom Lane, Justin Pryzby Discussion: https://postgr.es/m/20201102063333.GE22691@telsasoft.com Backpatch-through: 9.5
2020-11-11 01:18:35 +01:00
linkend="libpq-connstring">connection string</link>; these
doc: libpq connection options can override command-line flags Reported-by: Alexander Lakhin Discussion: https://postgr.es/m/16486-b9c93d71c02c4907@postgresql.org Backpatch-through: 9.5
2020-10-03 04:19:31 +02:00
will override any conflicting command line options.
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
</para>
<para>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
The option is called <literal>--dbname</literal> for consistency with other
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
client applications, but because <application>pg_receivewal</application>
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
doesn't connect to any particular database in the cluster, database
name in the connection string will be ignored.
</para>
</listitem>
</varlistentry>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<varlistentry>
<term><option>-h <replaceable class="parameter">host</replaceable></option></term>
<term><option>--host=<replaceable class="parameter">host</replaceable></option></term>
<listitem>
<para>
Specifies the host name of the machine on which the server is
running. If the value begins with a slash, it is used as the
directory for the Unix domain socket. The default is taken
from the <envar>PGHOST</envar> environment variable, if set,
else a Unix domain socket connection is attempted.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-p <replaceable class="parameter">port</replaceable></option></term>
<term><option>--port=<replaceable class="parameter">port</replaceable></option></term>
<listitem>
<para>
Specifies the TCP port or local Unix domain socket file
extension on which the server is listening for connections.
Defaults to the <envar>PGPORT</envar> environment variable, if
set, or a compiled-in default.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-U <replaceable>username</replaceable></option></term>
<term><option>--username=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
<para>
User name to connect as.
</para>
</listitem>
</varlistentry>
<varlistentry>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
<term><option>-w</option></term>
<term><option>--no-password</option></term>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<listitem>
<para>
Never issue a password prompt. If the server requires
password authentication and a password is not available by
other means such as a <filename>.pgpass</filename> file, the
connection attempt will fail. This option can be useful in
batch jobs and scripts where no user is present to enter a
password.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-W</option></term>
<term><option>--password</option></term>
<listitem>
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
Force <application>pg_receivewal</application> to prompt for a
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
password before connecting to a database.
</para>
<para>
This option is never essential, since
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<application>pg_receivewal</application> will automatically prompt
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
for a password if the server demands password authentication.
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
However, <application>pg_receivewal</application> will waste a
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
connection attempt finding out that the server wants a password.
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
In some cases it is worth typing <option>-W</option> to avoid the extra
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
connection attempt.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
Add support for managing physical replication slots to pg_receivexlog. pg_receivexlog already has the capability to use a replication slot to reserve WAL on the upstream node. But the used slot currently has to be created via SQL. To allow using slots directly, without involving SQL, add --create-slot and --drop-slot actions, analogous to the logical slot manipulation support in pg_recvlogical. Author: Michael Paquier Discussion: CABUevEx+zrOHZOQg+dPapNPFRJdsk59b=TSVf30Z71GnFXhQaw@mail.gmail.com
2014-10-06 12:51:37 +02:00
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<application>pg_receivewal</application> can perform one of the two
Add support for managing physical replication slots to pg_receivexlog. pg_receivexlog already has the capability to use a replication slot to reserve WAL on the upstream node. But the used slot currently has to be created via SQL. To allow using slots directly, without involving SQL, add --create-slot and --drop-slot actions, analogous to the logical slot manipulation support in pg_recvlogical. Author: Michael Paquier Discussion: CABUevEx+zrOHZOQg+dPapNPFRJdsk59b=TSVf30Z71GnFXhQaw@mail.gmail.com
2014-10-06 12:51:37 +02:00
following actions in order to control physical replication slots:
<variablelist>
<varlistentry>
<term><option>--create-slot</option></term>
<listitem>
<para>
Create a new physical replication slot with the name specified in
Don't start to stream after pg_receivexlog --create-slot. Immediately starting to stream after --create-slot is inconvenient in a number of situations (e.g. when configuring a slot for use in recovery.conf) and it's easy to just call pg_receivexlog twice in the rest of the cases. Author: Michael Paquier Discussion: CAB7nPqQ9qEtuDiKY3OpNzHcz5iUA+DUX9FcN9K8GUkCZvG7+Ew@mail.gmail.com Backpatch: 9.5, where the option was introduced
2015-08-10 13:28:18 +02:00
<option>--slot</option>, then exit.
Add support for managing physical replication slots to pg_receivexlog. pg_receivexlog already has the capability to use a replication slot to reserve WAL on the upstream node. But the used slot currently has to be created via SQL. To allow using slots directly, without involving SQL, add --create-slot and --drop-slot actions, analogous to the logical slot manipulation support in pg_recvlogical. Author: Michael Paquier Discussion: CABUevEx+zrOHZOQg+dPapNPFRJdsk59b=TSVf30Z71GnFXhQaw@mail.gmail.com
2014-10-06 12:51:37 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>--drop-slot</option></term>
<listitem>
<para>
Drop the replication slot with the name specified in
<option>--slot</option>, then exit.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<para>
Make documentation of --help and --version options more consistent Before, some places didn't document the short options (-? and -V), some documented both, some documented nothing, and they were listed in various orders. Now this is hopefully more consistent and complete.
2012-06-18 01:44:00 +02:00
Other options are also available:
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<variablelist>
<varlistentry>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
<term><option>-V</option></term>
<term><option>--version</option></term>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<listitem>
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
Print the <application>pg_receivewal</application> version and exit.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
</listitem>
</varlistentry>
<varlistentry>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
<term><option>-?</option></term>
<term><option>--help</option></term>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<listitem>
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
Show help about <application>pg_receivewal</application> command line
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
arguments, and exit.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</refsect1>
doc: Document pg_receivewal exit behavior and status
2017-08-16 02:26:58 +02:00
<refsect1>
<title>Exit Status</title>
<para>
<application>pg_receivewal</application> will exit with status 0 when
Handle SIGTERM in pg_receivewal and pg_recvlogical In pg_receivewal, compressed output is only flushed on clean exits. The reason to support SIGTERM as well as SIGINT (which is currently handled) is that pg_receivewal might well be running as a daemon, and systemd's default KillSignal is SIGTERM. Since pg_recvlogical is also supposed to run as a daemon, teach it about SIGTERM as well and update the documentation to match. While in there, change pg_receivewal's time_to_stop to be sig_atomic_t like it is in pg_recvlogical. Author: Christoph Berg <myon@debian.org> Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/Yvo/5No5S0c4EFMj@msg.df7cb.de
2022-09-14 16:32:24 +02:00
terminated by the <systemitem>SIGINT</systemitem> or
<systemitem>SIGTERM</systemitem> signal. (That is the
doc: Document pg_receivewal exit behavior and status
2017-08-16 02:26:58 +02:00
normal way to end it. Hence it is not an error.) For fatal errors or
other signals, the exit status will be nonzero.
</para>
</refsect1>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<refsect1>
<title>Environment</title>
<para>
Don't use SGML empty tags For DocBook XML compatibility, don't use SGML empty tags (</>) anymore, replace by the full tag name. Add a warning option to catch future occurrences. Alexander Lakhin, Jürgen Purtz
2017-10-09 03:44:17 +02:00
This utility, like most other <productname>PostgreSQL</productname> utilities,
uses the environment variables supported by <application>libpq</application>
Convert documentation to DocBook XML Since some preparation work had already been done, the only source changes left were changing empty-element tags like <xref linkend="foo"> to <xref linkend="foo"/>, and changing the DOCTYPE. The source files are still named *.sgml, but they are actually XML files now. Renaming could be considered later. In the build system, the intermediate step to convert from SGML to XML is removed. Everything is build straight from the source files again. The OpenSP (or the old SP) package is no longer needed. The documentation toolchain instructions are updated and are much simpler now. Peter Eisentraut, Alexander Lakhin, Jürgen Purtz
2017-11-23 15:39:47 +01:00
(see <xref linkend="libpq-envars"/>).
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
<para>
The environment variable <envar>PG_COLOR</envar> specifies whether to use
Doc: fix some description of environment variables with frontend tools This addresses a couple of issues in the documentation: - Description of PG_COLOR was missing for some tools (pg_archivecleanup and pg_test_fsync), while the other descriptions had grammar mistakes. - pgbench supports more environment variables: PGUSER, PGHOST and PGPORT. - vacuumlo, oid2name and pgbench support coloring (HEAD only) Author: Michael Paquier Reviewed-by: Fabien Coelho, Daniel Gustafsson, Juan José Santamaría Flecha Discussion: https://postgr.es/m/20200304075418.GJ2593@paquier.xyz Backpatch-through: 12
2020-03-09 02:53:22 +01:00
color in diagnostic messages. Possible values are
<literal>always</literal>, <literal>auto</literal> and
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
<literal>never</literal>.
</para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</refsect1>
<refsect1>
<title>Notes</title>
<para>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
When using <application>pg_receivewal</application> instead of
Restore archive_command documentation Commit 5ef1eefd76f404ddc59b885d50340e602b70f05f, which added archive_library, purged most mentions of archive_command from the documentation. This is inappropriate, since archive_command is still a feature in use and users will want to see information about it. This restores all the removed mentions and rephrases things so that archive_command and archive_library are presented as alternatives of each other. Reviewed-by: Nathan Bossart <nathandbossart@gmail.com> Discussion: https://www.postgresql.org/message-id/9366d634-a917-85a9-4991-b2a4859edaf9@enterprisedb.com
2022-09-17 11:34:20 +02:00
<xref linkend="guc-archive-command"/> or
Allow archiving via loadable modules. Running a shell command for each file to be archived has a lot of overhead and may not offer as much error checking as you want, or the exact semantics that you want. So, offer the option to call a loadable module for each file to be archived, rather than running a shell command. Also, add a 'basic_archive' contrib module as an example implementation that archives to a local directory. Nathan Bossart, with a little bit of kibitzing by me. Discussion: http://postgr.es/m/20220202224433.GA1036711@nathanxps13
2022-02-03 19:57:27 +01:00
<xref linkend="guc-archive-library"/> as the main WAL backup method, it is
doc: Update pg_receivexlog note The old note about how to use pg_receivexlog as an alternative to archive_command was obsoleted by replication slots.
2014-11-08 02:15:22 +01:00
strongly recommended to use replication slots. Otherwise, the server is
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
free to recycle or remove write-ahead log files before they are backed up,
doc: Update pg_receivexlog note The old note about how to use pg_receivexlog as an alternative to archive_command was obsoleted by replication slots.
2014-11-08 02:15:22 +01:00
because it does not have any information, either
Restore archive_command documentation Commit 5ef1eefd76f404ddc59b885d50340e602b70f05f, which added archive_library, purged most mentions of archive_command from the documentation. This is inappropriate, since archive_command is still a feature in use and users will want to see information about it. This restores all the removed mentions and rephrases things so that archive_command and archive_library are presented as alternatives of each other. Reviewed-by: Nathan Bossart <nathandbossart@gmail.com> Discussion: https://www.postgresql.org/message-id/9366d634-a917-85a9-4991-b2a4859edaf9@enterprisedb.com
2022-09-17 11:34:20 +02:00
from <xref linkend="guc-archive-command"/> or
<xref linkend="guc-archive-library"/> or the replication slots, about
doc: Update pg_receivexlog note The old note about how to use pg_receivexlog as an alternative to archive_command was obsoleted by replication slots.
2014-11-08 02:15:22 +01:00
how far the WAL stream has been archived. Note, however, that a
replication slot will fill up the server's disk space if the receiver does
not keep up with fetching the WAL data.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</para>
Allow group access on PGDATA Allow the cluster to be optionally init'd with read access for the group. This means a relatively non-privileged user can perform a backup of the cluster without requiring write privileges, which enhances security. The mode of PGDATA is used to determine whether group permissions are enabled for directory and file creates. This method was chosen as it's simple and works well for the various utilities that write into PGDATA. Changing the mode of PGDATA manually will not automatically change the mode of all the files contained therein. If the user would like to enable group access on an existing cluster then changing the mode of all the existing files will be required. Note that pg_upgrade will automatically change the mode of all migrated files if the new cluster is init'd with the -g option. Tests are included for the backend and all the utilities which operate on the PG data directory to ensure that the correct mode is set based on the data directory permissions. Author: David Steele <david@pgmasters.net> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
<para>
<application>pg_receivewal</application> will preserve group permissions on
the received WAL files if group permissions are enabled on the source
cluster.
</para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</refsect1>
<refsect1>
<title>Examples</title>
<para>
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
To stream the write-ahead log from the server at
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
<literal>mydbserver</literal> and store it in the local directory
<filename>/usr/local/pgsql/archive</filename>:
Tweak markup to avoid extra whitespace in man pages
2012-03-28 19:37:19 +02:00
<screen>
Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 22:23:46 +01:00
<prompt>$</prompt> <userinput>pg_receivewal -h mydbserver -D /usr/local/pgsql/archive</userinput>
Tweak markup to avoid extra whitespace in man pages
2012-03-28 19:37:19 +02:00
</screen></para>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</refsect1>
<refsect1>
<title>See Also</title>
<simplelist type="inline">
Convert documentation to DocBook XML Since some preparation work had already been done, the only source changes left were changing empty-element tags like <xref linkend="foo"> to <xref linkend="foo"/>, and changing the DOCTYPE. The source files are still named *.sgml, but they are actually XML files now. Renaming could be considered later. In the build system, the intermediate step to convert from SGML to XML is removed. Everything is build straight from the source files again. The OpenSP (or the old SP) package is no longer needed. The documentation toolchain instructions are updated and are much simpler now. Peter Eisentraut, Alexander Lakhin, Jürgen Purtz
2017-11-23 15:39:47 +01:00
<member><xref linkend="app-pgbasebackup"/></member>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
</simplelist>
</refsect1>
</refentry>
Reference in New Issue Copy Permalink