rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/bin/pg_basebackup/pg_basebackup.c

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

2662 lines
67 KiB
C
Raw Normal View History

Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*-------------------------------------------------------------------------
*
* pg_basebackup.c - receive a base backup using streaming replication protocol
*
* Author: Magnus Hagander <magnus@hagander.net>
*
Update copyright for 2021 Backpatch-through: 9.5
2021-01-02 19:06:25 +01:00
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*
* IDENTIFICATION
* src/bin/pg_basebackup/pg_basebackup.c
*-------------------------------------------------------------------------
*/
Make xlog_internal.h includable in frontend context. This makes unnecessary the ugly hack used to #include postgres.h in pg_basebackup. Based on Alvaro Herrera's patch
2012-12-13 13:59:13 +01:00
#include "postgres_fe.h"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#include <unistd.h>
#include <dirent.h>
Add missing header declarations for pg_basebackup and pg_{dump,restore} This fixes two compilation failures caused by 6f164e6. Interesting to see that missing <limits.h> dies not fail in Linux or even Windows. On MacOS, it fails, though. Per various buildfarm members.
2021-07-24 12:05:14 +02:00
#include <limits.h>
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#include <sys/stat.h>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include <sys/wait.h>
Add missing include, required on some platforms Noted by the buildfarm and Andres Freund
2014-02-12 20:04:13 +01:00
#include <signal.h>
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
#include <time.h>
Include <sys/select.h> where needed <sys/select.h> is required by POSIX.1-2001 to get the prototype of select(2), but nearly no systems enforce that because older standards let you get away with including some other headers. Recent OpenBSD hacking has removed that frail touch of friendliness, however, which broke some compiles; fix all the way back to 9.1 by adding the required standard. Only vacuumdb.c was reported to fail, but it seems easier to fix the whole lot in a fell swoop. Per bug #14334 by Sean Farrell.
2016-09-27 06:05:21 +02:00
#ifdef HAVE_SYS_SELECT_H
#include <sys/select.h>
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
Make WAL segment size configurable at initdb time. For performance reasons a larger segment size than the default 16MB can be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier to write scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk less frequently. But at the same time large segment size are disadvantageous for smaller databases. So far the segment size had to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load. Therefore change it to a initdb time setting. This includes a breaking changes to the xlogreader.h API, which now requires the current segment size to be configured. For that and similar reasons a number of binaries had to be taught how to recognize the current segment size. Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-20 07:03:48 +02:00
#include "access/xlog_internal.h"
Refactor dir/file permissions Consolidate directory and file create permissions for tools which work with the PG data directory by adding a new module (common/file_perm.c) that contains variables (pg_file_create_mode, pg_dir_create_mode) and constants to initialize them (0600 for files and 0700 for directories). Convert mkdir() calls in the backend to MakePGDirectory() if the original call used default permissions (always the case for regular PG directories). Add tests to make sure permissions in PGDATA are set correctly by the tools which modify the PG data directory. Authors: David Steele <david@pgmasters.net>, Adam Brightwell <adam.brightwell@crunchydata.com> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
#include "common/file_perm.h"
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
#include "common/file_utils.h"
Move logging.h and logging.c from src/fe_utils/ to src/common/. The original placement of this module in src/fe_utils/ is ill-considered, because several src/common/ modules have dependencies on it, meaning that libpgcommon and libpgfeutils now have mutual dependencies. That makes it pointless to have distinct libraries at all. The intended design is that libpgcommon is lower-level than libpgfeutils, so only dependencies from the latter to the former are acceptable. We already have the precedent that fe_memutils and a couple of other modules in src/common/ are frontend-only, so it's not stretching anything out of whack to treat logging.c as a frontend-only module in src/common/. To the extent that such modules help provide a common frontend/backend environment for the rest of common/ to use, it's a reasonable design. (logging.c does not yet provide an ereport() emulation, but one can dream.) Hence, move these files over, and revert basically all of the build-system changes made by commit cc8d41511. There are no places that need to grow new dependencies on libpgcommon, further reinforcing the idea that this is the right solution. Discussion: https://postgr.es/m/a912ffff-f6e4-778a-c86a-cf5c47a12933@2ndquadrant.com
2019-05-14 20:19:49 +02:00
#include "common/logging.h"
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
#include "common/string.h"
Unify parsing logic for command-line integer options Most of the integer options for command-line binaries now make use of a single routine able to do the job, fixing issues with the detection of sloppy values caused for example by the use of atoi(), that fails on strings beginning with numerical characters with junk trailing characters. This commit cuts down the number of strings requiring translation by 26 per my count, switching the code to have two error types for invalid and out-of-range values instead. Much more could be done here, with float or even int64 options, but int32 was the most appealing case as it is possible to rely on strtol() to do the job reliably. Note that there are some exceptions for now, like pg_ctl or pg_upgrade that use their own logging logic. A couple of negative TAP tests required some adjustments for the new errors generated. pg_dump and pg_restore tracked the maximum number of parallel jobs within the option parsing. The code is refactored a bit to track that in the code dedicated to parallelism instead. Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: David Rowley, Álvaro Herrera Discussion: https://postgr.es/m/CALj2ACXqdG9WhqVoJ9zYf-iZt7sgK7Szv5USs=he6NnWQ2ofTA@mail.gmail.com
2021-07-24 11:35:03 +02:00
#include "fe_utils/option_utils.h"
Split out recovery confing-writing code from pg_basebackup ... into a new file, fe_utils/recovery_gen.c. This can later be used by pg_rewind. Authors: Paul Guo, Jimmy Yih, Ashwin Agrawal. A few tweaks by Álvaro Herrera Reviewed-by: Michaël Paquier Discussion: https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
2019-09-25 19:35:24 +02:00
#include "fe_utils/recovery_gen.h"
Retire escapeConnectionParameter(). It is redundant with appendConnStrVal(), which became an extern function in commit 41f18f021a0882eccbeca62e2ed4b66c6b96e9c9. This changes the handling of out-of-memory and of certain inputs for which quoting is optional, but pg_basebackup has no need for unusual treatment thereof.
2016-08-22 04:05:57 +02:00
#include "fe_utils/string_utils.h"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#include "getopt_long.h"
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
#include "libpq-fe.h"
#include "pgtar.h"
#include "pgtime.h"
Make the order of the header file includes consistent in non-backend modules. Similar to commit 7e735035f2, this commit makes the order of header file inclusion consistent for non-backend modules. In passing, fix the case where we were using angle brackets (<>) for the local module includes instead of quotes (""). Author: Vignesh C Reviewed-by: Amit Kapila Discussion: https://postgr.es/m/CALDaNm2Sznv8RR6Ex-iJO6xAdsxgWhCoETkaYX=+9DW3q0QCfA@mail.gmail.com
2019-10-23 06:08:53 +02:00
#include "pqexpbuffer.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include "receivelog.h"
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
#include "replication/basebackup.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include "streamutil.h"
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
#define ERRCODE_DATA_CORRUPTED "XX001"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
typedef struct TablespaceListCell
{
struct TablespaceListCell *next;
char old_dir[MAXPGPATH];
char new_dir[MAXPGPATH];
} TablespaceListCell;
typedef struct TablespaceList
{
TablespaceListCell *head;
TablespaceListCell *tail;
} TablespaceList;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
typedef struct WriteTarState
{
int tablespacenum;
char filename[MAXPGPATH];
FILE *tarfile;
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char tarhdr[TAR_BLOCK_SIZE];
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
bool basetablespace;
bool in_tarhdr;
bool skip_file;
bool is_recovery_guc_supported;
bool is_postgresql_auto_conf;
bool found_postgresql_auto_conf;
int file_padding_len;
size_t tarhdrsz;
pgoff_t filesz;
#ifdef HAVE_LIBZ
gzFile ztarfile;
#endif
} WriteTarState;
typedef struct UnpackTarState
{
int tablespacenum;
char current_path[MAXPGPATH];
char filename[MAXPGPATH];
const char *mapped_tblspc_path;
pgoff_t current_len_left;
int current_padding;
FILE *file;
} UnpackTarState;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
typedef struct WriteManifestState
{
char filename[MAXPGPATH];
FILE *file;
} WriteManifestState;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
typedef void (*WriteDataCallback) (size_t nbytes, char *buf,
void *callback_data);
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/*
* pg_xlog has been renamed to pg_wal in version 10. This version number
* should be compared with PQserverVersion().
*/
#define MINIMUM_VERSION_FOR_PG_WAL 100000
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/*
* Temporary replication slots are supported from version 10.
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
Disable silently generation of manifests with servers <= 12 in pg_basebackup Since 0d8c9c1, pg_basebackup would generate an error if connected to a backend version older than 12 where backup manifests are not supported. Avoiding this error is possible by using the --no-manifest option. This error handling could be confusing for some users, where patching a backup script that interacts with multiple backend versions would cause the addition of --no-manifest to potentially not generate a backup manifest even for Postgres 13 and newer versions. As we want to encourage the use of backup manifests as much as possible, this commit silently disables manifests where not supported, instead of generating an error. While on it, rework a bit the code to make it more consistent with the surroundings when generating the BASE_BACKUP command. Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro Herrera, Kyotaro Horiguchi, Tom Lane, David Steele, and me. Author: Michael Paquier Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz
2020-04-16 06:57:07 +02:00
/*
* Backup manifests are supported from version 13.
*/
#define MINIMUM_VERSION_FOR_MANIFESTS 130000
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
/*
* Different ways to include WAL
*/
typedef enum
{
NO_WAL,
FETCH_WAL,
STREAM_WAL
} IncludeWal;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Global options */
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static char *basedir = NULL;
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
static TablespaceList tablespace_dirs = {NULL, NULL};
Remove useless empty string initializations This coding style probably stems from the days of shell scripts. Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-08-31 04:28:36 +02:00
static char *xlog_dir = NULL;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static char format = 'p'; /* p(lain)/t(ar) */
static char *label = "pg_basebackup base backup";
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool noclean = false;
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
static bool checksum_failure = false;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static bool showprogress = false;
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
static bool estimatesize = true;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static int verbose = 0;
static int compresslevel = 0;
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
static IncludeWal includewal = STREAM_WAL;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
static bool do_sync = true;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
static pg_time_t last_progress_report = 0;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
static int32 maxrate = 0; /* no limit by default */
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
static char *replication_slot = NULL;
static bool temp_replication_slot = true;
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
static bool create_slot = false;
static bool no_slot = false;
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
static bool verify_checksums = true;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
static bool manifest = true;
static bool manifest_force_encode = false;
static char *manifest_checksums = NULL;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool success = false;
static bool made_new_pgdata = false;
static bool found_existing_pgdata = false;
static bool made_new_xlogdir = false;
static bool found_existing_xlogdir = false;
static bool made_tablespace_dirs = false;
static bool found_tablespace_dirs = false;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Progress counters */
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
static uint64 totalsize_kb;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
static uint64 totaldone;
static int tablespacecount;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Pipe to communicate with background wal receiver process */
#ifndef WIN32
static int bgpipe[2] = {-1, -1};
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Handle to child process */
static pid_t bgchild = -1;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool in_log_streamer = false;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* End position for xlog streaming, empty string if unknown yet */
static XLogRecPtr xlogendptr;
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#ifndef WIN32
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
static int has_xlogendptr = 0;
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#else
static volatile LONG has_xlogendptr = 0;
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
/* Contents of configuration file to be generated */
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
static PQExpBuffer recoveryconfcontents = NULL;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Function headers */
static void usage(void);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
static void progress_report(int tablespacenum, const char *filename, bool force,
bool finished);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
static void ReceiveTarAndUnpackCopyChunk(size_t r, char *copybuf,
void *callback_data);
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
void *callback_data);
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
fix up a couple non-prototypes of the form foo() to be foo(void) -- found using -Wstrict-prototypes
2011-03-27 22:31:41 +02:00
static void BaseBackup(void);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
static const char *get_tablespace_mapping(const char *dir);
static void tablespace_list_append(const char *arg);
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static void
cleanup_directories_atexit(void)
{
if (success || in_log_streamer)
return;
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
if (!noclean && !checksum_failure)
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
{
if (made_new_pgdata)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("removing data directory \"%s\"", basedir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (!rmtree(basedir, true))
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("failed to remove data directory");
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
else if (found_existing_pgdata)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("removing contents of data directory \"%s\"", basedir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (!rmtree(basedir, false))
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("failed to remove contents of data directory");
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
if (made_new_xlogdir)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("removing WAL directory \"%s\"", xlog_dir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (!rmtree(xlog_dir, true))
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("failed to remove WAL directory");
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
else if (found_existing_xlogdir)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("removing contents of WAL directory \"%s\"", xlog_dir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (!rmtree(xlog_dir, false))
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("failed to remove contents of WAL directory");
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
}
else
{
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
if ((made_new_pgdata || found_existing_pgdata) && !checksum_failure)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("data directory \"%s\" not removed at user's request", basedir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (made_new_xlogdir || found_existing_xlogdir)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("WAL directory \"%s\" not removed at user's request", xlog_dir);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
if ((made_tablespace_dirs || found_tablespace_dirs) && !checksum_failure)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("changes to tablespace directories will not be undone");
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
}
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
static void
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
disconnect_atexit(void)
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
{
if (conn != NULL)
PQfinish(conn);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
}
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
#ifndef WIN32
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
/*
* On windows, our background thread dies along with the process. But on
* Unix, if we have started a subprocess, we want to kill it off so it
* doesn't remain running trying to stream data.
*/
static void
kill_bgchild_atexit(void)
{
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
if (bgchild > 0)
kill(bgchild, SIGTERM);
}
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
#endif
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
/*
* Split argument into old_dir and new_dir and append to tablespace mapping
* list.
*/
static void
tablespace_list_append(const char *arg)
{
TablespaceListCell *cell = (TablespaceListCell *) pg_malloc0(sizeof(TablespaceListCell));
char *dst;
char *dst_ptr;
const char *arg_ptr;
dst_ptr = dst = cell->old_dir;
for (arg_ptr = arg; *arg_ptr; arg_ptr++)
{
if (dst_ptr - dst >= MAXPGPATH)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("directory name too long");
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
if (*arg_ptr == '\\' && *(arg_ptr + 1) == '=')
; /* skip backslash escaping = */
else if (*arg_ptr == '=' && (arg_ptr == arg || *(arg_ptr - 1) != '\\'))
{
if (*cell->new_dir)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("multiple \"=\" signs in tablespace mapping");
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
else
dst = dst_ptr = cell->new_dir;
}
else
*dst_ptr++ = *arg_ptr;
}
if (!*cell->old_dir || !*cell->new_dir)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("invalid tablespace mapping format \"%s\", must be \"OLDDIR=NEWDIR\"", arg);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
/*
* This check isn't absolutely necessary. But all tablespaces are created
* with absolute directories, so specifying a non-absolute path here would
* just never match, possibly confusing users. It's also good to be
* consistent with the new_dir check.
*/
if (!is_absolute_path(cell->old_dir))
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("old directory is not an absolute path in tablespace mapping: %s",
cell->old_dir);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
if (!is_absolute_path(cell->new_dir))
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("new directory is not an absolute path in tablespace mapping: %s",
cell->new_dir);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
pg_basebackup: Fix comparison handling of tablespace mappings on Windows A candidate path needs to be canonicalized before being checked against the mappings, because the mappings are also canonicalized. This is especially relevant on Windows Reported-by: nb <nbedxp@gmail.com> Author: Michael Paquier <michael.paquier@gmail.com> Reviewed-by: Ashutosh Sharma <ashu.coek88@gmail.com>
2017-11-01 15:20:05 +01:00
/*
* Comparisons done with these values should involve similarly
* canonicalized path values. This is particularly sensitive on Windows
* where path values may not necessarily use Unix slashes.
*/
pg_basebackup: canonicalize old and new tablespace paths This avoids problems with double-slash-specified paths. Patch by Ian Barwick
2015-04-29 02:12:10 +02:00
canonicalize_path(cell->old_dir);
canonicalize_path(cell->new_dir);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
if (tablespace_dirs.tail)
tablespace_dirs.tail->next = cell;
else
tablespace_dirs.head = cell;
tablespace_dirs.tail = cell;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifdef HAVE_LIBZ
static const char *
Fix incorrect uses of gzFile gzFile is already a pointer, so code like gzFile *handle = gzopen(...) is wrong. This used to pass silently because gzFile used to be defined as void*, and you can assign a void* to a void**. But somewhere between zlib versions 1.2.3.4 and 1.2.6, the definition of gzFile was changed to struct gzFile_s *, and with that new definition this usage causes compiler warnings. So remove all those extra pointer decorations. There is a related issue in pg_backup_archiver.h, where FILE *FH; /* General purpose file handle */ is used throughout pg_dump as sometimes a real FILE* and sometimes a gzFile handle, which also causes warnings now. This is not yet fixed here, because it might need more code restructuring.
2012-03-02 21:30:01 +01:00
get_gz_error(gzFile gzf)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
int errnum;
const char *errmsg;
errmsg = gzerror(gzf, &errnum);
if (errnum == Z_ERRNO)
return strerror(errno);
else
return errmsg;
}
#endif
static void
usage(void)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
printf(_("%s takes a base backup of a running PostgreSQL server.\n\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions controlling the output:\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -D, --pgdata=DIRECTORY receive base backup into directory\n"));
printf(_(" -F, --format=p|t output format (plain (default), tar)\n"));
Revert changes to pg_basebackup and pg_waldump usage() code. Partially revert commit c079673dcb7f210617c9fc1470e6bf166d8a2971. There were complaints that splitting switch descriptions would complicate translation efforts. There are probably ways to resolve the formatting problem without doing that, but undo it while we're discussing.
2017-05-17 19:04:03 +02:00
printf(_(" -r, --max-rate=RATE maximum transfer rate to transfer data directory\n"
" (in kB/s, or use suffix \"k\" or \"M\")\n"));
printf(_(" -R, --write-recovery-conf\n"
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
" write configuration for replication\n"));
Revert changes to pg_basebackup and pg_waldump usage() code. Partially revert commit c079673dcb7f210617c9fc1470e6bf166d8a2971. There were complaints that splitting switch descriptions would complicate translation efforts. There are probably ways to resolve the formatting problem without doing that, but undo it while we're discussing.
2017-05-17 19:04:03 +02:00
printf(_(" -T, --tablespace-mapping=OLDDIR=NEWDIR\n"
" relocate tablespace in OLDDIR to NEWDIR\n"));
Sort pg_basebackup options better The --slot option somehow ended up under options controlling the output, and some other options were in a nonsensical place or were not moved after recent renamings, so tidy all that up a bit.
2017-09-26 17:58:22 +02:00
printf(_(" --waldir=WALDIR location for the write-ahead log directory\n"));
Revert changes to pg_basebackup and pg_waldump usage() code. Partially revert commit c079673dcb7f210617c9fc1470e6bf166d8a2971. There were complaints that splitting switch descriptions would complicate translation efforts. There are probably ways to resolve the formatting problem without doing that, but undo it while we're discussing.
2017-05-17 19:04:03 +02:00
printf(_(" -X, --wal-method=none|fetch|stream\n"
" include required WAL files with specified method\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -z, --gzip compress tar output\n"));
printf(_(" -Z, --compress=0-9 compress tar output with given compression level\n"));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
printf(_("\nGeneral options:\n"));
Revert changes to pg_basebackup and pg_waldump usage() code. Partially revert commit c079673dcb7f210617c9fc1470e6bf166d8a2971. There were complaints that splitting switch descriptions would complicate translation efforts. There are probably ways to resolve the formatting problem without doing that, but undo it while we're discussing.
2017-05-17 19:04:03 +02:00
printf(_(" -c, --checkpoint=fast|spread\n"
" set fast or spread checkpointing\n"));
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
printf(_(" -C, --create-slot create replication slot\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -l, --label=LABEL set backup label\n"));
initdb pg_basebackup: Rename --noxxx options to --no-xxx --noclean and --nosync were the only options spelled without a hyphen, so change this for consistency with other options. The options in pg_basebackup have not been in a release, so we just rename them. For initdb, we retain the old variants. Vik Fearing and me
2016-10-19 18:00:00 +02:00
printf(_(" -n, --no-clean do not clean up after errors\n"));
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -P, --progress show progress information\n"));
Sort pg_basebackup options better The --slot option somehow ended up under options controlling the output, and some other options were in a nonsensical place or were not moved after recent renamings, so tidy all that up a bit.
2017-09-26 17:58:22 +02:00
printf(_(" -S, --slot=SLOTNAME replication slot to use\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
Put new command-line options into alphabetical order in help output
2020-05-01 11:49:52 +02:00
printf(_(" --manifest-checksums=SHA{224,256,384,512}|CRC32C|NONE\n"
" use algorithm for manifest checksums\n"));
printf(_(" --manifest-force-encode\n"
" hex encode all file names in manifest\n"));
printf(_(" --no-estimate-size do not estimate backup size in server side\n"));
printf(_(" --no-manifest suppress generation of backup manifest\n"));
pg_basebackup: Remove short option -k -k meant --no-verify-checksums, which is the opposite of what initdb uses -k for. After discussion, a short option does not seem necessary, so just keep the long option. Discussion: https://www.postgresql.org/message-id/flat/d510f8aa-19e1-d06e-7630-ad27f7441d68%402ndquadrant.com
2018-05-21 16:01:49 +02:00
printf(_(" --no-slot prevent creation of temporary replication slot\n"));
printf(_(" --no-verify-checksums\n"
" do not verify checksums\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -?, --help show this help, then exit\n"));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
printf(_("\nConnection options:\n"));
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
printf(_(" -d, --dbname=CONNSTR connection string\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port number\n"));
Revert changes to pg_basebackup and pg_waldump usage() code. Partially revert commit c079673dcb7f210617c9fc1470e6bf166d8a2971. There were complaints that splitting switch descriptions would complicate translation efforts. There are probably ways to resolve the formatting problem without doing that, but undo it while we're discussing.
2017-05-17 19:04:03 +02:00
printf(_(" -s, --status-interval=INTERVAL\n"
" time between status packets sent to server (in seconds)\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -U, --username=NAME connect as specified database user\n"));
printf(_(" -w, --no-password never prompt for password\n"));
printf(_(" -W, --password force password prompt (should happen automatically)\n"));
Refer to bug report address by symbol rather than hardcoding Use the PACKAGE_BUGREPORT macro that is created by Autoconf for referring to the bug reporting address rather than hardcoding it everywhere. This makes it easier to change the address and it reduces translation work. Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Discussion: https://www.postgresql.org/message-id/flat/8d389c5f-7fb5-8e48-9a4a-68cec44786fa%402ndquadrant.com
2020-02-28 08:54:49 +01:00
printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
Add PostgreSQL home page to --help output Per emerging standard in GNU programs and elsewhere. Autoconf already has support for specifying a home page, so we can just that. Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Discussion: https://www.postgresql.org/message-id/flat/8d389c5f-7fb5-8e48-9a4a-68cec44786fa%402ndquadrant.com
2020-02-28 08:54:49 +01:00
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
Fix base backup streaming xlog from standby When backing up from a standby server, the backup process will not automatically switch xlog segment. So we must accept a partially transferred xlog file in this case, but rename it into position anyway. In passing, merge the two callbacks for segment end and stop stream into a single callback, since their implementations were close to identical, and rename this callback to reflect that it stops streaming rather than continues it. Patch by Magnus Hagander, review by Fujii Masao
2012-05-25 11:36:22 +02:00
* Called in the background process every time data is received.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
* On Unix, we check to see if there is any data on our pipe
* (which would mean we have a stop position), and if it is, check if
* it is time to stop.
* On Windows, we are in a single process, so we can just check if it's
* time to stop.
*/
static bool
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
if (!has_xlogendptr)
{
#ifndef WIN32
fd_set fds;
struct timeval tv;
int r;
/*
* Don't have the end pointer yet - check our pipe to see if it has
* been sent yet.
*/
FD_ZERO(&fds);
FD_SET(bgpipe[0], &fds);
MemSet(&tv, 0, sizeof(tv));
r = select(bgpipe[0] + 1, &fds, NULL, NULL, &tv);
if (r == 1)
{
char xlogend[64];
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
uint32 hi,
lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
MemSet(xlogend, 0, sizeof(xlogend));
Correct off-by-one when reading from pipe In pg_basebackup.c:reached_end_position(), we're reading from an internal pipe with our own background process but we're possibly reading more bytes than will actually fit into our buffer due to an off-by-one error. As we're reading from an internal pipe there's no real risk here, but it's good form to not depend on such convenient arrangements. Bug spotted by the Coverity scanner. Back-patch to 9.2 where this showed up.
2013-07-15 16:42:27 +02:00
r = read(bgpipe[0], xlogend, sizeof(xlogend) - 1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (r < 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not read from ready pipe: %m");
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
exit(1);
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (sscanf(xlogend, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not parse write-ahead log location \"%s\"",
xlogend);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
exit(1);
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
xlogendptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
has_xlogendptr = 1;
/*
* Fall through to check if we've reached the point further
* already.
*/
}
else
{
/*
* No data received on the pipe means we don't know the end
* position yet - so just say it's not time to stop yet.
*/
return false;
}
#else
/*
* On win32, has_xlogendptr is set by the main thread, so if it's not
* set here, we just go back and wait until it shows up.
*/
return false;
#endif
}
/*
* At this point we have an end pointer, so compare it to the current
* position to figure out if it's time to stop.
*/
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (segendpos >= xlogendptr)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
return true;
/*
* Have end pointer, but haven't reached it yet - so tell the caller to
* keep streaming.
*/
return false;
}
typedef struct
{
PGconn *bgconn;
XLogRecPtr startptr;
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
char xlog[MAXPGPATH]; /* directory or tarfile depending on mode */
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
char *sysidentifier;
int timeline;
} logstreamer_param;
static int
LogStreamerMain(logstreamer_param *param)
{
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
StreamCtl stream;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
in_log_streamer = true;
Fix order of MemSet arguments Noted by Tomas Vondra
2016-03-13 13:11:06 +01:00
MemSet(&stream, 0, sizeof(stream));
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.startpos = param->startptr;
stream.timeline = param->timeline;
stream.sysidentifier = param->sysidentifier;
stream.stream_stop = reached_end_position;
Avoid slow shutdown of pg_basebackup. pg_basebackup's child process did not pay any attention to the pipe from its parent while waiting for input from the source server. If no server data was arriving, it would only wake up and check the pipe every standby_message_timeout or so. This creates a problem since the parent process might determine and send the desired stop position only after the server has reached end-of-WAL and stopped sending data. In the src/test/recovery regression tests, the timing is repeatably such that it takes nearly 10 seconds for the child process to realize that it should shut down. It's not clear how often that would happen in real-world cases, but it sure seems like a bug --- and if the user turns off standby_message_timeout or sets it very large, the delay could be a lot worse. To fix, expand the StreamCtl API to allow the pipe input FD to be passed down to the low-level wait routine, and watch both sockets when sleeping. (Note: AFAICS this issue doesn't affect the Windows port, since it doesn't rely on a pipe to transfer the stop position to the child thread.) Discussion: https://postgr.es/m/6456.1493263884@sss.pgh.pa.us
2017-04-28 00:27:02 +02:00
#ifndef WIN32
stream.stop_socket = bgpipe[0];
#else
stream.stop_socket = PGINVALID_SOCKET;
#endif
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.standby_message_timeout = standby_message_timeout;
stream.synchronous = false;
Delay fsyncs of pg_basebackup until the end of backup Since the addition of fsync requests in bc34223 to make base backup data consistent on disk once pg_basebackup finishes, each tablespace tar file is individually flushed once completed, with an additional flush of the parent directory when the base backup finishes. While holding a connection to the server, a fsync request taking a long time may cause a failure of the base backup, which is annoying for any integration. A recent example of breakage can involve tcp_user_timeout, but wal_sender_timeout can cause similar problems. While reviewing the code, there was a second issue causing too many fsync requests to be done for the same WAL data. As recursive fsyncs are done at the end of the backup for both the plain and tar formats from the base target directory where everything is written, it is fine to disable fsyncs when fetching or streaming WAL. Reported-by: Ryohei Takahashi Author: Michael Paquier Reviewed-by: Ryohei Takahashi Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com Backpatch-through: 10
2019-09-04 06:21:11 +02:00
/* fsync happens at the end of pg_basebackup for all data */
stream.do_sync = false;
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.mark_done = true;
stream.partial_suffix = NULL;
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
stream.replication_slot = replication_slot;
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (format == 'p')
Delay fsyncs of pg_basebackup until the end of backup Since the addition of fsync requests in bc34223 to make base backup data consistent on disk once pg_basebackup finishes, each tablespace tar file is individually flushed once completed, with an additional flush of the parent directory when the base backup finishes. While holding a connection to the server, a fsync request taking a long time may cause a failure of the base backup, which is annoying for any integration. A recent example of breakage can involve tcp_user_timeout, but wal_sender_timeout can cause similar problems. While reviewing the code, there was a second issue causing too many fsync requests to be done for the same WAL data. As recursive fsyncs are done at the end of the backup for both the plain and tar formats from the base target directory where everything is written, it is fine to disable fsyncs when fetching or streaming WAL. Reported-by: Ryohei Takahashi Author: Michael Paquier Reviewed-by: Ryohei Takahashi Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com Backpatch-through: 10
2019-09-04 06:21:11 +02:00
stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
stream.do_sync);
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
else
Delay fsyncs of pg_basebackup until the end of backup Since the addition of fsync requests in bc34223 to make base backup data consistent on disk once pg_basebackup finishes, each tablespace tar file is individually flushed once completed, with an additional flush of the parent directory when the base backup finishes. While holding a connection to the server, a fsync request taking a long time may cause a failure of the base backup, which is annoying for any integration. A recent example of breakage can involve tcp_user_timeout, but wal_sender_timeout can cause similar problems. While reviewing the code, there was a second issue causing too many fsync requests to be done for the same WAL data. As recursive fsyncs are done at the end of the backup for both the plain and tar formats from the base target directory where everything is written, it is fine to disable fsyncs when fetching or streaming WAL. Reported-by: Ryohei Takahashi Author: Michael Paquier Reviewed-by: Ryohei Takahashi Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com Backpatch-through: 10
2019-09-04 06:21:11 +02:00
stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
stream.do_sync);
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
if (!ReceiveXlogStream(param->bgconn, &stream))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Any errors will already have been reported in the function process,
* but we need to tell the parent that we didn't shutdown in a nice
* way.
*/
return 1;
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (!stream.walmethod->finish())
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not finish writing WAL files: %m");
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
return 1;
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
PQfinish(param->bgconn);
Free walmethods before exiting Not strictly necessary since we quite after, but could become important in the future if we do restarts etc. Michael Paquier with nitpicking from me
2016-10-25 18:57:56 +02:00
if (format == 'p')
FreeWalDirectoryMethod();
else
FreeWalTarMethod();
pg_free(stream.walmethod);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
return 0;
}
/*
* Initiate background process for receiving xlog during the backup.
* The background stream will use its own database connection so we can
* stream the logfile in parallel with the backups.
*/
static void
StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
{
logstreamer_param *param;
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
uint32 hi,
lo;
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
char statusdir[MAXPGPATH];
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
param = pg_malloc0(sizeof(logstreamer_param));
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
param->timeline = timeline;
param->sysidentifier = sysidentifier;
/* Convert the starting position */
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (sscanf(startpos, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not parse write-ahead log location \"%s\"",
startpos);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
param->startptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Round off to even segment position */
Make WAL segment size configurable at initdb time. For performance reasons a larger segment size than the default 16MB can be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier to write scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk less frequently. But at the same time large segment size are disadvantageous for smaller databases. So far the segment size had to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load. Therefore change it to a initdb time setting. This includes a breaking changes to the xlogreader.h API, which now requires the current segment size to be configured. For that and similar reasons a number of binaries had to be taught how to recognize the current segment size. Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-20 07:03:48 +02:00
param->startptr -= XLogSegmentOffset(param->startptr, WalSegSz);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#ifndef WIN32
/* Create our background pipe */
Remove now redundant pgpipe code.
2012-03-29 05:24:07 +02:00
if (pipe(bgpipe) < 0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create pipe for background process: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
#endif
/* Get a second connection */
param->bgconn = GetConnection();
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
if (!param->bgconn)
/* Error message already written in GetConnection() */
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/* In post-10 cluster, pg_xlog has been renamed to pg_wal */
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
snprintf(param->xlog, sizeof(param->xlog), "%s/%s",
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/* Temporary replication slots are only supported in 10 and newer */
if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS)
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
temp_replication_slot = false;
/*
* Create replication slot if requested
*/
if (temp_replication_slot && !replication_slot)
replication_slot = psprintf("pg_basebackup_%d", (int) PQbackendPID(param->bgconn));
if (temp_replication_slot || create_slot)
{
if (!CreateReplicationSlot(param->bgconn, replication_slot, NULL,
Allow enabling two-phase option via replication protocol. Extend the replication command CREATE_REPLICATION_SLOT to support the TWO_PHASE option. This will allow decoding commands like PREPARE TRANSACTION, COMMIT PREPARED and ROLLBACK PREPARED for slots created with this option. The decoding of the transaction happens at prepare command. This patch also adds support of two-phase in pg_recvlogical via a new option --two-phase. This option will also be used by future patches that allow streaming of transactions at prepare time for built-in logical replication. With this, the out-of-core logical replication solutions can enable replication of two-phase transactions via replication protocol. Author: Ajin Cherian Reviewed-By: Jeff Davis, Vignesh C, Amit Kapila Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru https://postgr.es/m/64b9f783c6e125f18f88fbc0c0234e34e71d8639.camel@j-davis.com
2021-06-30 05:15:47 +02:00
temp_replication_slot, true, true, false, false))
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
if (verbose)
{
if (temp_replication_slot)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("created temporary replication slot \"%s\"",
replication_slot);
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
else
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("created replication slot \"%s\"",
replication_slot);
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
}
}
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (format == 'p')
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
{
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
/*
* Create pg_wal/archive_status or pg_xlog/archive_status (and thus
* pg_wal or pg_xlog) depending on the target server so we can write
* to basedir/pg_wal or basedir/pg_xlog as the directory entry in the
* tar file may arrive later.
*/
snprintf(statusdir, sizeof(statusdir), "%s/%s/archive_status",
basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
Refactor dir/file permissions Consolidate directory and file create permissions for tools which work with the PG data directory by adding a new module (common/file_perm.c) that contains variables (pg_file_create_mode, pg_dir_create_mode) and constants to initialize them (0600 for files and 0700 for directories). Convert mkdir() calls in the backend to MakePGDirectory() if the original call used default permissions (always the case for regular PG directories). Add tests to make sure permissions in PGDATA are set correctly by the tools which modify the PG data directory. Authors: David Steele <david@pgmasters.net>, Adam Brightwell <adam.brightwell@crunchydata.com> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
if (pg_mkdir_p(statusdir, pg_dir_create_mode) != 0 && errno != EEXIST)
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create directory \"%s\": %m", statusdir);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
}
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Start a child process and tell it to start streaming. On Unix, this is
* a fork(). On Windows, we create a thread.
*/
#ifndef WIN32
bgchild = fork();
if (bgchild == 0)
{
/* in child process */
exit(LogStreamerMain(param));
}
else if (bgchild < 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create background process: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
/*
* Else we are in the parent process and all is well.
*/
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
atexit(kill_bgchild_atexit);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#else /* WIN32 */
bgchild = _beginthreadex(NULL, 0, (void *) LogStreamerMain, param, 0, NULL);
if (bgchild == 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create background thread: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
#endif
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Verify that the given directory exists and is empty. If it does not
* exist, it is created. If it exists but is not empty, an error will
Fix typo in comment Michael Paquier
2017-05-15 11:08:02 +02:00
* be given and the process ended.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
switch (pg_check_dir(dirname))
{
case 0:
/*
* Does not exist, so create
*/
Refactor dir/file permissions Consolidate directory and file create permissions for tools which work with the PG data directory by adding a new module (common/file_perm.c) that contains variables (pg_file_create_mode, pg_dir_create_mode) and constants to initialize them (0600 for files and 0700 for directories). Convert mkdir() calls in the backend to MakePGDirectory() if the original call used default permissions (always the case for regular PG directories). Add tests to make sure permissions in PGDATA are set correctly by the tools which modify the PG data directory. Authors: David Steele <david@pgmasters.net>, Adam Brightwell <adam.brightwell@crunchydata.com> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
if (pg_mkdir_p(dirname, pg_dir_create_mode) == -1)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create directory \"%s\": %m", dirname);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (created)
*created = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return;
case 1:
/*
* Exists, empty
*/
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (found)
*found = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return;
case 2:
Warn about initdb using mount-points Add code to detect and warn about trying to initdb or create pg_xlog on mount points.
2013-02-17 00:52:50 +01:00
case 3:
case 4:
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Exists, not empty
*/
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("directory \"%s\" exists but is not empty", dirname);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case -1:
/*
* Access problem
*/
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not access directory \"%s\": %m", dirname);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
/*
* Print a progress report based on the global variables. If verbose output
* is enabled, also print the current file name.
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
*
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
* Progress report is written at maximum once per second, unless the force
* parameter is set to true.
*
* If finished is set to true, this is the last progress report. The cursor
* is moved to the next line.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(int tablespacenum, const char *filename,
bool force, bool finished)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
int percent;
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
char totaldone_str[32];
char totalsize_str[32];
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
pg_time_t now;
if (!showprogress)
return;
now = time(NULL);
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
if (now == last_progress_report && !force && !finished)
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
return; /* Max once per second */
last_progress_report = now;
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
percent = totalsize_kb ? (int) ((totaldone / 1024) * 100 / totalsize_kb) : 0;
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
Adjust total size in pg_basebackup progress report when reality changes When streaming including WAL, the size estimate will always be incorrect, since we don't know how much WAL is included. To make sure the output doesn't look completely unreasonable, this patch increases the total size whenever we go past the estimate, to make sure we never go above 100%.
2011-08-16 16:56:47 +02:00
/*
Fix pg_bsd_indent bug where newlines were not being trimmed from typedef lines. Update pg_bsd_indent required version to 1.1 (and update ftp site). Problem reported by Magnus.
2011-10-26 23:23:33 +02:00
* Avoid overflowing past 100% or the full size. This may make the total
* size number change as we approach the end of the backup (the estimate
* will always be wrong if WAL is included), but that's better than having
* the done column be bigger than the total.
Adjust total size in pg_basebackup progress report when reality changes When streaming including WAL, the size estimate will always be incorrect, since we don't know how much WAL is included. To make sure the output doesn't look completely unreasonable, this patch increases the total size whenever we go past the estimate, to make sure we never go above 100%.
2011-08-16 16:56:47 +02:00
*/
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
if (percent > 100)
percent = 100;
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
if (totaldone / 1024 > totalsize_kb)
totalsize_kb = totaldone / 1024;
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
/*
Fix pg_bsd_indent bug where newlines were not being trimmed from typedef lines. Update pg_bsd_indent required version to 1.1 (and update ftp site). Problem reported by Magnus.
2011-10-26 23:23:33 +02:00
* Separate step to keep platform-dependent format code out of
* translatable strings. And we only test for INT64_FORMAT availability
* in snprintf, not fprintf.
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
*/
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
snprintf(totaldone_str, sizeof(totaldone_str), INT64_FORMAT,
totaldone / 1024);
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
snprintf(totalsize_str, sizeof(totalsize_str), INT64_FORMAT, totalsize_kb);
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
#define VERBOSE_FILENAME_LENGTH 35
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (verbose)
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
{
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (!filename)
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
/*
* No filename given, so clear the status line (used for last
* call)
*/
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (100%%), %d/%d tablespace %*s",
"%*s/%s kB (100%%), %d/%d tablespaces %*s",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str,
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
tablespacenum, tablespacecount,
VERBOSE_FILENAME_LENGTH + 5, "");
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
else
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
{
bool truncate = (strlen(filename) > VERBOSE_FILENAME_LENGTH);
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (%d%%), %d/%d tablespace (%s%-*.*s)",
"%*s/%s kB (%d%%), %d/%d tablespaces (%s%-*.*s)",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str, percent,
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
tablespacenum, tablespacecount,
/* Prefix with "..." if we do leading truncation */
truncate ? "..." : "",
truncate ? VERBOSE_FILENAME_LENGTH - 3 : VERBOSE_FILENAME_LENGTH,
truncate ? VERBOSE_FILENAME_LENGTH - 3 : VERBOSE_FILENAME_LENGTH,
/* Truncate filename at beginning if it's too long */
truncate ? filename + strlen(filename) - VERBOSE_FILENAME_LENGTH + 3 : filename);
}
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (%d%%), %d/%d tablespace",
"%*s/%s kB (%d%%), %d/%d tablespaces",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str, percent,
tablespacenum, tablespacecount);
Move \r out of translatable strings The translation tools are very unhappy about seeing \r in translatable strings, so move it to a separate fprintf call.
2011-08-17 09:52:35 +02:00
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
/*
* Stay on the same line if reporting to a terminal and we're not done
* yet.
*/
Avoid non-constant format string argument to fprintf(). As Tom Lane pointed out, it could defeat the compiler's printf() format string verification. Backpatch to v12, like that patch that introduced it. Discussion: https://www.postgresql.org/message-id/1069283.1597672779%40sss.pgh.pa.us
2020-08-18 12:13:09 +02:00
fputc((!finished && isatty(fileno(stderr))) ? '\r' : '\n', stderr);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
static int32
parse_max_rate(char *src)
{
double result;
char *after_num;
char *suffix = NULL;
errno = 0;
result = strtod(src, &after_num);
if (src == after_num)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("transfer rate \"%s\" is not a valid value", src);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
if (errno != 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("invalid transfer rate \"%s\": %m", src);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
if (result <= 0)
{
/*
* Reject obviously wrong values here.
*/
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("transfer rate must be greater than zero");
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
/*
* Evaluate suffix, after skipping over possible whitespace. Lack of
* suffix means kilobytes.
*/
while (*after_num != '\0' && isspace((unsigned char) *after_num))
after_num++;
if (*after_num != '\0')
{
suffix = after_num;
if (*after_num == 'k')
{
/* kilobyte is the expected unit. */
after_num++;
}
else if (*after_num == 'M')
{
after_num++;
result *= 1024.0;
}
}
/* The rest can only consist of white space. */
while (*after_num != '\0' && isspace((unsigned char) *after_num))
after_num++;
if (*after_num != '\0')
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("invalid --max-rate unit: \"%s\"", suffix);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
/* Valid integer? */
if ((uint64) result != (uint64) ((uint32) result))
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("transfer rate \"%s\" exceeds integer range", src);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
/*
* The range is checked on the server side too, but avoid the server
* connection if a nonsensical value was passed.
*/
if (result < MAX_RATE_LOWER || result > MAX_RATE_UPPER)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("transfer rate \"%s\" is out of range", src);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
return (int32) result;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* Read a stream of COPY data and invoke the provided callback for each
* chunk.
*/
static void
ReceiveCopyData(PGconn *conn, WriteDataCallback callback,
void *callback_data)
{
PGresult *res;
/* Get the COPY data stream. */
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COPY_OUT)
{
pg_log_error("could not get COPY data stream: %s",
PQerrorMessage(conn));
exit(1);
}
PQclear(res);
/* Loop over chunks until done. */
while (1)
{
int r;
char *copybuf;
r = PQgetCopyData(conn, &copybuf, 0);
if (r == -1)
{
/* End of chunk. */
break;
}
else if (r == -2)
{
pg_log_error("could not read COPY data: %s",
PQerrorMessage(conn));
exit(1);
}
(*callback) (r, copybuf, callback_data);
PQfreemem(copybuf);
}
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
* Write a piece of tar data
*/
static void
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(WriteTarState *state, char *buf, int r)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
#ifdef HAVE_LIBZ
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state->ztarfile != NULL)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
errno = 0;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (gzwrite(state->ztarfile, buf, r) != r)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
errno = ENOSPC;
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not write to compressed file \"%s\": %s",
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->filename, get_gz_error(state->ztarfile));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
}
else
#endif
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
errno = 0;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (fwrite(buf, r, 1, state->tarfile) != 1)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
errno = ENOSPC;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not write to file \"%s\": %m",
state->filename);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
}
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Receive a tar format file from the connection to the server, and write
* the data from this file directly into a tar file. If compression is
* enabled, the data will be compressed while written to the file.
*
* The file will be named base.tar[.gz] if it's for the main data directory
* or <tablespaceoid>.tar[.gz] if it's for another tablespace.
*
* No attempt to inspect or validate the contents of the file is done.
*/
static void
ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
{
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char zerobuf[TAR_BLOCK_SIZE * 2];
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
WriteTarState state;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
memset(&state, 0, sizeof(state));
state.tablespacenum = rownum;
state.basetablespace = PQgetisnull(res, rownum, 0);
state.in_tarhdr = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Fix compatibility of pg_basebackup -R with 11 and older versions When 2dedf4d9 has integrated recovery.conf into postgresql.conf, it also changed pg_basebackup -R in the way recovery configuration is generated. However this implementation forgot the fact that pg_basebackup needs to keep compatibility with older server versions as well. Reported-by: Devrim Gündüz Author: Sergei Kornilov, Michael Paquier Discussion: https://postgr.es/m/3458f7cd12d74acd90180a671c8d5a081d60e162.camel@gunduz.org
2019-03-08 02:14:03 +01:00
/* recovery.conf is integrated into postgresql.conf in 12 and newer */
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (PQserverVersion(conn) >= MINIMUM_VERSION_FOR_RECOVERY_GUC)
state.is_recovery_guc_supported = true;
Fix compatibility of pg_basebackup -R with 11 and older versions When 2dedf4d9 has integrated recovery.conf into postgresql.conf, it also changed pg_basebackup -R in the way recovery configuration is generated. However this implementation forgot the fact that pg_basebackup needs to keep compatibility with older server versions as well. Reported-by: Devrim Gündüz Author: Sergei Kornilov, Michael Paquier Discussion: https://postgr.es/m/3458f7cd12d74acd90180a671c8d5a081d60e162.camel@gunduz.org
2019-03-08 02:14:03 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state.basetablespace)
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
{
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Base tablespaces
*/
if (strcmp(basedir, "-") == 0)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
Fix pg_basebackup output to stdout on Windows. When writing a backup to stdout with pg_basebackup on Windows, put stdout to binary mode. Any CR bytes in the output will otherwise be output incorrectly as CR+LF. In the passing, standardize on using "_setmode" instead of "setmode", for the sake of consistency. They both do the same thing, but according to MSDN documentation, setmode is deprecated. Fixes bug #14634, reported by Henry Boehlert. Patch by Haribabu Kommi. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/20170428082818.24366.13134@wrigleys.postgresql.org
2017-07-14 15:02:53 +02:00
#ifdef WIN32
_setmode(fileno(stdout), _O_BINARY);
#endif
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
Fix fuzzy error handling in pg_basebackup when opening gzFile First, this code did not bother checking for a failure when calling dup(). Then, per zlib, gzerror() returns NULL for a NULL input, which can happen if passing down to gzdopen() an invalid file descriptor or if there was an allocation failure. No back-patch is done as this would unlikely be a problem in the field. Per Coverity. Reported-by: Tom Lane
2020-02-04 05:56:04 +01:00
int fd = dup(fileno(stdout));
Initial pgindent and pgperltidy run for v13. Includes some manual cleanup of places that pgindent messed up, most of which weren't per project style anyway. Notably, it seems some people didn't absorb the style rules of commit c9d297751, because there were a bunch of new occurrences of function calls with a newline just after the left paren, all with faulty expectations about how the rest of the call would get indented.
2020-05-14 19:06:38 +02:00
Fix fuzzy error handling in pg_basebackup when opening gzFile First, this code did not bother checking for a failure when calling dup(). Then, per zlib, gzerror() returns NULL for a NULL input, which can happen if passing down to gzdopen() an invalid file descriptor or if there was an allocation failure. No back-patch is done as this would unlikely be a problem in the field. Per Coverity. Reported-by: Tom Lane
2020-02-04 05:56:04 +01:00
if (fd < 0)
{
pg_log_error("could not duplicate stdout: %m");
exit(1);
}
state.ztarfile = gzdopen(fd, "wb");
if (state.ztarfile == NULL)
{
pg_log_error("could not open output file: %m");
exit(1);
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (gzsetparams(state.ztarfile, compresslevel,
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
Z_DEFAULT_STRATEGY) != Z_OK)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not set compression level %d: %s",
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
compresslevel, get_gz_error(state.ztarfile));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
}
}
else
#endif
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state.tarfile = stdout;
strcpy(state.filename, "-");
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
{
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
snprintf(state.filename, sizeof(state.filename),
"%s/base.tar.gz", basedir);
state.ztarfile = gzopen(state.filename, "wb");
if (gzsetparams(state.ztarfile, compresslevel,
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
Z_DEFAULT_STRATEGY) != Z_OK)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not set compression level %d: %s",
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
compresslevel, get_gz_error(state.ztarfile));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
else
#endif
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
snprintf(state.filename, sizeof(state.filename),
"%s/base.tar", basedir);
state.tarfile = fopen(state.filename, "wb");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
{
/*
* Specific tablespace
*/
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
snprintf(state.filename, sizeof(state.filename),
"%s/%s.tar.gz",
basedir, PQgetvalue(res, rownum, 0));
state.ztarfile = gzopen(state.filename, "wb");
if (gzsetparams(state.ztarfile, compresslevel,
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
Z_DEFAULT_STRATEGY) != Z_OK)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not set compression level %d: %s",
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
compresslevel, get_gz_error(state.ztarfile));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
else
#endif
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
snprintf(state.filename, sizeof(state.filename), "%s/%s.tar",
basedir, PQgetvalue(res, rownum, 0));
state.tarfile = fopen(state.filename, "wb");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (!state.ztarfile)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
/* Compression is in use */
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create compressed file \"%s\": %s",
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state.filename, get_gz_error(state.ztarfile));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
else
#endif
{
/* Either no zlib support, or zlib support but compresslevel = 0 */
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (!state.tarfile)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not create file \"%s\": %m", state.filename);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* End of copy data. If requested, and this is the base tablespace, write
* configuration file into the tarfile. When done, close the file (but not
* stdout).
*
* Also, write two completely empty blocks at the end of the tar file, as
* required by some tar programs.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
MemSet(zerobuf, 0, sizeof(zerobuf));
if (state.basetablespace && writerecoveryconf)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char header[TAR_BLOCK_SIZE];
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* If postgresql.auto.conf has not been found in the streamed data,
* add recovery configuration to postgresql.auto.conf if recovery
* parameters are GUCs. If the instance connected to is older than
* 12, create recovery.conf with this data otherwise.
*/
if (!state.found_postgresql_auto_conf || !state.is_recovery_guc_supported)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
int padding;
tarCreateHeader(header,
state.is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf",
NULL,
recoveryconfcontents->len,
pg_file_create_mode, 04000, 02000,
time(NULL));
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
padding = tarPaddingBytesRequired(recoveryconfcontents->len);
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(&state, header, sizeof(header));
writeTarData(&state, recoveryconfcontents->data,
recoveryconfcontents->len);
if (padding)
writeTarData(&state, zerobuf, padding);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* standby.signal is supported only if recovery parameters are GUCs.
*/
if (state.is_recovery_guc_supported)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
tarCreateHeader(header, "standby.signal", NULL,
0, /* zero-length file */
pg_file_create_mode, 04000, 02000,
time(NULL));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(&state, header, sizeof(header));
Fix bogus tar-file padding logic for standby.signal. When pg_basebackup -R is used, we inject standby.signal into the tar file for the main tablespace. The proper thing to do is to pad each file injected into the tar file out to a 512-byte boundary by appending nulls, but here the file is of length 0 and we add 511 zero bytes. Since 0 is already a multiple of 512, we should not add any zero bytes. Do that instead. Patch by me, reviewed by Tom Lane. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-27 19:04:35 +02:00
/*
* we don't need to pad out to a multiple of the tar block size
* here, because the file is zero length, which is a multiple of
* any block size.
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
}
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
/*
* Normally, we emit the backup manifest as a separate file, but when
* we're writing a tarfile to stdout, we don't have that option, so
* include it in the one tarfile we've got.
*/
Code review for backup manifest. This commit prevents pg_basebackup from receiving backup_manifest file when --no-manifest is specified. Previously, when pg_basebackup was writing a tarfile to stdout, it tried to receive backup_manifest file even when --no-manifest was specified, and reported an error. Also remove unused -m option from pg_basebackup. Also fix typo in BASE_BACKUP command documentation. Author: Fujii Masao Reviewed-by: Michael Paquier, Robert Haas Discussion: https://postgr.es/m/01e3ed3a-8729-5aaa-ca84-e60e3ca59db8@oss.nttdata.com
2020-04-15 04:15:12 +02:00
if (strcmp(basedir, "-") == 0 && manifest)
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
{
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char header[TAR_BLOCK_SIZE];
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
PQExpBufferData buf;
initPQExpBuffer(&buf);
ReceiveBackupManifestInMemory(conn, &buf);
if (PQExpBufferDataBroken(buf))
{
pg_log_error("out of memory");
exit(1);
}
tarCreateHeader(header, "backup_manifest", NULL, buf.len,
pg_file_create_mode, 04000, 02000,
time(NULL));
writeTarData(&state, header, sizeof(header));
writeTarData(&state, buf.data, buf.len);
termPQExpBuffer(&buf);
}
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
/* 2 * TAR_BLOCK_SIZE bytes empty data at end of file */
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(&state, zerobuf, sizeof(zerobuf));
#ifdef HAVE_LIBZ
if (state.ztarfile != NULL)
{
if (gzclose(state.ztarfile) != 0)
{
pg_log_error("could not close compressed file \"%s\": %s",
state.filename, get_gz_error(state.ztarfile));
exit(1);
}
}
else
#endif
{
if (strcmp(basedir, "-") != 0)
{
if (fclose(state.tarfile) != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not close file \"%s\": %m",
state.filename);
exit(1);
}
}
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(rownum, state.filename, true, false);
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* Do not sync the resulting tar file yet, all files are synced once at
* the end.
*/
}
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* Receive one chunk of tar-format data from the server.
*/
static void
ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data)
{
WriteTarState *state = callback_data;
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (!writerecoveryconf || !state->basetablespace)
{
/*
* When not writing config file, or when not working on the base
* tablespace, we never have to look for an existing configuration
* file in the stream.
*/
writeTarData(state, copybuf, r);
}
else
{
/*
* Look for a config file in the existing tar stream. If it's there,
* we must skip it so we can later overwrite it with our own version
* of the file.
*
* To do this, we have to process the individual files inside the TAR
* stream. The stream consists of a header and zero or more chunks,
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
* each with a length equal to TAR_BLOCK_SIZE. The stream from the
* server is broken up into smaller pieces, so we have to track the
* size of the files to find the next header structure.
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
*/
int rr = r;
int pos = 0;
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
while (rr > 0)
{
if (state->in_tarhdr)
{
Fix compatibility of pg_basebackup -R with 11 and older versions When 2dedf4d9 has integrated recovery.conf into postgresql.conf, it also changed pg_basebackup -R in the way recovery configuration is generated. However this implementation forgot the fact that pg_basebackup needs to keep compatibility with older server versions as well. Reported-by: Devrim Gündüz Author: Sergei Kornilov, Michael Paquier Discussion: https://postgr.es/m/3458f7cd12d74acd90180a671c8d5a081d60e162.camel@gunduz.org
2019-03-08 02:14:03 +01:00
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* We're currently reading a header structure inside the TAR
* stream, i.e. the file metadata.
Fix compatibility of pg_basebackup -R with 11 and older versions When 2dedf4d9 has integrated recovery.conf into postgresql.conf, it also changed pg_basebackup -R in the way recovery configuration is generated. However this implementation forgot the fact that pg_basebackup needs to keep compatibility with older server versions as well. Reported-by: Devrim Gündüz Author: Sergei Kornilov, Michael Paquier Discussion: https://postgr.es/m/3458f7cd12d74acd90180a671c8d5a081d60e162.camel@gunduz.org
2019-03-08 02:14:03 +01:00
*/
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
if (state->tarhdrsz < TAR_BLOCK_SIZE)
Fix compatibility of pg_basebackup -R with 11 and older versions When 2dedf4d9 has integrated recovery.conf into postgresql.conf, it also changed pg_basebackup -R in the way recovery configuration is generated. However this implementation forgot the fact that pg_basebackup needs to keep compatibility with older server versions as well. Reported-by: Devrim Gündüz Author: Sergei Kornilov, Michael Paquier Discussion: https://postgr.es/m/3458f7cd12d74acd90180a671c8d5a081d60e162.camel@gunduz.org
2019-03-08 02:14:03 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* Copy the header structure into tarhdr in case the
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
* header is not aligned properly or it's not returned in
* whole by the last PQgetCopyData call.
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
*/
int hdrleft;
int bytes2copy;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
hdrleft = TAR_BLOCK_SIZE - state->tarhdrsz;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
bytes2copy = (rr > hdrleft ? hdrleft : rr);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
memcpy(&state->tarhdr[state->tarhdrsz], copybuf + pos,
bytes2copy);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
rr -= bytes2copy;
pos += bytes2copy;
state->tarhdrsz += bytes2copy;
pg_basebackup: Error handling fixes. Thomas Ogrisegg and Fujii Masao
2012-03-28 18:19:22 +02:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
else
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* We have the complete header structure in tarhdr, look
* at the file metadata: we may want append recovery info
* into postgresql.auto.conf and skip standby.signal file
* if recovery parameters are integrated as GUCs, and
* recovery.conf otherwise. In both cases we must
* calculate tar padding.
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state->is_recovery_guc_supported)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->skip_file =
(strcmp(&state->tarhdr[0], "standby.signal") == 0);
state->is_postgresql_auto_conf =
(strcmp(&state->tarhdr[0], "postgresql.auto.conf") == 0);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
else
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->skip_file =
(strcmp(&state->tarhdr[0], "recovery.conf") == 0);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->filesz = read_tar_number(&state->tarhdr[124], 12);
state->file_padding_len =
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
tarPaddingBytesRequired(state->filesz);
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state->is_recovery_guc_supported &&
state->is_postgresql_auto_conf &&
writerecoveryconf)
{
/* replace tar header */
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char header[TAR_BLOCK_SIZE];
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
tarCreateHeader(header, "postgresql.auto.conf", NULL,
state->filesz + recoveryconfcontents->len,
pg_file_create_mode, 04000, 02000,
time(NULL));
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(state, header, sizeof(header));
}
else
{
/* copy stream with padding */
state->filesz += state->file_padding_len;
if (!state->skip_file)
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* If we're not skipping the file, write the tar
* header unmodified.
*/
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
writeTarData(state, state->tarhdr, TAR_BLOCK_SIZE);
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/* Next part is the file, not the header */
state->in_tarhdr = false;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
}
else
{
/*
* We're processing a file's contents.
*/
if (state->filesz > 0)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* We still have data to read (and possibly write).
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
int bytes2write;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
bytes2write = (state->filesz > rr ? rr : state->filesz);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (!state->skip_file)
writeTarData(state, copybuf + pos, bytes2write);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
rr -= bytes2write;
pos += bytes2write;
state->filesz -= bytes2write;
}
else if (state->is_recovery_guc_supported &&
state->is_postgresql_auto_conf &&
writerecoveryconf)
{
/* append recovery config to postgresql.auto.conf */
int padding;
int tailsize;
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
tailsize = (TAR_BLOCK_SIZE - state->file_padding_len) + recoveryconfcontents->len;
padding = tarPaddingBytesRequired(tailsize);
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
writeTarData(state, recoveryconfcontents->data,
recoveryconfcontents->len);
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (padding)
{
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
char zerobuf[TAR_BLOCK_SIZE];
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
MemSet(zerobuf, 0, sizeof(zerobuf));
writeTarData(state, zerobuf, padding);
}
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/* skip original file padding */
state->is_postgresql_auto_conf = false;
state->skip_file = true;
state->filesz += state->file_padding_len;
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->found_postgresql_auto_conf = true;
}
else
{
/*
* No more data in the current file, the next piece of
* data (if any) will be a new file header structure.
*/
state->in_tarhdr = true;
state->skip_file = false;
state->is_postgresql_auto_conf = false;
state->tarhdrsz = 0;
state->filesz = 0;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
}
totaldone += r;
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(state->tablespacenum, state->filename, false, false);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
/*
* Retrieve tablespace path, either relocated or original depending on whether
* -T was passed or not.
*/
static const char *
get_tablespace_mapping(const char *dir)
{
TablespaceListCell *cell;
pg_basebackup: Fix comparison handling of tablespace mappings on Windows A candidate path needs to be canonicalized before being checked against the mappings, because the mappings are also canonicalized. This is especially relevant on Windows Reported-by: nb <nbedxp@gmail.com> Author: Michael Paquier <michael.paquier@gmail.com> Reviewed-by: Ashutosh Sharma <ashu.coek88@gmail.com>
2017-11-01 15:20:05 +01:00
char canon_dir[MAXPGPATH];
/* Canonicalize path for comparison consistency */
strlcpy(canon_dir, dir, sizeof(canon_dir));
canonicalize_path(canon_dir);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
for (cell = tablespace_dirs.head; cell; cell = cell->next)
pg_basebackup: Fix comparison handling of tablespace mappings on Windows A candidate path needs to be canonicalized before being checked against the mappings, because the mappings are also canonicalized. This is especially relevant on Windows Reported-by: nb <nbedxp@gmail.com> Author: Michael Paquier <michael.paquier@gmail.com> Reviewed-by: Ashutosh Sharma <ashu.coek88@gmail.com>
2017-11-01 15:20:05 +01:00
if (strcmp(canon_dir, cell->old_dir) == 0)
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
return cell->new_dir;
return dir;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Receive a tar format stream from the connection to the server, and unpack
* the contents of it into a directory. Only files, directories and
* symlinks are supported, no other kinds of special files.
*
* If the data is for the main data directory, it will be restored in the
* specified directory. If it's for another tablespace, it will be restored
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
* in the original or mapped directory.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
UnpackTarState state;
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
bool basetablespace;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
memset(&state, 0, sizeof(state));
state.tablespacenum = rownum;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
basetablespace = PQgetisnull(res, rownum, 0);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (basetablespace)
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
strlcpy(state.current_path, basedir, sizeof(state.current_path));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
strlcpy(state.current_path,
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
get_tablespace_mapping(PQgetvalue(res, rownum, 1)),
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
sizeof(state.current_path));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
ReceiveCopyData(conn, ReceiveTarAndUnpackCopyChunk, &state);
if (state.file)
fclose(state.file);
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(rownum, state.filename, true, false);
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state.file != NULL)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("COPY stream ended before last file was finished");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (basetablespace && writerecoveryconf)
WriteRecoveryConfig(conn, basedir, recoveryconfcontents);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* No data is synced here, everything is done for all tablespaces at the
* end.
*/
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
static void
ReceiveTarAndUnpackCopyChunk(size_t r, char *copybuf, void *callback_data)
{
UnpackTarState *state = callback_data;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state->file == NULL)
{
#ifndef WIN32
int filemode;
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* No current file, so this must be the header for a new file
*/
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
if (r != TAR_BLOCK_SIZE)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("invalid tar block header size: %zu", r);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
totaldone += TAR_BLOCK_SIZE;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->current_len_left = read_tar_number(&copybuf[124], 12);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Fix compilation warning of pg_basebackup with MinGW Several buildfarm members have been complaining about that with gcc, like jacana. Weirdly enough, Visual Studio's compilers do not find this issue. Author: Michael Paquier Reviewed-by: Andrew Dunstan Discussion: https://postgr.es/m/20190719050830.GK1859@paquier.xyz
2019-07-21 15:27:11 +02:00
#ifndef WIN32
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/* Set permissions on the file */
filemode = read_tar_number(&copybuf[100], 8);
Fix compilation warning of pg_basebackup with MinGW Several buildfarm members have been complaining about that with gcc, like jacana. Weirdly enough, Visual Studio's compilers do not find this issue. Author: Michael Paquier Reviewed-by: Andrew Dunstan Discussion: https://postgr.es/m/20190719050830.GK1859@paquier.xyz
2019-07-21 15:27:11 +02:00
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
* All files are padded up to a multiple of TAR_BLOCK_SIZE
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
*/
state->current_padding =
Assorted cleanup of tar-related code. Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-04-24 16:38:10 +02:00
tarPaddingBytesRequired(state->current_len_left);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* First part of header is zero terminated filename
*/
snprintf(state->filename, sizeof(state->filename),
"%s/%s", state->current_path, copybuf);
if (state->filename[strlen(state->filename) - 1] == '/')
{
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* Ends in a slash means directory or symlink to directory
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (copybuf[156] == '5')
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* Directory. Remove trailing slash first.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->filename[strlen(state->filename) - 1] = '\0';
if (mkdir(state->filename, pg_dir_create_mode) != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* When streaming WAL, pg_wal (or pg_xlog for pre-9.6
* clusters) will have been created by the wal receiver
* process. Also, when the WAL directory location was
* specified, pg_wal (or pg_xlog) has already been created
* as a symbolic link before starting the actual backup.
* So just ignore creation failures on related
* directories.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (!((pg_str_endswith(state->filename, "/pg_wal") ||
pg_str_endswith(state->filename, "/pg_xlog") ||
pg_str_endswith(state->filename, "/archive_status")) &&
errno == EEXIST))
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not create directory \"%s\": %m",
state->filename);
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifndef WIN32
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (chmod(state->filename, (mode_t) filemode))
Add some missing exit() calls in error paths for various binaries The following changes are done: - In pg_archivecleanup, the cleanup of older WAL segments would never fail immediately. - In pgbench, the initialization of a thread barrier would not fail hard. - In pg_recvlogical, a stat() failure never got the call. - In pg_basebackup, two chmod() reported a failure without exit()'ing when unpacking some tar data freshly received. It may be possible to continue writing some data even after this failure, but that could be confusing to the user at the end. These are arguably bugs, but they would happen for code paths where a failure is unlikely going to happen, so no backpatch is done. Reviewed-by: Robert Haas, Fabien Coelho Discussion: https://postgr.es/m/YQDMdB+B68yePFeT@paquier.xyz
2021-07-29 04:42:58 +02:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not set permissions on directory \"%s\": %m",
state->filename);
Add some missing exit() calls in error paths for various binaries The following changes are done: - In pg_archivecleanup, the cleanup of older WAL segments would never fail immediately. - In pgbench, the initialization of a thread barrier would not fail hard. - In pg_recvlogical, a stat() failure never got the call. - In pg_basebackup, two chmod() reported a failure without exit()'ing when unpacking some tar data freshly received. It may be possible to continue writing some data even after this failure, but that could be confusing to the user at the end. These are arguably bugs, but they would happen for code paths where a failure is unlikely going to happen, so no backpatch is done. Reviewed-by: Robert Haas, Fabien Coelho Discussion: https://postgr.es/m/YQDMdB+B68yePFeT@paquier.xyz
2021-07-29 04:42:58 +02:00
exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
}
else if (copybuf[156] == '2')
{
/*
* Symbolic link
*
* It's most likely a link in pg_tblspc directory, to the
* location of a tablespace. Apply any tablespace mapping
* given on the command line (--tablespace-mapping). (We
* blindly apply the mapping without checking that the link
* really is inside pg_tblspc. We don't expect there to be
* other symlinks in a data directory, but if there are, you
* can call it an undocumented feature that you can map them
* too.)
*/
state->filename[strlen(state->filename) - 1] = '\0'; /* Remove trailing slash */
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->mapped_tblspc_path =
get_tablespace_mapping(&copybuf[157]);
if (symlink(state->mapped_tblspc_path, state->filename) != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not create symbolic link from \"%s\" to \"%s\": %m",
state->filename, state->mapped_tblspc_path);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
else
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("unrecognized link indicator \"%c\"",
copybuf[156]);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
return; /* directory or link handled */
}
/*
* regular file
*/
state->file = fopen(state->filename, "wb");
if (!state->file)
{
pg_log_error("could not create file \"%s\": %m", state->filename);
exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifndef WIN32
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (chmod(state->filename, (mode_t) filemode))
Add some missing exit() calls in error paths for various binaries The following changes are done: - In pg_archivecleanup, the cleanup of older WAL segments would never fail immediately. - In pgbench, the initialization of a thread barrier would not fail hard. - In pg_recvlogical, a stat() failure never got the call. - In pg_basebackup, two chmod() reported a failure without exit()'ing when unpacking some tar data freshly received. It may be possible to continue writing some data even after this failure, but that could be confusing to the user at the end. These are arguably bugs, but they would happen for code paths where a failure is unlikely going to happen, so no backpatch is done. Reviewed-by: Robert Haas, Fabien Coelho Discussion: https://postgr.es/m/YQDMdB+B68yePFeT@paquier.xyz
2021-07-29 04:42:58 +02:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not set permissions on file \"%s\": %m",
state->filename);
Add some missing exit() calls in error paths for various binaries The following changes are done: - In pg_archivecleanup, the cleanup of older WAL segments would never fail immediately. - In pgbench, the initialization of a thread barrier would not fail hard. - In pg_recvlogical, a stat() failure never got the call. - In pg_basebackup, two chmod() reported a failure without exit()'ing when unpacking some tar data freshly received. It may be possible to continue writing some data even after this failure, but that could be confusing to the user at the end. These are arguably bugs, but they would happen for code paths where a failure is unlikely going to happen, so no backpatch is done. Reviewed-by: Robert Haas, Fabien Coelho Discussion: https://postgr.es/m/YQDMdB+B68yePFeT@paquier.xyz
2021-07-29 04:42:58 +02:00
exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (state->current_len_left == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
/*
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
* Done with this file, next one will be a new tar header
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
fclose(state->file);
state->file = NULL;
return;
}
} /* new file */
else
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
/*
* Continuing blocks in existing file
*/
if (state->current_len_left == 0 && r == state->current_padding)
{
/*
* Received the padding block for this file, ignore it and close
* the file, then move on to the next tar header.
*/
fclose(state->file);
state->file = NULL;
totaldone += r;
return;
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
errno = 0;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
if (fwrite(copybuf, r, 1, state->file) != 1)
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
errno = ENOSPC;
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
pg_log_error("could not write to file \"%s\": %m", state->filename);
exit(1);
}
totaldone += r;
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(state->tablespacenum, state->filename, false, false);
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
pg_basebackup: Refactor code for reading COPY and tar data. Add a new function ReceiveCopyData that does just that, taking a callback as an argument to specify what should be done with each chunk as it is received. This allows a single copy of the logic to be shared between ReceiveTarFile and ReceiveAndUnpackTarFile, and eliminates a few #ifdef conditions based on HAVE_LIBZ. While this is slightly more code, it's arguably clearer, and there is a pending patch that introduces additional calls to ReceiveCopyData. This commit is not intended to result in any functional change. Discussion: http://postgr.es/m/CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com
2019-12-05 21:14:09 +01:00
state->current_len_left -= r;
if (state->current_len_left == 0 && state->current_padding == 0)
{
/*
* Received the last block, and there is no padding to be
* expected. Close the file and move on to the next tar header.
*/
fclose(state->file);
state->file = NULL;
return;
}
} /* continuing data in existing file */
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
/*
* Receive the backup manifest file and write it out to a file.
*/
static void
ReceiveBackupManifest(PGconn *conn)
{
WriteManifestState state;
snprintf(state.filename, sizeof(state.filename),
"%s/backup_manifest.tmp", basedir);
state.file = fopen(state.filename, "wb");
if (state.file == NULL)
{
pg_log_error("could not create file \"%s\": %m", state.filename);
exit(1);
}
ReceiveCopyData(conn, ReceiveBackupManifestChunk, &state);
fclose(state.file);
}
/*
* Receive one chunk of the backup manifest file and write it out to a file.
*/
static void
ReceiveBackupManifestChunk(size_t r, char *copybuf, void *callback_data)
{
WriteManifestState *state = callback_data;
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
errno = 0;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
if (fwrite(copybuf, r, 1, state->file) != 1)
{
Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com
2020-06-19 22:46:07 +02:00
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
errno = ENOSPC;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
pg_log_error("could not write to file \"%s\": %m", state->filename);
exit(1);
}
}
/*
* Receive the backup manifest file and write it out to a file.
*/
static void
ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf)
{
ReceiveCopyData(conn, ReceiveBackupManifestInMemoryChunk, buf);
}
/*
* Receive one chunk of the backup manifest file and write it out to a file.
*/
static void
ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data)
{
PQExpBuffer buf = callback_data;
appendPQExpBuffer(buf, copybuf, r);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
static void
fix up a couple non-prototypes of the form foo() to be foo(void) -- found using -Wstrict-prototypes
2011-03-27 22:31:41 +02:00
BaseBackup(void)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
PGresult *res;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
char *sysidentifier;
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
TimeLineID latesttli;
TimeLineID starttli;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
char *basebkp;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
char escaped_label[MAXPGPATH];
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
char *maxrate_clause = NULL;
Disable silently generation of manifests with servers <= 12 in pg_basebackup Since 0d8c9c1, pg_basebackup would generate an error if connected to a backend version older than 12 where backup manifests are not supported. Avoiding this error is possible by using the --no-manifest option. This error handling could be confusing for some users, where patching a backup script that interacts with multiple backend versions would cause the addition of --no-manifest to potentially not generate a backup manifest even for Postgres 13 and newer versions. As we want to encourage the use of backup manifests as much as possible, this commit silently disables manifests where not supported, instead of generating an error. While on it, rework a bit the code to make it more consistent with the surroundings when generating the BASE_BACKUP command. Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro Herrera, Kyotaro Horiguchi, Tom Lane, David Steele, and me. Author: Michael Paquier Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz
2020-04-16 06:57:07 +02:00
char *manifest_clause = NULL;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
char *manifest_checksums_clause = "";
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
int i;
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
char xlogstart[64];
char xlogend[64];
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
int minServerMajor,
maxServerMajor;
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
int serverVersion,
serverMajor;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
int writing_to_stdout;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
Assert(conn != NULL);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
/*
* Check server version. BASE_BACKUP command was introduced in 9.1, so we
* can't work with servers older than 9.1.
*/
minServerMajor = 901;
maxServerMajor = PG_VERSION_NUM / 100;
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
serverVersion = PQserverVersion(conn);
serverMajor = serverVersion / 100;
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
if (serverMajor < minServerMajor || serverMajor > maxServerMajor)
{
const char *serverver = PQparameterStatus(conn, "server_version");
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("incompatible server version %s",
serverver ? serverver : "'unknown'");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
}
/*
* If WAL streaming was requested, also check that the server is new
* enough for that.
*/
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (includewal == STREAM_WAL && !CheckServerVersionForStreaming(conn))
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
{
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
/*
* Error message already written in CheckServerVersionForStreaming(),
* but add a hint about using -X none.
*/
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("HINT: use -X none or -X fetch to disable log streaming");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
* Build contents of configuration file if requested
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
*/
if (writerecoveryconf)
Split out recovery confing-writing code from pg_basebackup ... into a new file, fe_utils/recovery_gen.c. This can later be used by pg_rewind. Authors: Paul Guo, Jimmy Yih, Ashwin Agrawal. A few tweaks by Álvaro Herrera Reviewed-by: Michaël Paquier Discussion: https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
2019-09-25 19:35:24 +02:00
recoveryconfcontents = GenerateRecoveryConfig(conn, replication_slot);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
*/
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
if (!RunIdentifySystem(conn, &sysidentifier, &latesttli, NULL, NULL))
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Start the actual backup
*/
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQescapeStringConn(conn, escaped_label, label, sizeof(escaped_label), &i);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
if (maxrate > 0)
maxrate_clause = psprintf("MAX_RATE %u", maxrate);
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
if (manifest)
{
if (manifest_force_encode)
manifest_clause = "MANIFEST 'force-encode'";
else
manifest_clause = "MANIFEST 'yes'";
if (manifest_checksums != NULL)
manifest_checksums_clause = psprintf("MANIFEST_CHECKSUMS '%s'",
manifest_checksums);
}
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("initiating base backup, waiting for checkpoint to complete");
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
if (showprogress && !verbose)
pg_basebackup: Fix progress messages when writing to a file The progress messages print out \r to keep overwriting the same line on the screen. But this does not yield useful results when writing the output to a file. So in that case, print out \n instead. Author: Martín Marqués <martin@2ndquadrant.com> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
2017-12-01 15:21:34 +01:00
{
fprintf(stderr, "waiting for checkpoint");
if (isatty(fileno(stderr)))
fprintf(stderr, "\r");
else
fprintf(stderr, "\n");
}
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
basebkp =
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s %s %s %s",
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
escaped_label,
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
estimatesize ? "PROGRESS" : "",
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal == FETCH_WAL ? "WAL" : "",
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
fastcheckpoint ? "FAST" : "",
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal == NO_WAL ? "" : "NOWAIT",
Map basebackup tablespaces using a tablespace_map file Windows can't reliably restore symbolic links from a tar format, so instead during backup start we create a tablespace_map file, which is used by the restoring postgres to create the correct links in pg_tblspc. The backup protocol also now has an option to request this file to be included in the backup stream, and this is used by pg_basebackup when operating in tar mode. This is done on all platforms, not just Windows. This means that pg_basebackup will not not work in tar mode against 9.4 and older servers, as this protocol option isn't implemented there. Amit Kapila, reviewed by Dilip Kumar, with a little editing from me.
2015-05-12 15:29:10 +02:00
maxrate_clause ? maxrate_clause : "",
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
format == 't' ? "TABLESPACE_MAP" : "",
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
verify_checksums ? "" : "NOVERIFY_CHECKSUMS",
Disable silently generation of manifests with servers <= 12 in pg_basebackup Since 0d8c9c1, pg_basebackup would generate an error if connected to a backend version older than 12 where backup manifests are not supported. Avoiding this error is possible by using the --no-manifest option. This error handling could be confusing for some users, where patching a backup script that interacts with multiple backend versions would cause the addition of --no-manifest to potentially not generate a backup manifest even for Postgres 13 and newer versions. As we want to encourage the use of backup manifests as much as possible, this commit silently disables manifests where not supported, instead of generating an error. While on it, rework a bit the code to make it more consistent with the surroundings when generating the BASE_BACKUP command. Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro Herrera, Kyotaro Horiguchi, Tom Lane, David Steele, and me. Author: Michael Paquier Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz
2020-04-16 06:57:07 +02:00
manifest_clause ? manifest_clause : "",
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
manifest_checksums_clause);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
if (PQsendQuery(conn, basebkp) == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not send replication command \"%s\": %s",
"BASE_BACKUP", PQerrorMessage(conn));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
/*
Standardize "WAL location" terminology Other previously used terms were "WAL position" or "log position".
2017-05-12 19:51:27 +02:00
* Get the starting WAL location
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not initiate base backup: %s",
PQerrorMessage(conn));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
if (PQntuples(res) != 1)
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("server returned unexpected response to BASE_BACKUP command; got %d rows and %d fields, expected %d rows and %d fields",
PQntuples(res), PQnfields(res), 1, 2);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
}
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
Prevent potential overruns of fixed-size buffers. Coverity identified a number of places in which it couldn't prove that a string being copied into a fixed-size buffer would fit. We believe that most, perhaps all of these are in fact safe, or are copying data that is coming from a trusted source so that any overrun is not really a security issue. Nonetheless it seems prudent to forestall any risk by using strlcpy() and similar functions. Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports. In addition, fix a potential null-pointer-dereference crash in contrib/chkpass. The crypt(3) function is defined to return NULL on failure, but chkpass.c didn't check for that before using the result. The main practical case in which this could be an issue is if libc is configured to refuse to execute unapproved hashing algorithms (e.g., "FIPS mode"). This ideally should've been a separate commit, but since it touches code adjacent to one of the buffer overrun changes, I included it in this commit to avoid last-minute merge issues. This issue was reported by Honza Horak. Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 17:20:21 +01:00
strlcpy(xlogstart, PQgetvalue(res, 0, 0), sizeof(xlogstart));
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("checkpoint completed");
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
/*
* 9.3 and later sends the TLI of the starting point. With older servers,
* assume it's the same as the latest timeline reported by
* IDENTIFY_SYSTEM.
*/
if (PQnfields(res) >= 2)
starttli = atoi(PQgetvalue(res, 0, 1));
else
starttli = latesttli;
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
PQclear(res);
MemSet(xlogend, 0, sizeof(xlogend));
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (verbose && includewal != NO_WAL)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("write-ahead log start point: %s on timeline %u",
xlogstart, starttli);
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Get the header
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not get backup header: %s",
PQerrorMessage(conn));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (PQntuples(res) < 1)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("no data returned from server");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
/*
* Sum up the total size, for progress reporting
*/
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
totalsize_kb = totaldone = 0;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
tablespacecount = PQntuples(res);
for (i = 0; i < PQntuples(res); i++)
{
Improve base backup protocol documentation Document that the tablespace sizes are in units of kilobytes. Make the pg_basebackup source code a bit clearer about this, too. Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
totalsize_kb += atol(PQgetvalue(res, i, 2));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Verify tablespace directories are empty. Don't bother with the
* first once since it can be relocated, and it will be checked before
* we do anything anyway.
*/
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
if (format == 'p' && !PQgetisnull(res, i, 1))
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
{
More unconstify use Replace casts whose only purpose is to cast away const with the unconstify() macro. Discussion: https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
2019-01-29 01:16:24 +01:00
char *path = unconstify(char *, get_tablespace_mapping(PQgetvalue(res, i, 1)));
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(path, &made_tablespace_dirs, &found_tablespace_dirs);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
/*
* When writing to stdout, require a single tablespace
*/
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
writing_to_stdout = format == 't' && strcmp(basedir, "-") == 0;
if (writing_to_stdout && PQntuples(res) > 1)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("can only write single tablespace to stdout, database has %d",
PQntuples(res));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* If we're streaming WAL, start the streaming session before we start
* receiving the actual data chunks.
*/
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (includewal == STREAM_WAL)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("starting background WAL receiver");
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
StartLogStreamer(xlogstart, starttli, sysidentifier);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Start receiving chunks
*/
for (i = 0; i < PQntuples(res); i++)
{
if (format == 't')
ReceiveTarFile(conn, res, i);
else
ReceiveAndUnpackTarFile(conn, res, i);
} /* Loop over all tablespaces */
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
/*
* Now receive backup manifest, if appropriate.
*
* If we're writing a tarfile to stdout, ReceiveTarFile will have already
* processed the backup manifest and included it in the output tarfile.
* Such a configuration doesn't allow for writing multiple files.
*
* If we're talking to an older server, it won't send a backup manifest,
* so don't try to receive one.
*/
if (!writing_to_stdout && manifest)
ReceiveBackupManifest(conn);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (showprogress)
Fix printing last progress report line in client programs. A number of client programs have a "--progress" option that when printing to a TTY, updates the current line by printing a '\r' and overwriting it. After the last line, '\n' needs to be printed to move the cursor to the next line. pg_basebackup and pgbench got this right, but pg_rewind and pg_checksums were slightly wrong. pg_rewind printed the newline to stdout instead of stderr, and pg_checksums printed the newline even when not printing to a TTY. Fix them, and also add a 'finished' argument to pg_basebackup's progress_report() function, to keep it consistent with the other programs. Backpatch to v12. pg_rewind's newline was broken with the logging changes in commit cc8d415117 in v12, and pg_checksums was introduced in v12. Discussion: https://www.postgresql.org/message-id/82b539e5-ae33-34b0-1aee-22b3379fd3eb@iki.fi
2020-08-17 08:27:29 +02:00
progress_report(PQntuples(res), NULL, true, true);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQclear(res);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Get the stop position
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not get write-ahead log end position from server: %s",
PQerrorMessage(conn));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
}
if (PQntuples(res) != 1)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("no write-ahead log end position returned from server");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
}
Prevent potential overruns of fixed-size buffers. Coverity identified a number of places in which it couldn't prove that a string being copied into a fixed-size buffer would fit. We believe that most, perhaps all of these are in fact safe, or are copying data that is coming from a trusted source so that any overrun is not really a security issue. Nonetheless it seems prudent to forestall any risk by using strlcpy() and similar functions. Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports. In addition, fix a potential null-pointer-dereference crash in contrib/chkpass. The crypt(3) function is defined to return NULL on failure, but chkpass.c didn't check for that before using the result. The main practical case in which this could be an issue is if libc is configured to refuse to execute unapproved hashing algorithms (e.g., "FIPS mode"). This ideally should've been a separate commit, but since it touches code adjacent to one of the buffer overrun changes, I included it in this commit to avoid last-minute merge issues. This issue was reported by Honza Horak. Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 17:20:21 +01:00
strlcpy(xlogend, PQgetvalue(res, 0, 0), sizeof(xlogend));
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (verbose && includewal != NO_WAL)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("write-ahead log end point: %s", xlogend);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
PQclear(res);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
const char *sqlstate = PQresultErrorField(res, PG_DIAG_SQLSTATE);
if (sqlstate &&
strcmp(sqlstate, ERRCODE_DATA_CORRUPTED) == 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("checksum error occurred");
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
checksum_failure = true;
}
else
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("final receive failed: %s",
PQerrorMessage(conn));
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
}
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (bgchild > 0)
{
#ifndef WIN32
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
int status;
Modernize our code for looking up descriptive strings for Unix signals. At least as far back as the 2008 spec, POSIX has defined strsignal(3) for looking up descriptive strings for signal numbers. We hadn't gotten the word though, and were still using the crufty old sys_siglist array, which is in no standard even though most Unixen provide it. Aside from not being formally standards-compliant, this was just plain ugly because it involved #ifdef's at every place using the code. To eliminate the #ifdef's, create a portability function pg_strsignal, which wraps strsignal(3) if available and otherwise falls back to sys_siglist[] if available. The set of Unixen with neither API is probably empty these days, but on any platform with neither, you'll just get "unrecognized signal". All extant callers print the numeric signal number too, so no need to work harder than that. Along the way, upgrade pg_basebackup's child-error-exit reporting to match the rest of the system. Discussion: https://postgr.es/m/25758.1544983503@sss.pgh.pa.us
2018-12-17 01:38:57 +01:00
pid_t r;
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#else
DWORD status;
pgindent run for 9.6
2016-06-10 00:02:36 +02:00
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
/*
* get a pointer sized version of bgchild to avoid warnings about
* casting to a different size on WIN64.
*/
intptr_t bgchild_handle = bgchild;
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
uint32 hi,
lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#endif
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("waiting for background process to finish streaming ...");
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#ifndef WIN32
Remove now redundant pgpipe code.
2012-03-29 05:24:07 +02:00
if (write(bgpipe[1], xlogend, strlen(xlogend)) != strlen(xlogend))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("could not send command to background pipe: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
/* Just wait for the background process to exit */
r = waitpid(bgchild, &status, 0);
Modernize our code for looking up descriptive strings for Unix signals. At least as far back as the 2008 spec, POSIX has defined strsignal(3) for looking up descriptive strings for signal numbers. We hadn't gotten the word though, and were still using the crufty old sys_siglist array, which is in no standard even though most Unixen provide it. Aside from not being formally standards-compliant, this was just plain ugly because it involved #ifdef's at every place using the code. To eliminate the #ifdef's, create a portability function pg_strsignal, which wraps strsignal(3) if available and otherwise falls back to sys_siglist[] if available. The set of Unixen with neither API is probably empty these days, but on any platform with neither, you'll just get "unrecognized signal". All extant callers print the numeric signal number too, so no need to work harder than that. Along the way, upgrade pg_basebackup's child-error-exit reporting to match the rest of the system. Discussion: https://postgr.es/m/25758.1544983503@sss.pgh.pa.us
2018-12-17 01:38:57 +01:00
if (r == (pid_t) -1)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not wait for child process: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
if (r != bgchild)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("child %d died, expected %d", (int) r, (int) bgchild);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Modernize our code for looking up descriptive strings for Unix signals. At least as far back as the 2008 spec, POSIX has defined strsignal(3) for looking up descriptive strings for signal numbers. We hadn't gotten the word though, and were still using the crufty old sys_siglist array, which is in no standard even though most Unixen provide it. Aside from not being formally standards-compliant, this was just plain ugly because it involved #ifdef's at every place using the code. To eliminate the #ifdef's, create a portability function pg_strsignal, which wraps strsignal(3) if available and otherwise falls back to sys_siglist[] if available. The set of Unixen with neither API is probably empty these days, but on any platform with neither, you'll just get "unrecognized signal". All extant callers print the numeric signal number too, so no need to work harder than that. Along the way, upgrade pg_basebackup's child-error-exit reporting to match the rest of the system. Discussion: https://postgr.es/m/25758.1544983503@sss.pgh.pa.us
2018-12-17 01:38:57 +01:00
if (status != 0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("%s", wait_result_to_str(status));
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
/* Exited normally, we're happy! */
#else /* WIN32 */
/*
* On Windows, since we are in the same process, we can just store the
* value directly in the variable, and then set the flag that says
* it's there.
*/
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
if (sscanf(xlogend, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not parse write-ahead log location \"%s\"",
xlogend);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
xlogendptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
InterlockedIncrement(&has_xlogendptr);
/* First wait for the thread to exit */
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
if (WaitForSingleObjectEx((HANDLE) bgchild_handle, INFINITE, FALSE) !=
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
WAIT_OBJECT_0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
_dosmaperr(GetLastError());
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not wait for child thread: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
if (GetExitCodeThread((HANDLE) bgchild_handle, &status) == 0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
_dosmaperr(GetLastError());
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not get child thread exit status: %m");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
if (status != 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("child thread exited with error %u",
(unsigned int) status);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
/* Exited normally, we're happy */
#endif
}
Integrate recovery.conf into postgresql.conf recovery.conf settings are now set in postgresql.conf (or other GUC sources). Currently, all the affected settings are PGC_POSTMASTER; this could be refined in the future case by case. Recovery is now initiated by a file recovery.signal. Standby mode is initiated by a file standby.signal. The standby_mode setting is gone. If a recovery.conf file is found, an error is issued. The trigger_file setting has been renamed to promote_trigger_file as part of the move. The documentation chapter "Recovery Configuration" has been integrated into "Server Configuration". pg_basebackup -R now appends settings to postgresql.auto.conf and creates a standby.signal file. Author: Fujii Masao <masao.fujii@gmail.com> Author: Simon Riggs <simon@2ndquadrant.com> Author: Abhijit Menon-Sen <ams@2ndquadrant.com> Author: Sergei Kornilov <sk@zsrv.org> Discussion: https://www.postgresql.org/message-id/flat/607741529606767@web3g.yandex.ru/
2018-11-25 16:31:16 +01:00
/* Free the configuration file contents */
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
destroyPQExpBuffer(recoveryconfcontents);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* End of copy data. Final result is already checked inside the loop.
*/
Check number of fields in IDENTIFY_SYSTEM response Jaime Casanova
2012-01-20 13:57:02 +01:00
PQclear(res);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQfinish(conn);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
conn = NULL;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
/*
* Make data persistent on disk once backup is completed. For tar format
Delay fsyncs of pg_basebackup until the end of backup Since the addition of fsync requests in bc34223 to make base backup data consistent on disk once pg_basebackup finishes, each tablespace tar file is individually flushed once completed, with an additional flush of the parent directory when the base backup finishes. While holding a connection to the server, a fsync request taking a long time may cause a failure of the base backup, which is annoying for any integration. A recent example of breakage can involve tcp_user_timeout, but wal_sender_timeout can cause similar problems. While reviewing the code, there was a second issue causing too many fsync requests to be done for the same WAL data. As recursive fsyncs are done at the end of the backup for both the plain and tar formats from the base target directory where everything is written, it is fine to disable fsyncs when fetching or streaming WAL. Reported-by: Ryohei Takahashi Author: Michael Paquier Reviewed-by: Ryohei Takahashi Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com Backpatch-through: 10
2019-09-04 06:21:11 +02:00
* sync the parent directory and all its contents as each tar file was not
* synced after being completed. In plain format, all the data of the
* base directory is synced, taking into account all the tablespaces.
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
* Errors are not considered fatal.
*/
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
if (do_sync)
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
{
Add verbosity to pg_basebackup for sync This is useful to know when the data copy has been finished. The current situation can be confusing for users as the last message is "waiting for background process to finish streaming", so it looks like this is taking time but the final sync is instead. Author: Jeff Janes Discussion: https://postgr.es/m/CAMkU=1ypeoMJ=tFBG8vP13sxEtXd4Pm_x1SqsJdW_RvzpcvN=A@mail.gmail.com
2018-07-29 00:53:11 +02:00
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("syncing data to disk ...");
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
if (format == 't')
{
if (strcmp(basedir, "-") != 0)
Delay fsyncs of pg_basebackup until the end of backup Since the addition of fsync requests in bc34223 to make base backup data consistent on disk once pg_basebackup finishes, each tablespace tar file is individually flushed once completed, with an additional flush of the parent directory when the base backup finishes. While holding a connection to the server, a fsync request taking a long time may cause a failure of the base backup, which is annoying for any integration. A recent example of breakage can involve tcp_user_timeout, but wal_sender_timeout can cause similar problems. While reviewing the code, there was a second issue causing too many fsync requests to be done for the same WAL data. As recursive fsyncs are done at the end of the backup for both the plain and tar formats from the base target directory where everything is written, it is fine to disable fsyncs when fetching or streaming WAL. Reported-by: Ryohei Takahashi Author: Michael Paquier Reviewed-by: Ryohei Takahashi Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com Backpatch-through: 10
2019-09-04 06:21:11 +02:00
(void) fsync_dir_recurse(basedir);
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
}
else
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
(void) fsync_pgdata(basedir, serverVersion);
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
}
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
}
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
/*
* After synchronizing data to disk, perform a durable rename of
* backup_manifest.tmp to backup_manifest, if we wrote such a file. This
* way, a failure or system crash before we reach this point will leave us
* without a backup_manifest file, decreasing the chances that a directory
* we leave behind will be mistaken for a valid backup.
*/
if (!writing_to_stdout && manifest)
{
char tmp_filename[MAXPGPATH];
char filename[MAXPGPATH];
if (verbose)
pg_log_info("renaming backup_manifest.tmp to backup_manifest");
snprintf(tmp_filename, MAXPGPATH, "%s/backup_manifest.tmp", basedir);
snprintf(filename, MAXPGPATH, "%s/backup_manifest", basedir);
/* durable_rename emits its own log message in case of failure */
if (durable_rename(tmp_filename, filename) != 0)
exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (verbose)
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_info("base backup completed");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
int
main(int argc, char **argv)
{
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
{"format", required_argument, NULL, 'F'},
{"checkpoint", required_argument, NULL, 'c'},
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
{"create-slot", no_argument, NULL, 'C'},
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
{"max-rate", required_argument, NULL, 'r'},
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{"write-recovery-conf", no_argument, NULL, 'R'},
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
{"slot", required_argument, NULL, 'S'},
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
{"tablespace-mapping", required_argument, NULL, 'T'},
Rename command line options for ongoing xlog -> wal conversion. initdb and pg_basebackup now have a --waldir option rather --xlogdir, and pg_basebackup now has --wal-method rather than --xlog-method.
2017-02-09 22:42:51 +01:00
{"wal-method", required_argument, NULL, 'X'},
Add pg_basebackup -z option for compression with default level
2011-05-30 00:02:02 +02:00
{"gzip", no_argument, NULL, 'z'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"compress", required_argument, NULL, 'Z'},
{"label", required_argument, NULL, 'l'},
initdb pg_basebackup: Rename --noxxx options to --no-xxx --noclean and --nosync were the only options spelled without a hyphen, so change this for consistency with other options. The options in pg_basebackup have not been in a release, so we just rename them. For initdb, we retain the old variants. Vik Fearing and me
2016-10-19 18:00:00 +02:00
{"no-clean", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
{"dbname", required_argument, NULL, 'd'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"host", required_argument, NULL, 'h'},
{"port", required_argument, NULL, 'p'},
{"username", required_argument, NULL, 'U'},
{"no-password", no_argument, NULL, 'w'},
{"password", no_argument, NULL, 'W'},
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
{"status-interval", required_argument, NULL, 's'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"verbose", no_argument, NULL, 'v'},
{"progress", no_argument, NULL, 'P'},
Rename command line options for ongoing xlog -> wal conversion. initdb and pg_basebackup now have a --waldir option rather --xlogdir, and pg_basebackup now has --wal-method rather than --xlog-method.
2017-02-09 22:42:51 +01:00
{"waldir", required_argument, NULL, 1},
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
{"no-slot", no_argument, NULL, 2},
pg_basebackup: Remove short option -k -k meant --no-verify-checksums, which is the opposite of what initdb uses -k for. After discussion, a short option does not seem necessary, so just keep the long option. Discussion: https://www.postgresql.org/message-id/flat/d510f8aa-19e1-d06e-7630-ad27f7441d68%402ndquadrant.com
2018-05-21 16:01:49 +02:00
{"no-verify-checksums", no_argument, NULL, 3},
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
{"no-estimate-size", no_argument, NULL, 4},
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
{"no-manifest", no_argument, NULL, 5},
{"manifest-force-encode", no_argument, NULL, 6},
{"manifest-checksums", required_argument, NULL, 7},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{NULL, 0, NULL, 0}
};
int c;
int option_index;
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_logging_init(argv[0]);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname = get_progname(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_basebackup"));
if (argc > 1)
{
if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
{
usage();
exit(0);
}
else if (strcmp(argv[1], "-V") == 0
|| strcmp(argv[1], "--version") == 0)
{
puts("pg_basebackup (PostgreSQL) " PG_VERSION);
exit(0);
}
}
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
atexit(cleanup_directories_atexit);
Code review for backup manifest. This commit prevents pg_basebackup from receiving backup_manifest file when --no-manifest is specified. Previously, when pg_basebackup was writing a tarfile to stdout, it tried to receive backup_manifest file even when --no-manifest was specified, and reported an error. Also remove unused -m option from pg_basebackup. Also fix typo in BASE_BACKUP command documentation. Author: Fujii Masao Reviewed-by: Michael Paquier, Robert Haas Discussion: https://postgr.es/m/01e3ed3a-8729-5aaa-ca84-e60e3ca59db8@oss.nttdata.com
2020-04-15 04:15:12 +02:00
while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvP",
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
long_options, &option_index)) != -1)
{
switch (c)
{
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
case 'C':
create_slot = true;
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'D':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
basedir = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'F':
if (strcmp(optarg, "p") == 0 || strcmp(optarg, "plain") == 0)
format = 'p';
else if (strcmp(optarg, "t") == 0 || strcmp(optarg, "tar") == 0)
format = 't';
else
{
Remove redundant newlines from error messages These are no longer needed/allowed with the new logging API.
2019-07-03 00:18:43 +02:00
pg_log_error("invalid output format \"%s\", must be \"plain\" or \"tar\"",
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
exit(1);
}
break;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
case 'r':
maxrate = parse_max_rate(optarg);
break;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
case 'R':
writerecoveryconf = true;
break;
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
case 'S':
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/*
* When specifying replication slot name, use a permanent
* slot.
*/
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
replication_slot = pg_strdup(optarg);
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
temp_replication_slot = false;
break;
case 2:
no_slot = true;
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
break;
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
case 'T':
tablespace_list_append(optarg);
break;
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
case 'X':
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
if (strcmp(optarg, "n") == 0 ||
strcmp(optarg, "none") == 0)
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = NO_WAL;
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
}
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
else if (strcmp(optarg, "f") == 0 ||
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
strcmp(optarg, "fetch") == 0)
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = FETCH_WAL;
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
else if (strcmp(optarg, "s") == 0 ||
strcmp(optarg, "stream") == 0)
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = STREAM_WAL;
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
else
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("invalid wal-method option \"%s\", must be \"fetch\", \"stream\", or \"none\"",
optarg);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
exit(1);
}
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
break;
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
case 1:
xlog_dir = pg_strdup(optarg);
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'l':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
label = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
case 'n':
noclean = true;
break;
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
case 'N':
do_sync = false;
break;
Add pg_basebackup -z option for compression with default level
2011-05-30 00:02:02 +02:00
case 'z':
#ifdef HAVE_LIBZ
compresslevel = Z_DEFAULT_COMPRESSION;
#else
compresslevel = 1; /* will be rejected below */
#endif
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'Z':
Unify parsing logic for command-line integer options Most of the integer options for command-line binaries now make use of a single routine able to do the job, fixing issues with the detection of sloppy values caused for example by the use of atoi(), that fails on strings beginning with numerical characters with junk trailing characters. This commit cuts down the number of strings requiring translation by 26 per my count, switching the code to have two error types for invalid and out-of-range values instead. Much more could be done here, with float or even int64 options, but int32 was the most appealing case as it is possible to rely on strtol() to do the job reliably. Note that there are some exceptions for now, like pg_ctl or pg_upgrade that use their own logging logic. A couple of negative TAP tests required some adjustments for the new errors generated. pg_dump and pg_restore tracked the maximum number of parallel jobs within the option parsing. The code is refactored a bit to track that in the code dedicated to parallelism instead. Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: David Rowley, Álvaro Herrera Discussion: https://postgr.es/m/CALj2ACXqdG9WhqVoJ9zYf-iZt7sgK7Szv5USs=he6NnWQ2ofTA@mail.gmail.com
2021-07-24 11:35:03 +02:00
if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
&compresslevel))
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
exit(1);
break;
case 'c':
Use pg_strcasecmp instead of strcasecmp for portability Per buildfarm.
2011-01-23 17:35:02 +01:00
if (pg_strcasecmp(optarg, "fast") == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fastcheckpoint = true;
Use pg_strcasecmp instead of strcasecmp for portability Per buildfarm.
2011-01-23 17:35:02 +01:00
else if (pg_strcasecmp(optarg, "spread") == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fastcheckpoint = false;
else
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("invalid checkpoint argument \"%s\", must be \"fast\" or \"spread\"",
optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
exit(1);
}
break;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
case 'd':
connection_string = pg_strdup(optarg);
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'h':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbhost = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'p':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbport = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'U':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbuser = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'w':
dbgetpassword = -1;
break;
case 'W':
dbgetpassword = 1;
break;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
case 's':
Unify parsing logic for command-line integer options Most of the integer options for command-line binaries now make use of a single routine able to do the job, fixing issues with the detection of sloppy values caused for example by the use of atoi(), that fails on strings beginning with numerical characters with junk trailing characters. This commit cuts down the number of strings requiring translation by 26 per my count, switching the code to have two error types for invalid and out-of-range values instead. Much more could be done here, with float or even int64 options, but int32 was the most appealing case as it is possible to rely on strtol() to do the job reliably. Note that there are some exceptions for now, like pg_ctl or pg_upgrade that use their own logging logic. A couple of negative TAP tests required some adjustments for the new errors generated. pg_dump and pg_restore tracked the maximum number of parallel jobs within the option parsing. The code is refactored a bit to track that in the code dedicated to parallelism instead. Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: David Rowley, Álvaro Herrera Discussion: https://postgr.es/m/CALj2ACXqdG9WhqVoJ9zYf-iZt7sgK7Szv5USs=he6NnWQ2ofTA@mail.gmail.com
2021-07-24 11:35:03 +02:00
if (!option_parse_int(optarg, "-s/--status-interval", 0,
INT_MAX / 1000,
&standby_message_timeout))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
exit(1);
Unify parsing logic for command-line integer options Most of the integer options for command-line binaries now make use of a single routine able to do the job, fixing issues with the detection of sloppy values caused for example by the use of atoi(), that fails on strings beginning with numerical characters with junk trailing characters. This commit cuts down the number of strings requiring translation by 26 per my count, switching the code to have two error types for invalid and out-of-range values instead. Much more could be done here, with float or even int64 options, but int32 was the most appealing case as it is possible to rely on strtol() to do the job reliably. Note that there are some exceptions for now, like pg_ctl or pg_upgrade that use their own logging logic. A couple of negative TAP tests required some adjustments for the new errors generated. pg_dump and pg_restore tracked the maximum number of parallel jobs within the option parsing. The code is refactored a bit to track that in the code dedicated to parallelism instead. Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: David Rowley, Álvaro Herrera Discussion: https://postgr.es/m/CALj2ACXqdG9WhqVoJ9zYf-iZt7sgK7Szv5USs=he6NnWQ2ofTA@mail.gmail.com
2021-07-24 11:35:03 +02:00
standby_message_timeout *= 1000;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'v':
verbose++;
break;
case 'P':
showprogress = true;
break;
pg_basebackup: Remove short option -k -k meant --no-verify-checksums, which is the opposite of what initdb uses -k for. After discussion, a short option does not seem necessary, so just keep the long option. Discussion: https://www.postgresql.org/message-id/flat/d510f8aa-19e1-d06e-7630-ad27f7441d68%402ndquadrant.com
2018-05-21 16:01:49 +02:00
case 3:
Validate page level checksums in base backups When base backups are run over the replication protocol (for example using pg_basebackup), verify the checksums of all data blocks if checksums are enabled. If checksum failures are encountered, log them as warnings but don't abort the backup. This becomes the default behaviour in pg_basebackup (provided checksums are enabled on the server), so add a switch (-k) to disable the checks if necessary. Author: Michael Banck Reviewed-By: Magnus Hagander, David Steele Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
verify_checksums = false;
break;
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
case 4:
estimatesize = false;
break;
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
case 5:
manifest = false;
break;
case 6:
manifest_force_encode = true;
break;
case 7:
manifest_checksums = pg_strdup(optarg);
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
default:
/*
* getopt_long already emitted a complaint
*/
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
/*
* Any non-option arguments?
*/
if (optind < argc)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("too many command-line arguments (first is \"%s\")",
argv[optind]);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Required arguments
*/
if (basedir == NULL)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("no target directory specified");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Mutually exclusive arguments
*/
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (format == 'p' && compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("only tar mode backups can be compressed");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (format == 't' && includewal == STREAM_WAL && strcmp(basedir, "-") == 0)
Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me.
2016-12-21 12:27:37 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("cannot stream write-ahead logs in tar mode to stdout");
Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me.
2016-12-21 12:27:37 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Fix incorrect comparison due to bad merge Noted by Fujii Masao
2017-01-16 18:20:57 +01:00
if (replication_slot && includewal != STREAM_WAL)
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("replication slots can only be used with WAL streaming");
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
if (no_slot)
{
if (replication_slot)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("--no-slot cannot be used with slot name");
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
temp_replication_slot = false;
}
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
if (create_slot)
{
if (!replication_slot)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("%s needs a slot to be specified using --slot",
"--create-slot");
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
if (no_slot)
{
Error message refactoring Take some untranslatable things out of the message and replace by format placeholders, to reduce translatable strings and reduce translation mistakes.
2020-06-15 08:22:52 +02:00
pg_log_error("%s and %s are incompatible options",
"--create-slot", "--no-slot");
pg_basebackup: Add option to create replication slot When requesting a particular replication slot, the new pg_basebackup option -C/--create-slot creates it before starting to replicate from it. Further refactor the slot creation logic to include the temporary slot creation logic into the same function. Add new arguments is_temporary and preserve_wal to CreateReplicationSlot(). Print in --verbose mode that a slot has been created. Author: Michael Banck <michael.banck@credativ.de>
2017-09-26 22:07:52 +02:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
Remove useless empty string initializations This coding style probably stems from the days of shell scripts. Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-08-31 04:28:36 +02:00
if (xlog_dir)
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
{
if (format != 'p')
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("WAL directory location can only be specified in plain mode");
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/* clean up xlog directory name, check it's absolute */
canonicalize_path(xlog_dir);
if (!is_absolute_path(xlog_dir))
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("WAL directory location must be an absolute path");
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifndef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("this build does not support compression");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
exit(1);
}
#endif
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
if (showprogress && !estimatesize)
{
Error message refactoring Take some untranslatable things out of the message and replace by format placeholders, to reduce translatable strings and reduce translation mistakes.
2020-06-15 08:22:52 +02:00
pg_log_error("%s and %s are incompatible options",
"--progress", "--no-estimate-size");
Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com
2020-03-19 09:09:00 +01:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
if (!manifest && manifest_checksums != NULL)
{
Error message refactoring Take some untranslatable things out of the message and replace by format placeholders, to reduce translatable strings and reduce translation mistakes.
2020-06-15 08:22:52 +02:00
pg_log_error("%s and %s are incompatible options",
"--no-manifest", "--manifest-checksums");
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
if (!manifest && manifest_force_encode)
{
Error message refactoring Take some untranslatable things out of the message and replace by format placeholders, to reduce translatable strings and reduce translation mistakes.
2020-06-15 08:22:52 +02:00
pg_log_error("%s and %s are incompatible options",
"--no-manifest", "--manifest-force-encode");
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consistent, and (3) a checksum for the manifest itself. By default, we use CRC-32C when checksumming data files, because we are trying to detect corruption and user error, not foil an adversary. However, pg_basebackup and the server-side BASE_BACKUP command now have options to select a different algorithm, so users wanting a cryptographic hash function can select SHA-224, SHA-256, SHA-384, or SHA-512. Users not wanting file checksums at all can disable them, or disable generating of the backup manifest altogether. Using a cryptographic hash function in place of CRC-32C consumes significantly more CPU cycles, which may slow down backups in some cases. A new tool called pg_validatebackup can validate a backup against the manifest. If no checksums are present, it can still check that the right files exist and that they have the expected sizes. If checksums are present, it can also verify that each file has the expected checksum. Additionally, it calls pg_waldump to verify that the expected WAL files are present and parseable. Only plain format backups can be validated directly, but tar format backups can be validated after extracting them. Robert Haas, with help, ideas, review, and testing from David Steele, Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan Chalke, Amit Kapila, Andres Freund, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/* connection in replication mode to server */
conn = GetConnection();
if (!conn)
{
/* Error message already written in GetConnection() */
exit(1);
}
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
atexit(disconnect_atexit);
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
Allow group access on PGDATA Allow the cluster to be optionally init'd with read access for the group. This means a relatively non-privileged user can perform a backup of the cluster without requiring write privileges, which enhances security. The mode of PGDATA is used to determine whether group permissions are enabled for directory and file creates. This method was chosen as it's simple and works well for the various utilities that write into PGDATA. Changing the mode of PGDATA manually will not automatically change the mode of all the files contained therein. If the user would like to enable group access on an existing cluster then changing the mode of all the existing files will be required. Note that pg_upgrade will automatically change the mode of all migrated files if the new cluster is init'd with the -g option. Tests are included for the backend and all the utilities which operate on the PG data directory to ensure that the correct mode is set based on the data directory permissions. Author: David Steele <david@pgmasters.net> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
/*
* Set umask so that directories/files are created with the same
* permissions as directories/files in the source data directory.
*
* pg_mode_mask is set to owner-only by default and then updated in
* GetConnection() where we get the mode from the server-side with
* RetrieveDataDirCreatePerm() and then call SetDataDirectoryCreatePerm().
*/
umask(pg_mode_mask);
Disable silently generation of manifests with servers <= 12 in pg_basebackup Since 0d8c9c1, pg_basebackup would generate an error if connected to a backend version older than 12 where backup manifests are not supported. Avoiding this error is possible by using the --no-manifest option. This error handling could be confusing for some users, where patching a backup script that interacts with multiple backend versions would cause the addition of --no-manifest to potentially not generate a backup manifest even for Postgres 13 and newer versions. As we want to encourage the use of backup manifests as much as possible, this commit silently disables manifests where not supported, instead of generating an error. While on it, rework a bit the code to make it more consistent with the surroundings when generating the BASE_BACKUP command. Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro Herrera, Kyotaro Horiguchi, Tom Lane, David Steele, and me. Author: Michael Paquier Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz
2020-04-16 06:57:07 +02:00
/* Backup manifests are supported in 13 and newer versions */
if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_MANIFESTS)
manifest = false;
Allow group access on PGDATA Allow the cluster to be optionally init'd with read access for the group. This means a relatively non-privileged user can perform a backup of the cluster without requiring write privileges, which enhances security. The mode of PGDATA is used to determine whether group permissions are enabled for directory and file creates. This method was chosen as it's simple and works well for the various utilities that write into PGDATA. Changing the mode of PGDATA manually will not automatically change the mode of all the files contained therein. If the user would like to enable group access on an existing cluster then changing the mode of all the existing files will be required. Note that pg_upgrade will automatically change the mode of all migrated files if the new cluster is init'd with the -g option. Tests are included for the backend and all the utilities which operate on the PG data directory to ensure that the correct mode is set based on the data directory permissions. Author: David Steele <david@pgmasters.net> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 23:45:39 +02:00
/*
* Verify that the target directory exists, or create it. For plaintext
* backups, always require the directory. For tar backups, require it
* unless we are writing to stdout.
*/
if (format == 'p' || strcmp(basedir, "-") != 0)
verify_dir_is_empty_or_create(basedir, &made_new_pgdata, &found_existing_pgdata);
Make WAL segment size configurable at initdb time. For performance reasons a larger segment size than the default 16MB can be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier to write scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk less frequently. But at the same time large segment size are disadvantageous for smaller databases. So far the segment size had to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load. Therefore change it to a initdb time setting. This includes a breaking changes to the xlogreader.h API, which now requires the current segment size to be configured. For that and similar reasons a number of binaries had to be taught how to recognize the current segment size. Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-20 07:03:48 +02:00
/* determine remote server's xlog segment size */
if (!RetrieveWalSegSize(conn))
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Make WAL segment size configurable at initdb time. For performance reasons a larger segment size than the default 16MB can be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier to write scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk less frequently. But at the same time large segment size are disadvantageous for smaller databases. So far the segment size had to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load. Therefore change it to a initdb time setting. This includes a breaking changes to the xlogreader.h API, which now requires the current segment size to be configured. For that and similar reasons a number of binaries had to be taught how to recognize the current segment size. Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-20 07:03:48 +02:00
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
/* Create pg_wal symlink, if required */
Remove useless empty string initializations This coding style probably stems from the days of shell scripts. Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-08-31 04:28:36 +02:00
if (xlog_dir)
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
{
char *linkloc;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(xlog_dir, &made_new_xlogdir, &found_existing_xlogdir);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/*
* Form name of the place where the symlink must go. pg_xlog has been
* renamed to pg_wal in post-10 clusters.
*/
linkloc = psprintf("%s/%s", basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
#ifdef HAVE_SYMLINK
if (symlink(xlog_dir, linkloc) != 0)
{
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("could not create symbolic link \"%s\": %m", linkloc);
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
}
#else
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_error("symlinks are not supported on this platform");
pg_basebackup: Use atexit() Instead of using our custom disconnect_and_exit(), just register the desired cleanup using atexit() and use the standard exit() to leave the program. Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/flat/ec4135ba-84e9-28bf-b584-0e78d47448d5@2ndquadrant.com/
2018-12-29 13:21:47 +01:00
exit(1);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
#endif
free(linkloc);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
BaseBackup();
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
success = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return 0;
}