rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/bin/pg_basebackup/pg_basebackup.c

2430 lines
60 KiB
C
Raw Normal View History

Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*-------------------------------------------------------------------------
*
* pg_basebackup.c - receive a base backup using streaming replication protocol
*
* Author: Magnus Hagander <magnus@hagander.net>
*
Update copyright via script for 2017
2017-01-03 19:48:53 +01:00
* Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*
* IDENTIFICATION
* src/bin/pg_basebackup/pg_basebackup.c
*-------------------------------------------------------------------------
*/
Make xlog_internal.h includable in frontend context. This makes unnecessary the ugly hack used to #include postgres.h in pg_basebackup. Based on Alvaro Herrera's patch
2012-12-13 13:59:13 +01:00
#include "postgres_fe.h"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#include <unistd.h>
#include <dirent.h>
#include <sys/stat.h>
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include <sys/wait.h>
Add missing include, required on some platforms Noted by the buildfarm and Andres Freund
2014-02-12 20:04:13 +01:00
#include <signal.h>
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
#include <time.h>
Include <sys/select.h> where needed <sys/select.h> is required by POSIX.1-2001 to get the prototype of select(2), but nearly no systems enforce that because older standards let you get away with including some other headers. Recent OpenBSD hacking has removed that frail touch of friendliness, however, which broke some compiles; fix all the way back to 9.1 by adding the required standard. Only vacuumdb.c was reported to fail, but it seems easier to fix the whole lot in a fell swoop. Per bug #14334 by Sean Farrell.
2016-09-27 06:05:21 +02:00
#ifdef HAVE_SYS_SELECT_H
#include <sys/select.h>
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
#include "common/file_utils.h"
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
#include "common/string.h"
Retire escapeConnectionParameter(). It is redundant with appendConnStrVal(), which became an extern function in commit 41f18f021a0882eccbeca62e2ed4b66c6b96e9c9. This changes the handling of out-of-memory and of certain inputs for which quoting is optional, but pg_basebackup has no need for unusual treatment thereof.
2016-08-22 04:05:57 +02:00
#include "fe_utils/string_utils.h"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#include "getopt_long.h"
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
#include "libpq-fe.h"
#include "pqexpbuffer.h"
#include "pgtar.h"
#include "pgtime.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include "receivelog.h"
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
#include "replication/basebackup.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include "streamutil.h"
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
typedef struct TablespaceListCell
{
struct TablespaceListCell *next;
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
char old_dir[MAXPGPATH];
char new_dir[MAXPGPATH];
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
} TablespaceListCell;
typedef struct TablespaceList
{
TablespaceListCell *head;
TablespaceListCell *tail;
} TablespaceList;
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/*
* pg_xlog has been renamed to pg_wal in version 10. This version number
* should be compared with PQserverVersion().
*/
#define MINIMUM_VERSION_FOR_PG_WAL 100000
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/*
* Temporary replication slots are supported from version 10.
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
/*
* Different ways to include WAL
*/
typedef enum
{
NO_WAL,
FETCH_WAL,
STREAM_WAL
} IncludeWal;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Global options */
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static char *basedir = NULL;
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
static TablespaceList tablespace_dirs = {NULL, NULL};
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
static char *xlog_dir = "";
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
static char format = 'p'; /* p(lain)/t(ar) */
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static char *label = "pg_basebackup base backup";
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool noclean = false;
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
static bool showprogress = false;
static int verbose = 0;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static int compresslevel = 0;
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
static IncludeWal includewal = STREAM_WAL;
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
static bool do_sync = true;
Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund. I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 10:27:30 +01:00
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
static pg_time_t last_progress_report = 0;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
static int32 maxrate = 0; /* no limit by default */
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
static char *replication_slot = NULL;
static bool temp_replication_slot = true;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool success = false;
static bool made_new_pgdata = false;
static bool found_existing_pgdata = false;
static bool made_new_xlogdir = false;
static bool found_existing_xlogdir = false;
static bool made_tablespace_dirs = false;
static bool found_tablespace_dirs = false;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Progress counters */
static uint64 totalsize;
static uint64 totaldone;
static int tablespacecount;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Pipe to communicate with background wal receiver process */
#ifndef WIN32
static int bgpipe[2] = {-1, -1};
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Handle to child process */
static pid_t bgchild = -1;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static bool in_log_streamer = false;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* End position for xlog streaming, empty string if unknown yet */
static XLogRecPtr xlogendptr;
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#ifndef WIN32
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
static int has_xlogendptr = 0;
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#else
static volatile LONG has_xlogendptr = 0;
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/* Contents of recovery.conf to be generated */
static PQExpBuffer recoveryconfcontents = NULL;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Function headers */
static void usage(void);
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
static void disconnect_and_exit(int code);
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
static void progress_report(int tablespacenum, const char *filename, bool force);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
static void GenerateRecoveryConf(PGconn *conn);
static void WriteRecoveryConf(void);
fix up a couple non-prototypes of the form foo() to be foo(void) -- found using -Wstrict-prototypes
2011-03-27 22:31:41 +02:00
static void BaseBackup(void);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
static const char *get_tablespace_mapping(const char *dir);
static void tablespace_list_append(const char *arg);
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
static void
cleanup_directories_atexit(void)
{
if (success || in_log_streamer)
return;
if (!noclean)
{
if (made_new_pgdata)
{
fprintf(stderr, _("%s: removing data directory \"%s\"\n"),
progname, basedir);
if (!rmtree(basedir, true))
fprintf(stderr, _("%s: failed to remove data directory\n"),
progname);
}
else if (found_existing_pgdata)
{
fprintf(stderr,
_("%s: removing contents of data directory \"%s\"\n"),
progname, basedir);
if (!rmtree(basedir, false))
fprintf(stderr, _("%s: failed to remove contents of data directory\n"),
progname);
}
if (made_new_xlogdir)
{
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: removing WAL directory \"%s\"\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname, xlog_dir);
if (!rmtree(xlog_dir, true))
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: failed to remove WAL directory\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname);
}
else if (found_existing_xlogdir)
{
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: removing contents of WAL directory \"%s\"\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname, xlog_dir);
if (!rmtree(xlog_dir, false))
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: failed to remove contents of WAL directory\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname);
}
}
else
{
if (made_new_pgdata || found_existing_pgdata)
fprintf(stderr,
_("%s: data directory \"%s\" not removed at user's request\n"),
progname, basedir);
if (made_new_xlogdir || found_existing_xlogdir)
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: WAL directory \"%s\" not removed at user's request\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname, xlog_dir);
}
if (made_tablespace_dirs || found_tablespace_dirs)
fprintf(stderr,
Add missing newline in message
2016-12-15 16:45:31 +01:00
_("%s: changes to tablespace directories will not be undone\n"),
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
progname);
}
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
static void
disconnect_and_exit(int code)
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
{
if (conn != NULL)
PQfinish(conn);
#ifndef WIN32
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
/*
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
* On windows, our background thread dies along with the process. But on
* Unix, if we have started a subprocess, we want to kill it off so it
* doesn't remain running trying to stream data.
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
*/
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
if (bgchild > 0)
Kill pg_basebackup background process when exiting If an error occurs in the foreground (backup) process of pg_basebackup, and we exit in a controlled way, the background process (streaming xlog process) would stay around and keep streaming.
2014-02-09 13:10:14 +01:00
kill(bgchild, SIGTERM);
#endif
exit(code);
}
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
/*
* Split argument into old_dir and new_dir and append to tablespace mapping
* list.
*/
static void
tablespace_list_append(const char *arg)
{
TablespaceListCell *cell = (TablespaceListCell *) pg_malloc0(sizeof(TablespaceListCell));
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
char *dst;
char *dst_ptr;
const char *arg_ptr;
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
dst_ptr = dst = cell->old_dir;
for (arg_ptr = arg; *arg_ptr; arg_ptr++)
{
if (dst_ptr - dst >= MAXPGPATH)
{
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
fprintf(stderr, _("%s: directory name too long\n"), progname);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
exit(1);
}
if (*arg_ptr == '\\' && *(arg_ptr + 1) == '=')
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
; /* skip backslash escaping = */
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
else if (*arg_ptr == '=' && (arg_ptr == arg || *(arg_ptr - 1) != '\\'))
{
if (*cell->new_dir)
{
fprintf(stderr, _("%s: multiple \"=\" signs in tablespace mapping\n"), progname);
exit(1);
}
else
dst = dst_ptr = cell->new_dir;
}
else
*dst_ptr++ = *arg_ptr;
}
if (!*cell->old_dir || !*cell->new_dir)
{
fprintf(stderr,
_("%s: invalid tablespace mapping format \"%s\", must be \"OLDDIR=NEWDIR\"\n"),
progname, arg);
exit(1);
}
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
/*
* This check isn't absolutely necessary. But all tablespaces are created
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
* with absolute directories, so specifying a non-absolute path here would
* just never match, possibly confusing users. It's also good to be
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
* consistent with the new_dir check.
*/
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
if (!is_absolute_path(cell->old_dir))
{
Misc message style and doc fixes. Euler Taveira
2014-05-15 13:49:11 +02:00
fprintf(stderr, _("%s: old directory is not an absolute path in tablespace mapping: %s\n"),
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
progname, cell->old_dir);
exit(1);
}
if (!is_absolute_path(cell->new_dir))
{
Misc message style and doc fixes. Euler Taveira
2014-05-15 13:49:11 +02:00
fprintf(stderr, _("%s: new directory is not an absolute path in tablespace mapping: %s\n"),
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
progname, cell->new_dir);
exit(1);
}
pg_basebackup: canonicalize old and new tablespace paths This avoids problems with double-slash-specified paths. Patch by Ian Barwick
2015-04-29 02:12:10 +02:00
canonicalize_path(cell->old_dir);
canonicalize_path(cell->new_dir);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
if (tablespace_dirs.tail)
tablespace_dirs.tail->next = cell;
else
tablespace_dirs.head = cell;
tablespace_dirs.tail = cell;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifdef HAVE_LIBZ
static const char *
Fix incorrect uses of gzFile gzFile is already a pointer, so code like gzFile *handle = gzopen(...) is wrong. This used to pass silently because gzFile used to be defined as void*, and you can assign a void* to a void**. But somewhere between zlib versions 1.2.3.4 and 1.2.6, the definition of gzFile was changed to struct gzFile_s *, and with that new definition this usage causes compiler warnings. So remove all those extra pointer decorations. There is a related issue in pg_backup_archiver.h, where FILE *FH; /* General purpose file handle */ is used throughout pg_dump as sometimes a real FILE* and sometimes a gzFile handle, which also causes warnings now. This is not yet fixed here, because it might need more code restructuring.
2012-03-02 21:30:01 +01:00
get_gz_error(gzFile gzf)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
int errnum;
const char *errmsg;
errmsg = gzerror(gzf, &errnum);
if (errnum == Z_ERRNO)
return strerror(errno);
else
return errmsg;
}
#endif
static void
usage(void)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
printf(_("%s takes a base backup of a running PostgreSQL server.\n\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions controlling the output:\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -D, --pgdata=DIRECTORY receive base backup into directory\n"));
printf(_(" -F, --format=p|t output format (plain (default), tar)\n"));
Preventive maintenance in advance of pgindent run. Reformat various places in which pgindent will make a mess, and fix a few small violations of coding style that I happened to notice while perusing the diffs from a pgindent dry run. There is one actual bug fix here: the need-to-enlarge-the-buffer code path in icu_convert_case was obviously broken. Perhaps it's unreachable in our usage? Or maybe this is just sadly undertested.
2017-05-17 02:36:35 +02:00
printf(_(" -r, --max-rate=RATE maximum transfer rate to transfer data directory\n"));
printf(_(" (in kB/s, or use suffix \"k\" or \"M\")\n"));
printf(_(" -R, --write-recovery-conf\n"));
printf(_(" write recovery.conf for replication\n"));
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
printf(_(" -S, --slot=SLOTNAME replication slot to use\n"));
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
printf(_(" --no-slot prevent creation of temporary replication slot\n"));
Preventive maintenance in advance of pgindent run. Reformat various places in which pgindent will make a mess, and fix a few small violations of coding style that I happened to notice while perusing the diffs from a pgindent dry run. There is one actual bug fix here: the need-to-enlarge-the-buffer code path in icu_convert_case was obviously broken. Perhaps it's unreachable in our usage? Or maybe this is just sadly undertested.
2017-05-17 02:36:35 +02:00
printf(_(" -T, --tablespace-mapping=OLDDIR=NEWDIR\n"));
printf(_(" relocate tablespace in OLDDIR to NEWDIR\n"));
printf(_(" -X, --wal-method=none|fetch|stream\n"));
printf(_(" include required WAL files with specified method\n"));
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
printf(_(" --waldir=WALDIR location for the write-ahead log directory\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -z, --gzip compress tar output\n"));
printf(_(" -Z, --compress=0-9 compress tar output with given compression level\n"));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
printf(_("\nGeneral options:\n"));
Preventive maintenance in advance of pgindent run. Reformat various places in which pgindent will make a mess, and fix a few small violations of coding style that I happened to notice while perusing the diffs from a pgindent dry run. There is one actual bug fix here: the need-to-enlarge-the-buffer code path in icu_convert_case was obviously broken. Perhaps it's unreachable in our usage? Or maybe this is just sadly undertested.
2017-05-17 02:36:35 +02:00
printf(_(" -c, --checkpoint=fast|spread\n"));
printf(_(" set fast or spread checkpointing\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -l, --label=LABEL set backup label\n"));
initdb pg_basebackup: Rename --noxxx options to --no-xxx --noclean and --nosync were the only options spelled without a hyphen, so change this for consistency with other options. The options in pg_basebackup have not been in a release, so we just rename them. For initdb, we retain the old variants. Vik Fearing and me
2016-10-19 18:00:00 +02:00
printf(_(" -n, --no-clean do not clean up after errors\n"));
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -P, --progress show progress information\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -?, --help show this help, then exit\n"));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
printf(_("\nConnection options:\n"));
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
printf(_(" -d, --dbname=CONNSTR connection string\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port number\n"));
Preventive maintenance in advance of pgindent run. Reformat various places in which pgindent will make a mess, and fix a few small violations of coding style that I happened to notice while perusing the diffs from a pgindent dry run. There is one actual bug fix here: the need-to-enlarge-the-buffer code path in icu_convert_case was obviously broken. Perhaps it's unreachable in our usage? Or maybe this is just sadly undertested.
2017-05-17 02:36:35 +02:00
printf(_(" -s, --status-interval=INTERVAL\n"));
printf(_(" time between status packets sent to server (in seconds)\n"));
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
printf(_(" -U, --username=NAME connect as specified database user\n"));
printf(_(" -w, --no-password never prompt for password\n"));
printf(_(" -W, --password force password prompt (should happen automatically)\n"));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
printf(_("\nReport bugs to <pgsql-bugs@postgresql.org>.\n"));
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
Fix base backup streaming xlog from standby When backing up from a standby server, the backup process will not automatically switch xlog segment. So we must accept a partially transferred xlog file in this case, but rename it into position anyway. In passing, merge the two callbacks for segment end and stop stream into a single callback, since their implementations were close to identical, and rename this callback to reflect that it stops streaming rather than continues it. Patch by Magnus Hagander, review by Fujii Masao
2012-05-25 11:36:22 +02:00
* Called in the background process every time data is received.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
* On Unix, we check to see if there is any data on our pipe
* (which would mean we have a stop position), and if it is, check if
* it is time to stop.
* On Windows, we are in a single process, so we can just check if it's
* time to stop.
*/
static bool
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
if (!has_xlogendptr)
{
#ifndef WIN32
fd_set fds;
struct timeval tv;
int r;
/*
* Don't have the end pointer yet - check our pipe to see if it has
* been sent yet.
*/
FD_ZERO(&fds);
FD_SET(bgpipe[0], &fds);
MemSet(&tv, 0, sizeof(tv));
r = select(bgpipe[0] + 1, &fds, NULL, NULL, &tv);
if (r == 1)
{
char xlogend[64];
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
uint32 hi,
lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
MemSet(xlogend, 0, sizeof(xlogend));
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
r = read(bgpipe[0], xlogend, sizeof(xlogend) - 1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (r < 0)
{
fprintf(stderr, _("%s: could not read from ready pipe: %s\n"),
progname, strerror(errno));
exit(1);
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (sscanf(xlogend, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: could not parse write-ahead log location \"%s\"\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, xlogend);
exit(1);
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
xlogendptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
has_xlogendptr = 1;
/*
* Fall through to check if we've reached the point further
* already.
*/
}
else
{
/*
* No data received on the pipe means we don't know the end
* position yet - so just say it's not time to stop yet.
*/
return false;
}
#else
/*
* On win32, has_xlogendptr is set by the main thread, so if it's not
* set here, we just go back and wait until it shows up.
*/
return false;
#endif
}
/*
* At this point we have an end pointer, so compare it to the current
* position to figure out if it's time to stop.
*/
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (segendpos >= xlogendptr)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
return true;
/*
* Have end pointer, but haven't reached it yet - so tell the caller to
* keep streaming.
*/
return false;
}
typedef struct
{
PGconn *bgconn;
XLogRecPtr startptr;
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
char xlog[MAXPGPATH]; /* directory or tarfile depending on mode */
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
char *sysidentifier;
int timeline;
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
bool temp_slot;
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
} logstreamer_param;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
static int
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
LogStreamerMain(logstreamer_param *param)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
StreamCtl stream;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
in_log_streamer = true;
Fix order of MemSet arguments Noted by Tomas Vondra
2016-03-13 13:11:06 +01:00
MemSet(&stream, 0, sizeof(stream));
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.startpos = param->startptr;
stream.timeline = param->timeline;
stream.sysidentifier = param->sysidentifier;
stream.stream_stop = reached_end_position;
Avoid slow shutdown of pg_basebackup. pg_basebackup's child process did not pay any attention to the pipe from its parent while waiting for input from the source server. If no server data was arriving, it would only wake up and check the pipe every standby_message_timeout or so. This creates a problem since the parent process might determine and send the desired stop position only after the server has reached end-of-WAL and stopped sending data. In the src/test/recovery regression tests, the timing is repeatably such that it takes nearly 10 seconds for the child process to realize that it should shut down. It's not clear how often that would happen in real-world cases, but it sure seems like a bug --- and if the user turns off standby_message_timeout or sets it very large, the delay could be a lot worse. To fix, expand the StreamCtl API to allow the pipe input FD to be passed down to the low-level wait routine, and watch both sockets when sleeping. (Note: AFAICS this issue doesn't affect the Windows port, since it doesn't rely on a pipe to transfer the stop position to the child thread.) Discussion: https://postgr.es/m/6456.1493263884@sss.pgh.pa.us
2017-04-28 00:27:02 +02:00
#ifndef WIN32
stream.stop_socket = bgpipe[0];
#else
stream.stop_socket = PGINVALID_SOCKET;
#endif
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.standby_message_timeout = standby_message_timeout;
stream.synchronous = false;
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
stream.do_sync = do_sync;
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
stream.mark_done = true;
stream.partial_suffix = NULL;
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
stream.replication_slot = replication_slot;
stream.temp_slot = param->temp_slot;
if (stream.temp_slot && !stream.replication_slot)
stream.replication_slot = psprintf("pg_basebackup_%d", (int) getpid());
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (format == 'p')
Add compression support to pg_receivexlog Author: Michael Paquier, review and small changes by me
2017-01-17 12:10:26 +01:00
stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0, do_sync);
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
else
stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel, do_sync);
Refactor receivelog.c parameters Much cruft had accumulated over time with a large number of parameters passed down between functions very deep. With this refactoring, instead introduce a StreamCtl structure that holds the parameters, and pass around a pointer to this structure instead. This makes it much easier to add or remove fields that are needed deeper down in the implementation without having to modify every function header in the file. Patch by me after much nagging from Andres Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:08:01 +01:00
if (!ReceiveXlogStream(param->bgconn, &stream))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Any errors will already have been reported in the function process,
* but we need to tell the parent that we didn't shutdown in a nice
* way.
*/
return 1;
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (!stream.walmethod->finish())
{
fprintf(stderr,
_("%s: could not finish writing WAL files: %s\n"),
progname, strerror(errno));
return 1;
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
PQfinish(param->bgconn);
Free walmethods before exiting Not strictly necessary since we quite after, but could become important in the future if we do restarts etc. Michael Paquier with nitpicking from me
2016-10-25 18:57:56 +02:00
if (format == 'p')
FreeWalDirectoryMethod();
else
FreeWalTarMethod();
pg_free(stream.walmethod);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
return 0;
}
/*
* Initiate background process for receiving xlog during the backup.
* The background stream will use its own database connection so we can
* stream the logfile in parallel with the backups.
*/
static void
StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
{
logstreamer_param *param;
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
uint32 hi,
lo;
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
char statusdir[MAXPGPATH];
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
param = pg_malloc0(sizeof(logstreamer_param));
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
param->timeline = timeline;
param->sysidentifier = sysidentifier;
/* Convert the starting position */
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
if (sscanf(startpos, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: could not parse write-ahead log location \"%s\"\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, startpos);
disconnect_and_exit(1);
}
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
param->startptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/* Round off to even segment position */
Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version.
2012-06-24 17:51:37 +02:00
param->startptr -= param->startptr % XLOG_SEG_SIZE;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#ifndef WIN32
/* Create our background pipe */
Remove now redundant pgpipe code.
2012-03-29 05:24:07 +02:00
if (pipe(bgpipe) < 0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not create pipe for background process: %s\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, strerror(errno));
disconnect_and_exit(1);
}
#endif
/* Get a second connection */
param->bgconn = GetConnection();
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
if (!param->bgconn)
/* Error message already written in GetConnection() */
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/* In post-10 cluster, pg_xlog has been renamed to pg_wal */
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
snprintf(param->xlog, sizeof(param->xlog), "%s/%s",
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/* Temporary replication slots are only supported in 10 and newer */
if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS)
param->temp_slot = false;
else
param->temp_slot = temp_replication_slot;
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
if (format == 'p')
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
{
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier
2016-10-23 15:16:31 +02:00
/*
* Create pg_wal/archive_status or pg_xlog/archive_status (and thus
* pg_wal or pg_xlog) depending on the target server so we can write to
* basedir/pg_wal or basedir/pg_xlog as the directory entry in the tar
* file may arrive later.
*/
snprintf(statusdir, sizeof(statusdir), "%s/%s/archive_status",
basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
if (pg_mkdir_p(statusdir, S_IRWXU) != 0 && errno != EEXIST)
{
fprintf(stderr,
_("%s: could not create directory \"%s\": %s\n"),
progname, statusdir, strerror(errno));
disconnect_and_exit(1);
}
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Start a child process and tell it to start streaming. On Unix, this is
* a fork(). On Windows, we create a thread.
*/
#ifndef WIN32
bgchild = fork();
if (bgchild == 0)
{
/* in child process */
exit(LogStreamerMain(param));
}
else if (bgchild < 0)
{
fprintf(stderr, _("%s: could not create background process: %s\n"),
progname, strerror(errno));
disconnect_and_exit(1);
}
/*
* Else we are in the parent process and all is well.
*/
#else /* WIN32 */
bgchild = _beginthreadex(NULL, 0, (void *) LogStreamerMain, param, 0, NULL);
if (bgchild == 0)
{
fprintf(stderr, _("%s: could not create background thread: %s\n"),
progname, strerror(errno));
disconnect_and_exit(1);
}
#endif
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Verify that the given directory exists and is empty. If it does not
* exist, it is created. If it exists but is not empty, an error will
Fix typo in comment Michael Paquier
2017-05-15 11:08:02 +02:00
* be given and the process ended.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
switch (pg_check_dir(dirname))
{
case 0:
/*
* Does not exist, so create
*/
if (pg_mkdir_p(dirname, S_IRWXU) == -1)
{
fprintf(stderr,
_("%s: could not create directory \"%s\": %s\n"),
progname, dirname, strerror(errno));
disconnect_and_exit(1);
}
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (created)
*created = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return;
case 1:
/*
* Exists, empty
*/
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
if (found)
*found = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return;
case 2:
Warn about initdb using mount-points Add code to detect and warn about trying to initdb or create pg_xlog on mount points.
2013-02-17 00:52:50 +01:00
case 3:
case 4:
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Exists, not empty
*/
fprintf(stderr,
_("%s: directory \"%s\" exists but is not empty\n"),
progname, dirname);
disconnect_and_exit(1);
case -1:
/*
* Access problem
*/
fprintf(stderr, _("%s: could not access directory \"%s\": %s\n"),
progname, dirname, strerror(errno));
disconnect_and_exit(1);
}
}
/*
* Print a progress report based on the global variables. If verbose output
* is enabled, also print the current file name.
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
*
* Progress report is written at maximum once per second, unless the
* force parameter is set to true.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(int tablespacenum, const char *filename, bool force)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
int percent;
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
char totaldone_str[32];
char totalsize_str[32];
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
pg_time_t now;
if (!showprogress)
return;
now = time(NULL);
if (now == last_progress_report && !force)
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
return; /* Max once per second */
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
last_progress_report = now;
percent = totalsize ? (int) ((totaldone / 1024) * 100 / totalsize) : 0;
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
Adjust total size in pg_basebackup progress report when reality changes When streaming including WAL, the size estimate will always be incorrect, since we don't know how much WAL is included. To make sure the output doesn't look completely unreasonable, this patch increases the total size whenever we go past the estimate, to make sure we never go above 100%.
2011-08-16 16:56:47 +02:00
/*
Fix pg_bsd_indent bug where newlines were not being trimmed from typedef lines. Update pg_bsd_indent required version to 1.1 (and update ftp site). Problem reported by Magnus.
2011-10-26 23:23:33 +02:00
* Avoid overflowing past 100% or the full size. This may make the total
* size number change as we approach the end of the backup (the estimate
* will always be wrong if WAL is included), but that's better than having
* the done column be bigger than the total.
Adjust total size in pg_basebackup progress report when reality changes When streaming including WAL, the size estimate will always be incorrect, since we don't know how much WAL is included. To make sure the output doesn't look completely unreasonable, this patch increases the total size whenever we go past the estimate, to make sure we never go above 100%.
2011-08-16 16:56:47 +02:00
*/
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
if (percent > 100)
percent = 100;
Adjust total size in pg_basebackup progress report when reality changes When streaming including WAL, the size estimate will always be incorrect, since we don't know how much WAL is included. To make sure the output doesn't look completely unreasonable, this patch increases the total size whenever we go past the estimate, to make sure we never go above 100%.
2011-08-16 16:56:47 +02:00
if (totaldone / 1024 > totalsize)
totalsize = totaldone / 1024;
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
/*
Fix pg_bsd_indent bug where newlines were not being trimmed from typedef lines. Update pg_bsd_indent required version to 1.1 (and update ftp site). Problem reported by Magnus.
2011-10-26 23:23:33 +02:00
* Separate step to keep platform-dependent format code out of
* translatable strings. And we only test for INT64_FORMAT availability
* in snprintf, not fprintf.
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
*/
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
snprintf(totaldone_str, sizeof(totaldone_str), INT64_FORMAT,
totaldone / 1024);
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
snprintf(totalsize_str, sizeof(totalsize_str), INT64_FORMAT, totalsize);
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
#define VERBOSE_FILENAME_LENGTH 35
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (verbose)
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
{
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (!filename)
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
/*
* No filename given, so clear the status line (used for last
* call)
*/
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (100%%), %d/%d tablespace %*s",
"%*s/%s kB (100%%), %d/%d tablespaces %*s",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str,
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
tablespacenum, tablespacecount,
VERBOSE_FILENAME_LENGTH + 5, "");
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
else
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
{
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
bool truncate = (strlen(filename) > VERBOSE_FILENAME_LENGTH);
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (%d%%), %d/%d tablespace (%s%-*.*s)",
"%*s/%s kB (%d%%), %d/%d tablespaces (%s%-*.*s)",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str, percent,
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
tablespacenum, tablespacecount,
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
/* Prefix with "..." if we do leading truncation */
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
truncate ? "..." : "",
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
truncate ? VERBOSE_FILENAME_LENGTH - 3 : VERBOSE_FILENAME_LENGTH,
truncate ? VERBOSE_FILENAME_LENGTH - 3 : VERBOSE_FILENAME_LENGTH,
/* Truncate filename at beginning if it's too long */
Truncate filenames in the leadning end in pg_basebackup verbose output When truncating at the end, like before, the output would often end up just showing the path instead of the filename. Also increase the length of the filename by 5, which still keeps us at less than 80 characters in most outputs.
2013-01-17 14:38:49 +01:00
truncate ? filename + strlen(filename) - VERBOSE_FILENAME_LENGTH + 3 : filename);
}
Properly clear the verbose output of pg_basebackup Don't leave partial filename information on the last status or dangling characters if the previous filename was longer than the current one.
2011-03-19 16:38:50 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
fprintf(stderr,
Make size-output fixed length in pg_basebackup verbose mode This way the line doesn't shift right as the amount of data processed increases.
2013-01-17 14:43:33 +01:00
ngettext("%*s/%s kB (%d%%), %d/%d tablespace",
"%*s/%s kB (%d%%), %d/%d tablespaces",
Make pg_basebackup progress report translatable Also fix a potential portability bug, because INT64_FORMAT is only guaranteed to be available with snprintf, not fprintf.
2011-08-16 10:24:08 +02:00
tablespacecount),
Silence compiler warnings
2013-01-17 16:10:16 +01:00
(int) strlen(totalsize_str),
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
totaldone_str, totalsize_str, percent,
tablespacenum, tablespacecount);
Move \r out of translatable strings The translation tools are very unhappy about seeing \r in translatable strings, so move it to a separate fprintf call.
2011-08-17 09:52:35 +02:00
fprintf(stderr, "\r");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
static int32
parse_max_rate(char *src)
{
double result;
char *after_num;
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
char *suffix = NULL;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
errno = 0;
result = strtod(src, &after_num);
if (src == after_num)
{
fprintf(stderr,
_("%s: transfer rate \"%s\" is not a valid value\n"),
progname, src);
exit(1);
}
if (errno != 0)
{
fprintf(stderr,
_("%s: invalid transfer rate \"%s\": %s\n"),
progname, src, strerror(errno));
exit(1);
}
if (result <= 0)
{
/*
* Reject obviously wrong values here.
*/
fprintf(stderr, _("%s: transfer rate must be greater than zero\n"),
progname);
exit(1);
}
/*
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
* Evaluate suffix, after skipping over possible whitespace. Lack of
* suffix means kilobytes.
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
*/
while (*after_num != '\0' && isspace((unsigned char) *after_num))
after_num++;
if (*after_num != '\0')
{
suffix = after_num;
if (*after_num == 'k')
{
/* kilobyte is the expected unit. */
after_num++;
}
else if (*after_num == 'M')
{
after_num++;
result *= 1024.0;
}
}
/* The rest can only consist of white space. */
while (*after_num != '\0' && isspace((unsigned char) *after_num))
after_num++;
if (*after_num != '\0')
{
fprintf(stderr,
Misc message style and doc fixes. Euler Taveira
2014-05-15 13:49:11 +02:00
_("%s: invalid --max-rate unit: \"%s\"\n"),
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
progname, suffix);
exit(1);
}
/* Valid integer? */
if ((uint64) result != (uint64) ((uint32) result))
{
fprintf(stderr,
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
_("%s: transfer rate \"%s\" exceeds integer range\n"),
progname, src);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
exit(1);
}
/*
* The range is checked on the server side too, but avoid the server
* connection if a nonsensical value was passed.
*/
if (result < MAX_RATE_LOWER || result > MAX_RATE_UPPER)
{
fprintf(stderr,
_("%s: transfer rate \"%s\" is out of range\n"),
progname, src);
exit(1);
}
return (int32) result;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
* Write a piece of tar data
*/
static void
writeTarData(
#ifdef HAVE_LIBZ
gzFile ztarfile,
#endif
FILE *tarfile, char *buf, int r, char *current_file)
{
#ifdef HAVE_LIBZ
if (ztarfile != NULL)
{
if (gzwrite(ztarfile, buf, r) != r)
{
fprintf(stderr,
_("%s: could not write to compressed file \"%s\": %s\n"),
progname, current_file, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
if (fwrite(buf, r, 1, tarfile) != 1)
{
fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
progname, current_file, strerror(errno));
disconnect_and_exit(1);
}
}
}
#ifdef HAVE_LIBZ
#define WRITE_TAR_DATA(buf, sz) writeTarData(ztarfile, tarfile, buf, sz, filename)
#else
#define WRITE_TAR_DATA(buf, sz) writeTarData(tarfile, buf, sz, filename)
#endif
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Receive a tar format file from the connection to the server, and write
* the data from this file directly into a tar file. If compression is
* enabled, the data will be compressed while written to the file.
*
* The file will be named base.tar[.gz] if it's for the main data directory
* or <tablespaceoid>.tar[.gz] if it's for another tablespace.
*
* No attempt to inspect or validate the contents of the file is done.
*/
static void
ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
{
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
char filename[MAXPGPATH];
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
char *copybuf = NULL;
FILE *tarfile = NULL;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
char tarhdr[512];
bool basetablespace = PQgetisnull(res, rownum, 0);
bool in_tarhdr = true;
bool skip_file = false;
size_t tarhdrsz = 0;
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
pgoff_t filesz = 0;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifdef HAVE_LIBZ
Fix incorrect uses of gzFile gzFile is already a pointer, so code like gzFile *handle = gzopen(...) is wrong. This used to pass silently because gzFile used to be defined as void*, and you can assign a void* to a void**. But somewhere between zlib versions 1.2.3.4 and 1.2.6, the definition of gzFile was changed to struct gzFile_s *, and with that new definition this usage causes compiler warnings. So remove all those extra pointer decorations. There is a related issue in pg_backup_archiver.h, where FILE *FH; /* General purpose file handle */ is used throughout pg_dump as sometimes a real FILE* and sometimes a gzFile handle, which also causes warnings now. This is not yet fixed here, because it might need more code restructuring.
2012-03-02 21:30:01 +01:00
gzFile ztarfile = NULL;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (basetablespace)
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
{
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Base tablespaces
*/
if (strcmp(basedir, "-") == 0)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
ztarfile = gzdopen(dup(fileno(stdout)), "wb");
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
if (gzsetparams(ztarfile, compresslevel,
Z_DEFAULT_STRATEGY) != Z_OK)
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not set compression level %d: %s\n"),
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
progname, compresslevel, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
tarfile = stdout;
Avoid printing uninitialized filename variable in verbose mode When using verbose mode for pg_basebackup, in tar format sent to stdout, we'd print an unitialized buffer as the filename. Reported by Pontus Lundkvist
2014-02-09 11:54:33 +01:00
strcpy(filename, "-");
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
{
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
snprintf(filename, sizeof(filename), "%s/base.tar.gz", basedir);
ztarfile = gzopen(filename, "wb");
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
if (gzsetparams(ztarfile, compresslevel,
Z_DEFAULT_STRATEGY) != Z_OK)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not set compression level %d: %s\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, compresslevel, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
snprintf(filename, sizeof(filename), "%s/base.tar", basedir);
tarfile = fopen(filename, "wb");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
{
/*
* Specific tablespace
*/
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
snprintf(filename, sizeof(filename), "%s/%s.tar.gz", basedir,
PQgetvalue(res, rownum, 0));
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
ztarfile = gzopen(filename, "wb");
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
if (gzsetparams(ztarfile, compresslevel,
Z_DEFAULT_STRATEGY) != Z_OK)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not set compression level %d: %s\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, compresslevel, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
snprintf(filename, sizeof(filename), "%s/%s.tar", basedir,
PQgetvalue(res, rownum, 0));
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
tarfile = fopen(filename, "wb");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
#ifdef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
if (!ztarfile)
{
/* Compression is in use */
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not create compressed file \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, get_gz_error(ztarfile));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
}
else
#endif
{
/* Either no zlib support, or zlib support but compresslevel = 0 */
if (!tarfile)
{
fprintf(stderr, _("%s: could not create file \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
}
/*
* Get the COPY data stream
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COPY_OUT)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not get COPY data stream: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
while (1)
{
int r;
if (copybuf != NULL)
{
PQfreemem(copybuf);
copybuf = NULL;
}
r = PQgetCopyData(conn, &copybuf, 0);
if (r == -1)
{
/*
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
* End of chunk. If requested, and this is the base tablespace,
* write recovery.conf into the tarfile. When done, close the file
* (but not stdout).
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*
* Also, write two completely empty blocks at the end of the tar
* file, as required by some tar programs.
*/
char zerobuf[1024];
MemSet(zerobuf, 0, sizeof(zerobuf));
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (basetablespace && writerecoveryconf)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
char header[512];
int padding;
tarCreateHeader(header, "recovery.conf", NULL,
recoveryconfcontents->len,
0600, 04000, 02000,
time(NULL));
padding = ((recoveryconfcontents->len + 511) & ~511) - recoveryconfcontents->len;
WRITE_TAR_DATA(header, sizeof(header));
WRITE_TAR_DATA(recoveryconfcontents->data, recoveryconfcontents->len);
if (padding)
WRITE_TAR_DATA(zerobuf, padding);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/* 2 * 512 bytes empty data at end of file */
WRITE_TAR_DATA(zerobuf, sizeof(zerobuf));
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
#ifdef HAVE_LIBZ
pg_basebackup: Error handling fixes. Thomas Ogrisegg and Fujii Masao
2012-03-28 18:19:22 +02:00
if (ztarfile != NULL)
{
if (gzclose(ztarfile) != 0)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not close compressed file \"%s\": %s\n"),
pg_basebackup: Error handling fixes. Thomas Ogrisegg and Fujii Masao
2012-03-28 18:19:22 +02:00
progname, filename, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
Allow pg_basebackup compressed tar output to stdout
2011-05-29 00:17:37 +02:00
}
else
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
pg_basebackup: Error handling fixes. Thomas Ogrisegg and Fujii Masao
2012-03-28 18:19:22 +02:00
{
if (strcmp(basedir, "-") != 0)
{
if (fclose(tarfile) != 0)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not close file \"%s\": %s\n"),
pg_basebackup: Error handling fixes. Thomas Ogrisegg and Fujii Masao
2012-03-28 18:19:22 +02:00
progname, filename, strerror(errno));
disconnect_and_exit(1);
}
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
break;
}
else if (r == -2)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not read COPY data: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (!writerecoveryconf || !basetablespace)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
* When not writing recovery.conf, or when not working on the base
* tablespace, we never have to look for an existing recovery.conf
* file in the stream.
*/
WRITE_TAR_DATA(copybuf, r);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
else
{
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
* Look for a recovery.conf in the existing tar stream. If it's
* there, we must skip it so we can later overwrite it with our
* own version of the file.
*
* To do this, we have to process the individual files inside the
* TAR stream. The stream consists of a header and zero or more
* chunks, all 512 bytes long. The stream from the server is
* broken up into smaller pieces, so we have to track the size of
* the files to find the next header structure.
*/
int rr = r;
int pos = 0;
while (rr > 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (in_tarhdr)
{
/*
* We're currently reading a header structure inside the
* TAR stream, i.e. the file metadata.
*/
if (tarhdrsz < 512)
{
/*
* Copy the header structure into tarhdr in case the
* header is not aligned to 512 bytes or it's not
* returned in whole by the last PQgetCopyData call.
*/
int hdrleft;
int bytes2copy;
hdrleft = 512 - tarhdrsz;
bytes2copy = (rr > hdrleft ? hdrleft : rr);
memcpy(&tarhdr[tarhdrsz], copybuf + pos, bytes2copy);
rr -= bytes2copy;
pos += bytes2copy;
tarhdrsz += bytes2copy;
}
else
{
/*
* We have the complete header structure in tarhdr,
* look at the file metadata: - the subsequent file
* contents have to be skipped if the filename is
* recovery.conf - find out the size of the file
* padded to the next multiple of 512
*/
int padding;
skip_file = (strcmp(&tarhdr[0], "recovery.conf") == 0);
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
filesz = read_tar_number(&tarhdr[124], 12);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
padding = ((filesz + 511) & ~511) - filesz;
filesz += padding;
/* Next part is the file, not the header */
in_tarhdr = false;
/*
* If we're not skipping the file, write the tar
* header unmodified.
*/
if (!skip_file)
WRITE_TAR_DATA(tarhdr, 512);
}
}
else
{
/*
* We're processing a file's contents.
*/
if (filesz > 0)
{
/*
* We still have data to read (and possibly write).
*/
int bytes2write;
bytes2write = (filesz > rr ? rr : filesz);
if (!skip_file)
WRITE_TAR_DATA(copybuf + pos, bytes2write);
rr -= bytes2write;
pos += bytes2write;
filesz -= bytes2write;
}
else
{
/*
* No more data in the current file, the next piece of
* data (if any) will be a new file header structure.
*/
in_tarhdr = true;
skip_file = false;
tarhdrsz = 0;
filesz = 0;
}
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
}
totaldone += r;
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(rownum, filename, false);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
} /* while (1) */
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(rownum, filename, true);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (copybuf != NULL)
PQfreemem(copybuf);
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
/* sync the resulting tar file, errors are not considered fatal */
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
if (do_sync && strcmp(basedir, "-") != 0)
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
(void) fsync_fname(filename, false, progname);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
/*
* Retrieve tablespace path, either relocated or original depending on whether
* -T was passed or not.
*/
static const char *
get_tablespace_mapping(const char *dir)
{
TablespaceListCell *cell;
for (cell = tablespace_dirs.head; cell; cell = cell->next)
if (strcmp(dir, cell->old_dir) == 0)
return cell->new_dir;
return dir;
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Receive a tar format stream from the connection to the server, and unpack
* the contents of it into a directory. Only files, directories and
* symlinks are supported, no other kinds of special files.
*
* If the data is for the main data directory, it will be restored in the
* specified directory. If it's for another tablespace, it will be restored
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
* in the original or mapped directory.
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
static void
ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
{
char current_path[MAXPGPATH];
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
char filename[MAXPGPATH];
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
const char *mapped_tblspc_path;
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
pgoff_t current_len_left = 0;
Suppress uninitialized-variable warning.
2011-01-23 19:06:38 +01:00
int current_padding = 0;
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
bool basetablespace;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
char *copybuf = NULL;
FILE *file = NULL;
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
basetablespace = PQgetisnull(res, rownum, 0);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
if (basetablespace)
Prevent potential overruns of fixed-size buffers. Coverity identified a number of places in which it couldn't prove that a string being copied into a fixed-size buffer would fit. We believe that most, perhaps all of these are in fact safe, or are copying data that is coming from a trusted source so that any overrun is not really a security issue. Nonetheless it seems prudent to forestall any risk by using strlcpy() and similar functions. Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports. In addition, fix a potential null-pointer-dereference crash in contrib/chkpass. The crypt(3) function is defined to return NULL on failure, but chkpass.c didn't check for that before using the result. The main practical case in which this could be an issue is if libc is configured to refuse to execute unapproved hashing algorithms (e.g., "FIPS mode"). This ideally should've been a separate commit, but since it touches code adjacent to one of the buffer overrun changes, I included it in this commit to avoid last-minute merge issues. This issue was reported by Honza Horak. Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 17:20:21 +01:00
strlcpy(current_path, basedir, sizeof(current_path));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
else
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
strlcpy(current_path,
get_tablespace_mapping(PQgetvalue(res, rownum, 1)),
sizeof(current_path));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Get the COPY data
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COPY_OUT)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not get COPY data stream: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
while (1)
{
int r;
if (copybuf != NULL)
{
PQfreemem(copybuf);
copybuf = NULL;
}
r = PQgetCopyData(conn, &copybuf, 0);
if (r == -1)
{
/*
* End of chunk
*/
if (file)
fclose(file);
break;
}
else if (r == -2)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not read COPY data: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
if (file == NULL)
{
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
int filemode;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* No current file, so this must be the header for a new file
*/
if (r != 512)
{
Replace printf format %i by %d They are identical, but the overwhelming majority of the code uses %d, so standardize on that.
2011-07-26 21:54:29 +02:00
fprintf(stderr, _("%s: invalid tar block header size: %d\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, r);
disconnect_and_exit(1);
}
totaldone += 512;
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
current_len_left = read_tar_number(&copybuf[124], 12);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/* Set permissions on the file */
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
filemode = read_tar_number(&copybuf[100], 8);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* All files are padded up to 512 bytes
*/
current_padding =
((current_len_left + 511) & ~511) - current_len_left;
/*
* First part of header is zero terminated filename
*/
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
snprintf(filename, sizeof(filename), "%s/%s", current_path,
copybuf);
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (filename[strlen(filename) - 1] == '/')
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
/*
* Ends in a slash means directory or symlink to directory
*/
if (copybuf[156] == '5')
{
/*
* Directory
*/
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
filename[strlen(filename) - 1] = '\0'; /* Remove trailing slash */
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (mkdir(filename, S_IRWXU) != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
* When streaming WAL, pg_wal (or pg_xlog for pre-9.6
* clusters) will have been created by the wal receiver
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
* process. Also, when the WAL directory location
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
* was specified, pg_wal (or pg_xlog) has already been
* created as a symbolic link before starting the actual
* backup. So just ignore creation failures on related
* directories.
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
*/
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
if (!((pg_str_endswith(filename, "/pg_wal") ||
pg_str_endswith(filename, "/pg_xlog")||
pg_str_endswith(filename, "/archive_status")) &&
Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de
2015-01-03 20:51:52 +01:00
errno == EEXIST))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
fprintf(stderr,
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
_("%s: could not create directory \"%s\": %s\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, filename, strerror(errno));
disconnect_and_exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
#ifndef WIN32
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (chmod(filename, (mode_t) filemode))
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not set permissions on directory \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
}
else if (copybuf[156] == '2')
{
/*
* Symbolic link
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
*
Fix comment in pg_basebackup. The option is called --tablespace-mapping, not --tablespace. Amit Kapila
2014-08-22 12:45:38 +02:00
* It's most likely a link in pg_tblspc directory, to the
* location of a tablespace. Apply any tablespace mapping
pgindent run for 9.5
2015-05-24 03:35:49 +02:00
* given on the command line (--tablespace-mapping). (We
* blindly apply the mapping without checking that the
* link really is inside pg_tblspc. We don't expect there
* to be other symlinks in a data directory, but if there
* are, you can call it an undocumented feature that you
* can map them too.)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
filename[strlen(filename) - 1] = '\0'; /* Remove trailing slash */
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
mapped_tblspc_path = get_tablespace_mapping(&copybuf[157]);
if (symlink(mapped_tblspc_path, filename) != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
fprintf(stderr,
Message improvements
2011-05-24 20:25:28 +02:00
_("%s: could not create symbolic link from \"%s\" to \"%s\": %s\n"),
Change the way pg_basebackup's tablespace mapping is implemented. Previously, we would first create the symlinks the way they are in the original system, and at the end replace them with the mapped symlinks. That never really made much sense, so now we create the symlink pointing to the correct location to begin with, so that there's no need to fix them at the end. The old coding didn't work correctly on Windows, because Windows junction points look more like directories than files, and ought to be removed with rmdir rather than unlink. Also, it incorrectly used "%d" rather than "%u" to print an Oid, but that's gone now. Report and patch by Amit Kapila, with minor changes by me. Reviewed by MauMau. Backpatch to 9.4, where the --tablespace feature was added.
2014-08-22 09:16:26 +02:00
progname, filename, mapped_tblspc_path,
strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
}
else
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: unrecognized link indicator \"%c\"\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, copybuf[156]);
disconnect_and_exit(1);
}
continue; /* directory or link handled */
}
/*
* regular file
*/
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
file = fopen(filename, "wb");
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (!file)
{
fprintf(stderr, _("%s: could not create file \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
#ifndef WIN32
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
if (chmod(filename, (mode_t) filemode))
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fprintf(stderr, _("%s: could not set permissions on file \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#endif
if (current_len_left == 0)
{
/*
* Done with this file, next one will be a new tar header
*/
fclose(file);
file = NULL;
continue;
}
} /* new file */
else
{
/*
* Continuing blocks in existing file
*/
if (current_len_left == 0 && r == current_padding)
{
/*
* Received the padding block for this file, ignore it and
* close the file, then move on to the next tar header.
*/
fclose(file);
file = NULL;
totaldone += r;
continue;
}
if (fwrite(copybuf, r, 1, file) != 1)
{
fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
Use less cryptic variable names
2011-08-16 10:19:50 +02:00
progname, filename, strerror(errno));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
totaldone += r;
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(rownum, filename, false);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
current_len_left -= r;
if (current_len_left == 0 && current_padding == 0)
{
/*
* Received the last block, and there is no padding to be
* expected. Close the file and move on to the next tar
* header.
*/
fclose(file);
file = NULL;
continue;
}
} /* continuing data in existing file */
} /* loop over all data blocks */
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(rownum, filename, true);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (file != NULL)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: COPY stream ended before last file was finished\n"),
progname);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
if (copybuf != NULL)
PQfreemem(copybuf);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
Don't attempt to write recovery.conf when -R is not specified Fixes segmentation fault during regular use. Fujii Masao
2013-01-09 16:57:32 +01:00
if (basetablespace && writerecoveryconf)
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
WriteRecoveryConf();
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
/*
* No data is synced here, everything is done for all tablespaces at the
* end.
*/
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
/*
* Escape a string so that it can be used as a value in a key-value pair
* a configuration file.
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
*/
static char *
escape_quotes(const char *src)
{
char *result = escape_single_quotes_ascii(src);
if (!result)
{
fprintf(stderr, _("%s: out of memory\n"), progname);
exit(1);
}
return result;
}
/*
* Create a recovery.conf file in memory using a PQExpBuffer
*/
static void
GenerateRecoveryConf(PGconn *conn)
{
PQconninfoOption *connOptions;
PQconninfoOption *option;
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
PQExpBufferData conninfo_buf;
char *escaped;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
recoveryconfcontents = createPQExpBuffer();
if (!recoveryconfcontents)
{
pg_basebackup: Add missing newlines to several error messages
2013-03-10 21:53:04 +01:00
fprintf(stderr, _("%s: out of memory\n"), progname);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
disconnect_and_exit(1);
}
connOptions = PQconninfo(conn);
if (connOptions == NULL)
{
pg_basebackup: Add missing newlines to several error messages
2013-03-10 21:53:04 +01:00
fprintf(stderr, _("%s: out of memory\n"), progname);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
disconnect_and_exit(1);
}
appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
initPQExpBuffer(&conninfo_buf);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
for (option = connOptions; option && option->keyword; option++)
{
/*
* Do not emit this setting if: - the setting is "replication",
* "dbname" or "fallback_application_name", since these would be
* overridden by the libpqwalreceiver module anyway. - not set or
* empty.
*/
if (strcmp(option->keyword, "replication") == 0 ||
strcmp(option->keyword, "dbname") == 0 ||
strcmp(option->keyword, "fallback_application_name") == 0 ||
(option->val == NULL) ||
(option->val != NULL && option->val[0] == '\0'))
continue;
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
/* Separate key-value pairs with spaces */
if (conninfo_buf.len != 0)
Use appendStringInfoString/Char et al where appropriate. Patch by David Rowley. Backpatch to 9.5, as some of the calls were new in 9.5, and keeping the code in sync with master makes future backpatching easier.
2015-07-02 11:32:48 +02:00
appendPQExpBufferChar(&conninfo_buf, ' ');
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
* Write "keyword=value" pieces, the value string is escaped and/or
* quoted if necessary.
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
*/
Retire escapeConnectionParameter(). It is redundant with appendConnStrVal(), which became an extern function in commit 41f18f021a0882eccbeca62e2ed4b66c6b96e9c9. This changes the handling of out-of-memory and of certain inputs for which quoting is optional, but pg_basebackup has no need for unusual treatment thereof.
2016-08-22 04:05:57 +02:00
appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
appendConnStrVal(&conninfo_buf, option->val);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
}
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
/*
* Escape the connection string, so that it can be put in the config file.
* Note that this is different from the escaping of individual connection
* options above!
*/
escaped = escape_quotes(conninfo_buf.data);
appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
free(escaped);
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
if (replication_slot)
{
escaped = escape_quotes(replication_slot);
appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
free(escaped);
}
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
if (PQExpBufferBroken(recoveryconfcontents) ||
PQExpBufferDataBroken(conninfo_buf))
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{
pg_basebackup: Add missing newlines to several error messages
2013-03-10 21:53:04 +01:00
fprintf(stderr, _("%s: out of memory\n"), progname);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
disconnect_and_exit(1);
}
Fix escaping in generated recovery.conf file. In the primary_conninfo line that "pg_basebackup -R" generates, single quotes in parameter values need to be escaped into \\'; the libpq parser requires the quotes to be escaped into \', and recovery.conf parser requires the \ to be escaped into \\. Also, don't quote parameter values unnecessarily, to make the connection string prettier. Most options in a libpq connection string don't need quoting. Reported by Hari Babu, closer analysis by Zoltan Boszormenyi, although I didn't use his patch.
2013-05-20 18:34:27 +02:00
termPQExpBuffer(&conninfo_buf);
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
PQconninfoFree(connOptions);
}
/*
* Write a recovery.conf file into the directory specified in basedir,
* with the contents already collected in memory.
*/
static void
WriteRecoveryConf(void)
{
char filename[MAXPGPATH];
FILE *cf;
sprintf(filename, "%s/recovery.conf", basedir);
cf = fopen(filename, "w");
if (cf == NULL)
{
pg_basebackup: Add missing newlines to several error messages
2013-03-10 21:53:04 +01:00
fprintf(stderr, _("%s: could not create file \"%s\": %s\n"), progname, filename, strerror(errno));
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
disconnect_and_exit(1);
}
if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
{
fprintf(stderr,
_("%s: could not write to file \"%s\": %s\n"),
progname, filename, strerror(errno));
disconnect_and_exit(1);
}
fclose(cf);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
static void
fix up a couple non-prototypes of the form foo() to be foo(void) -- found using -Wstrict-prototypes
2011-03-27 22:31:41 +02:00
BaseBackup(void)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
PGresult *res;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
char *sysidentifier;
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
TimeLineID latesttli;
TimeLineID starttli;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
char *basebkp;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
char escaped_label[MAXPGPATH];
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
char *maxrate_clause = NULL;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
int i;
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
char xlogstart[64];
char xlogend[64];
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
int minServerMajor,
maxServerMajor;
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
int serverVersion,
serverMajor;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
Assert(conn != NULL);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
/*
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
* Check server version. BASE_BACKUP command was introduced in 9.1, so we
* can't work with servers older than 9.1.
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
*/
minServerMajor = 901;
maxServerMajor = PG_VERSION_NUM / 100;
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
serverVersion = PQserverVersion(conn);
serverMajor = serverVersion / 100;
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
if (serverMajor < minServerMajor || serverMajor > maxServerMajor)
{
const char *serverver = PQparameterStatus(conn, "server_version");
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
fprintf(stderr, _("%s: incompatible server version %s\n"),
progname, serverver ? serverver : "'unknown'");
disconnect_and_exit(1);
}
/*
* If WAL streaming was requested, also check that the server is new
* enough for that.
*/
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (includewal == STREAM_WAL && !CheckServerVersionForStreaming(conn))
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
{
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
/*
* Error message already written in CheckServerVersionForStreaming(),
* but add a hint about using -X none.
*/
fprintf(stderr, _("HINT: use -X none or -X fetch to disable log streaming\n"));
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
disconnect_and_exit(1);
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/*
* Build contents of recovery.conf if requested
*/
if (writerecoveryconf)
GenerateRecoveryConf(conn);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
*/
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
if (!RunIdentifySystem(conn, &sysidentifier, &latesttli, NULL, NULL))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
disconnect_and_exit(1);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Start the actual backup
*/
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQescapeStringConn(conn, escaped_label, label, sizeof(escaped_label), &i);
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
if (maxrate > 0)
maxrate_clause = psprintf("MAX_RATE %u", maxrate);
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
if (verbose)
fprintf(stderr,
_("%s: initiating base backup, waiting for checkpoint to complete\n"),
progname);
if (showprogress && !verbose)
Write "waiting for checkpoint" on regular progress row When reporting progress, make the "waiting for checkpoint" test be overwritten by the file-based progress once it's completed. This is more consistent with how we report the rest of the progress. Suggested by Jeff Janes
2017-04-01 17:04:14 +02:00
fprintf(stderr, "waiting for checkpoint\r");
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
basebkp =
Map basebackup tablespaces using a tablespace_map file Windows can't reliably restore symbolic links from a tar format, so instead during backup start we create a tablespace_map file, which is used by the restoring postgres to create the correct links in pg_tblspc. The backup protocol also now has an option to request this file to be included in the backup stream, and this is used by pg_basebackup when operating in tar mode. This is done on all platforms, not just Windows. This means that pg_basebackup will not not work in tar mode against 9.4 and older servers, as this protocol option isn't implemented there. Amit Kapila, reviewed by Dilip Kumar, with a little editing from me.
2015-05-12 15:29:10 +02:00
psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s",
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
escaped_label,
showprogress ? "PROGRESS" : "",
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal == FETCH_WAL ? "WAL" : "",
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
fastcheckpoint ? "FAST" : "",
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal == NO_WAL ? "" : "NOWAIT",
Map basebackup tablespaces using a tablespace_map file Windows can't reliably restore symbolic links from a tar format, so instead during backup start we create a tablespace_map file, which is used by the restoring postgres to create the correct links in pg_tblspc. The backup protocol also now has an option to request this file to be included in the backup stream, and this is used by pg_basebackup when operating in tar mode. This is done on all platforms, not just Windows. This means that pg_basebackup will not not work in tar mode against 9.4 and older servers, as this protocol option isn't implemented there. Amit Kapila, reviewed by Dilip Kumar, with a little editing from me.
2015-05-12 15:29:10 +02:00
maxrate_clause ? maxrate_clause : "",
pgindent run for 9.5
2015-05-24 03:35:49 +02:00
format == 't' ? "TABLESPACE_MAP" : "");
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
if (PQsendQuery(conn, basebkp) == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr, _("%s: could not send replication command \"%s\": %s"),
progname, "BASE_BACKUP", PQerrorMessage(conn));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
/*
Standardize "WAL location" terminology Other previously used terms were "WAL position" or "log position".
2017-05-12 19:51:27 +02:00
* Get the starting WAL location
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not initiate base backup: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
if (PQntuples(res) != 1)
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
{
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
fprintf(stderr,
_("%s: server returned unexpected response to BASE_BACKUP command; got %d rows and %d fields, expected %d rows and %d fields\n"),
progname, PQntuples(res), PQnfields(res), 1, 2);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
disconnect_and_exit(1);
}
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
Prevent potential overruns of fixed-size buffers. Coverity identified a number of places in which it couldn't prove that a string being copied into a fixed-size buffer would fit. We believe that most, perhaps all of these are in fact safe, or are copying data that is coming from a trusted source so that any overrun is not really a security issue. Nonetheless it seems prudent to forestall any risk by using strlcpy() and similar functions. Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports. In addition, fix a potential null-pointer-dereference crash in contrib/chkpass. The crypt(3) function is defined to return NULL on failure, but chkpass.c didn't check for that before using the result. The main practical case in which this could be an issue is if libc is configured to refuse to execute unapproved hashing algorithms (e.g., "FIPS mode"). This ideally should've been a separate commit, but since it touches code adjacent to one of the buffer overrun changes, I included it in this commit to avoid last-minute merge issues. This issue was reported by Honza Horak. Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 17:20:21 +01:00
strlcpy(xlogstart, PQgetvalue(res, 0, 0), sizeof(xlogstart));
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
Clarify the role of checkpoint at the begininng of base backups Output a message about checkpoint starting in verbose mode of pg_basebackup, and make the documentation state more clearly that this happens. Author: Michael Banck
2017-02-26 21:27:51 +01:00
if (verbose)
fprintf(stderr, _("%s: checkpoint completed\n"), progname);
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
/*
* 9.3 and later sends the TLI of the starting point. With older servers,
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
* assume it's the same as the latest timeline reported by
* IDENTIFY_SYSTEM.
Make pg_basebackup work with pre-9.3 servers, and add server version check. A new 'starttli' field was added to the response of BASE_BACKUP command. Make pg_basebackup tolerate the case that it's missing, so that it still works with older servers. Add an explicit check for the server version, so that you get a nicer error message if you try to use it with a pre-9.1 server. The streaming protocol message format changed in 9.3, so -X stream still won't work with pre-9.3 servers. I added a version check to ReceiveXLogStream() earlier, but write that slightly differently, so that in 9.4, it will still work with a 9.3 server. (In 9.4, the error message needs to be adjusted to "9.3 or above", though). Also, if the version check fails, don't retry.
2013-03-22 12:02:59 +01:00
*/
if (PQnfields(res) >= 2)
starttli = atoi(PQgetvalue(res, 0, 1));
else
starttli = latesttli;
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
PQclear(res);
MemSet(xlogend, 0, sizeof(xlogend));
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (verbose && includewal != NO_WAL)
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: write-ahead log start point: %s on timeline %u\n"),
Add missing progname prefix to some messages Author: Michael Banck
2017-02-26 21:29:34 +01:00
progname, xlogstart, starttli);
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Get the header
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: could not get backup header: %s"),
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (PQntuples(res) < 1)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: no data returned from server\n"), progname);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
disconnect_and_exit(1);
}
/*
* Sum up the total size, for progress reporting
*/
totalsize = totaldone = 0;
tablespacecount = PQntuples(res);
for (i = 0; i < PQntuples(res); i++)
{
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
totalsize += atol(PQgetvalue(res, i, 2));
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Verify tablespace directories are empty. Don't bother with the
* first once since it can be relocated, and it will be checked before
* we do anything anyway.
*/
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
if (format == 'p' && !PQgetisnull(res, i, 1))
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
{
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
char *path = (char *) get_tablespace_mapping(PQgetvalue(res, i, 1));
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(path, &made_tablespace_dirs, &found_tablespace_dirs);
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
/*
* When writing to stdout, require a single tablespace
*/
if (format == 't' && strcmp(basedir, "-") == 0 && PQntuples(res) > 1)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: can only write single tablespace to stdout, database has %d\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQntuples(res));
disconnect_and_exit(1);
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*
* If we're streaming WAL, start the streaming session before we start
* receiving the actual data chunks.
*/
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (includewal == STREAM_WAL)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
if (verbose)
fprintf(stderr, _("%s: starting background WAL receiver\n"),
progname);
Make pg_receivexlog and pg_basebackup -X stream work across timeline switches. This mirrors the changes done earlier to the server in standby mode. When receivelog reaches the end of a timeline, as reported by the server, it fetches the timeline history file of the next timeline, and restarts streaming from the new timeline by issuing a new START_STREAMING command. When pg_receivexlog crosses a timeline, it leaves the .partial suffix on the last segment on the old timeline. This helps you to tell apart a partial segment left in the directory because of a timeline switch, and a completed segment. If you just follow a single server, it won't make a difference, but it can be significant in more complicated scenarios where new WAL is still generated on the old timeline. This includes two small changes to the streaming replication protocol: First, when you reach the end of timeline while streaming, the server now sends the TLI of the next timeline in the server's history to the client. pg_receivexlog uses that as the next timeline, so that it doesn't need to parse the timeline history file like a standby server does. Second, when BASE_BACKUP command sends the begin and end WAL positions, it now also sends the timeline IDs corresponding the positions.
2013-01-17 19:23:00 +01:00
StartLogStreamer(xlogstart, starttli, sysidentifier);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* Start receiving chunks
*/
for (i = 0; i < PQntuples(res); i++)
{
if (format == 't')
ReceiveTarFile(conn, res, i);
else
ReceiveAndUnpackTarFile(conn, res, i);
} /* Loop over all tablespaces */
if (showprogress)
{
Limit pg_basebackup progress output to 1/second This prevents pg_basebackup from generating excessive output when dumping large clusters. The status is now updated once / second, still making it possible to see that there is progress happening, but limiting the total bandwidth. Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:47:09 +01:00
progress_report(PQntuples(res), NULL, true);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fprintf(stderr, "\n"); /* Need to move to next line */
}
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQclear(res);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
/*
* Get the stop position
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: could not get write-ahead log end position from server: %s"),
pg_basebackup: Error message improvements. Fujii Masao
2012-03-28 14:43:35 +02:00
progname, PQerrorMessage(conn));
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
disconnect_and_exit(1);
}
if (PQntuples(res) != 1)
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: no write-ahead log end position returned from server\n"),
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
progname);
disconnect_and_exit(1);
}
Prevent potential overruns of fixed-size buffers. Coverity identified a number of places in which it couldn't prove that a string being copied into a fixed-size buffer would fit. We believe that most, perhaps all of these are in fact safe, or are copying data that is coming from a trusted source so that any overrun is not really a security issue. Nonetheless it seems prudent to forestall any risk by using strlcpy() and similar functions. Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports. In addition, fix a potential null-pointer-dereference crash in contrib/chkpass. The crypt(3) function is defined to return NULL on failure, but chkpass.c didn't check for that before using the result. The main practical case in which this could be an issue is if libc is configured to refuse to execute unapproved hashing algorithms (e.g., "FIPS mode"). This ideally should've been a separate commit, but since it touches code adjacent to one of the buffer overrun changes, I included it in this commit to avoid last-minute merge issues. This issue was reported by Honza Horak. Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 17:20:21 +01:00
strlcpy(xlogend, PQgetvalue(res, 0, 0), sizeof(xlogend));
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (verbose && includewal != NO_WAL)
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: write-ahead log end point: %s\n"), progname, xlogend);
Include more status information in walsender results Add the current xlog insert location to the response of IDENTIFY_SYSTEM, and adds result sets containing start and stop location of backups to BASE_BACKUP responses.
2011-02-03 13:46:23 +01:00
PQclear(res);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
Message style cleanup
2011-05-04 19:56:52 +02:00
fprintf(stderr, _("%s: final receive failed: %s"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (bgchild > 0)
{
#ifndef WIN32
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
int status;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
int r;
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
#else
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
DWORD status;
pgindent run for 9.6
2016-06-10 00:02:36 +02:00
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
/*
* get a pointer sized version of bgchild to avoid warnings about
* casting to a different size on WIN64.
*/
intptr_t bgchild_handle = bgchild;
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
uint32 hi,
lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#endif
if (verbose)
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
pg_basebackup: Small message punctuation improvements
2012-12-19 13:01:11 +01:00
_("%s: waiting for background process to finish streaming ...\n"), progname);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#ifndef WIN32
Remove now redundant pgpipe code.
2012-03-29 05:24:07 +02:00
if (write(bgpipe[1], xlogend, strlen(xlogend)) != strlen(xlogend))
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: could not send command to background pipe: %s\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, strerror(errno));
disconnect_and_exit(1);
}
/* Just wait for the background process to exit */
r = waitpid(bgchild, &status, 0);
if (r == -1)
{
fprintf(stderr, _("%s: could not wait for child process: %s\n"),
progname, strerror(errno));
disconnect_and_exit(1);
}
if (r != bgchild)
{
Replace printf format %i by %d see also ce8d7bb6440710058503d213b2aafcdf56a5b481
2012-03-30 19:40:46 +02:00
fprintf(stderr, _("%s: child %d died, expected %d\n"),
Cast some printf arguments to avoid possibly-nonportable behavior. Per compiler warnings on buildfarm member black_firefly.
2012-03-24 01:18:04 +01:00
progname, r, (int) bgchild);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
disconnect_and_exit(1);
}
if (!WIFEXITED(status))
{
fprintf(stderr, _("%s: child process did not exit normally\n"),
progname);
disconnect_and_exit(1);
}
if (WEXITSTATUS(status) != 0)
{
Replace printf format %i by %d see also ce8d7bb6440710058503d213b2aafcdf56a5b481
2012-03-30 19:40:46 +02:00
fprintf(stderr, _("%s: child process exited with error %d\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, WEXITSTATUS(status));
disconnect_and_exit(1);
}
/* Exited normally, we're happy! */
#else /* WIN32 */
/*
* On Windows, since we are in the same process, we can just store the
* value directly in the variable, and then set the flag that says
* it's there.
*/
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
if (sscanf(xlogend, "%X/%X", &hi, &lo) != 2)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: could not parse write-ahead log location \"%s\"\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, xlogend);
Check number of fields in IDENTIFY_SYSTEM response Jaime Casanova
2012-01-20 13:57:02 +01:00
disconnect_and_exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
I missed some references to xlogid/xrecoff in Win32-only code. Fix.
2012-06-24 21:14:31 +02:00
xlogendptr = ((uint64) hi) << 32 | lo;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
InterlockedIncrement(&has_xlogendptr);
/* First wait for the thread to exit */
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
if (WaitForSingleObjectEx((HANDLE) bgchild_handle, INFINITE, FALSE) !=
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
WAIT_OBJECT_0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
_dosmaperr(GetLastError());
fprintf(stderr, _("%s: could not wait for child thread: %s\n"),
progname, strerror(errno));
disconnect_and_exit(1);
}
Support building with Visual Studio 2015 Adjust the way we detect the locale. As a result the minumum Windows version supported by VS2015 and later is Windows Vista. Add some tweaks to remove new compiler warnings. Remove documentation references to the now obsolete msysGit. Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich. Backpatch to 9.5
2016-04-29 13:59:47 +02:00
if (GetExitCodeThread((HANDLE) bgchild_handle, &status) == 0)
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
_dosmaperr(GetLastError());
fprintf(stderr, _("%s: could not get child thread exit status: %s\n"),
progname, strerror(errno));
disconnect_and_exit(1);
}
if (status != 0)
{
fprintf(stderr, _("%s: child thread exited with error %u\n"),
Miscellaneous cleanup to silence compiler warnings seen on Mingw. Remove some dead code, conditionally declare some items or call some code, and fix one or two declarations.
2011-12-11 00:15:15 +01:00
progname, (unsigned int) status);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
disconnect_and_exit(1);
}
/* Exited normally, we're happy */
#endif
}
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
/* Free the recovery.conf contents */
destroyPQExpBuffer(recoveryconfcontents);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
/*
* End of copy data. Final result is already checked inside the loop.
*/
Check number of fields in IDENTIFY_SYSTEM response Jaime Casanova
2012-01-20 13:57:02 +01:00
PQclear(res);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
PQfinish(conn);
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
/*
* Make data persistent on disk once backup is completed. For tar
* format once syncing the parent directory is fine, each tar file
* created per tablespace has been already synced. In plain format,
* all the data of the base directory is synced, taking into account
* all the tablespaces. Errors are not considered fatal.
*/
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
if (do_sync)
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
{
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
if (format == 't')
{
if (strcmp(basedir, "-") != 0)
(void) fsync_fname(basedir, true, progname);
}
else
{
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
(void) fsync_pgdata(basedir, progname, serverVersion);
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
}
pg_basebackup pg_receivexlog: Issue fsync more carefully Several places weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for details about required fsyncs. This adds a couple of functions in src/common/ that have an equivalent in the backend: durable_rename(), fsync_parent_path() From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
if (verbose)
Add missing progname prefix to some messages Author: Michael Banck
2017-02-26 21:29:34 +01:00
fprintf(stderr, _("%s: base backup completed\n"), progname);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
}
int
main(int argc, char **argv)
{
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
{"format", required_argument, NULL, 'F'},
{"checkpoint", required_argument, NULL, 'c'},
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
{"max-rate", required_argument, NULL, 'r'},
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
{"write-recovery-conf", no_argument, NULL, 'R'},
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
{"slot", required_argument, NULL, 'S'},
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
{"tablespace-mapping", required_argument, NULL, 'T'},
Rename command line options for ongoing xlog -> wal conversion. initdb and pg_basebackup now have a --waldir option rather --xlogdir, and pg_basebackup now has --wal-method rather than --xlog-method.
2017-02-09 22:42:51 +01:00
{"wal-method", required_argument, NULL, 'X'},
Add pg_basebackup -z option for compression with default level
2011-05-30 00:02:02 +02:00
{"gzip", no_argument, NULL, 'z'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"compress", required_argument, NULL, 'Z'},
{"label", required_argument, NULL, 'l'},
initdb pg_basebackup: Rename --noxxx options to --no-xxx --noclean and --nosync were the only options spelled without a hyphen, so change this for consistency with other options. The options in pg_basebackup have not been in a release, so we just rename them. For initdb, we retain the old variants. Vik Fearing and me
2016-10-19 18:00:00 +02:00
{"no-clean", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
{"dbname", required_argument, NULL, 'd'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"host", required_argument, NULL, 'h'},
{"port", required_argument, NULL, 'p'},
{"username", required_argument, NULL, 'U'},
{"no-password", no_argument, NULL, 'w'},
{"password", no_argument, NULL, 'W'},
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
{"status-interval", required_argument, NULL, 's'},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{"verbose", no_argument, NULL, 'v'},
{"progress", no_argument, NULL, 'P'},
Rename command line options for ongoing xlog -> wal conversion. initdb and pg_basebackup now have a --waldir option rather --xlogdir, and pg_basebackup now has --wal-method rather than --xlog-method.
2017-02-09 22:42:51 +01:00
{"waldir", required_argument, NULL, 1},
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
{"no-slot", no_argument, NULL, 2},
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{NULL, 0, NULL, 0}
};
int c;
int option_index;
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
bool no_slot = false;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname = get_progname(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_basebackup"));
if (argc > 1)
{
if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
{
usage();
exit(0);
}
else if (strcmp(argv[1], "-V") == 0
|| strcmp(argv[1], "--version") == 0)
{
puts("pg_basebackup (PostgreSQL) " PG_VERSION);
exit(0);
}
}
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
atexit(cleanup_directories_atexit);
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
while ((c = getopt_long(argc, argv, "D:F:r:RT:X:l:nNzZ:d:c:h:p:U:s:S:wWvP",
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
basedir = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'F':
if (strcmp(optarg, "p") == 0 || strcmp(optarg, "plain") == 0)
format = 'p';
else if (strcmp(optarg, "t") == 0 || strcmp(optarg, "tar") == 0)
format = 't';
else
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
_("%s: invalid output format \"%s\", must be \"plain\" or \"tar\"\n"),
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
progname, optarg);
exit(1);
}
break;
Allow BASE_BACKUP to be throttled A new MAX_RATE option allows imposing a limit to the network transfer rate from the server side. This is useful to limit the stress that taking a base backup has on the server. pg_basebackup is now able to specify a value to the server, too. Author: Antonin Houska Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi, Fujii Masao, and Álvaro Herrera.
2014-02-27 22:55:57 +01:00
case 'r':
maxrate = parse_max_rate(optarg);
break;
Add support for generating minimal recovery.conf when doing base backups Adds commandline option -R to pg_basebackup that creates a recovery.conf which enables standby mode using the same parameters that pg_basebackup used to connect to the master, and writes it into the output directory (or injects it in the tar file when tar format is used). Zoltan Boszormenyi, modified by Magnus Hagander, reviewed by Amit Kapila & Fujii Masao
2013-01-05 16:54:06 +01:00
case 'R':
writerecoveryconf = true;
break;
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
case 'S':
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
/*
* When specifying replication slot name, use a permanent
* slot.
*/
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
replication_slot = pg_strdup(optarg);
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
temp_replication_slot = false;
break;
case 2:
no_slot = true;
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
break;
pg_basebackup: Add support for relocating tablespaces Tablespaces can be relocated in plain backup mode by specifying one or more -T olddir=newdir options. Author: Steeve Lennmark <steevel@handeldsbanken.se> Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 19:38:06 +01:00
case 'T':
tablespace_list_append(optarg);
break;
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
case 'X':
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
if (strcmp(optarg, "n") == 0 ||
strcmp(optarg, "none") == 0)
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = NO_WAL;
Revert behaviour of -x/--xlog to 9.1 semantics To replace it, add -X/--xlog-method that allows the specification of fetch or stream. Do this to avoid unnecessary backwards-incompatiblity. Spotted and suggested by Peter Eisentraut.
2012-06-10 13:43:51 +02:00
}
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
else if (strcmp(optarg, "f") == 0 ||
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
strcmp(optarg, "fetch") == 0)
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = FETCH_WAL;
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
else if (strcmp(optarg, "s") == 0 ||
strcmp(optarg, "stream") == 0)
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
{
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
includewal = STREAM_WAL;
Make wal streaming the default mode for pg_basebackup Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
else
{
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
fprintf(stderr,
Replace reference to "xlog-method" with "wal-method" in error message. Commit 62e8b38 renamed "--xlog-method" option for pg_basebackup to "--wal-method", but forgot to update the error message mentioning that option.
2017-02-14 17:25:48 +01:00
_("%s: invalid wal-method option \"%s\", must be \"fetch\", \"stream\" or \"none\"\n"),
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
progname, optarg);
exit(1);
}
Add option to include WAL in base backup When included, this makes the base backup a complete working "clone" of the initial database, ready to have a postmaster started against it without the need to set up any log archiving or similar. Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
2011-01-30 21:30:09 +01:00
break;
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
case 1:
xlog_dir = pg_strdup(optarg);
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'l':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
label = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
case 'n':
noclean = true;
break;
pg_basebackup: Add --nosync option This is useful for testing, similar to initdb's --nosync. From: Michael Paquier <michael.paquier@gmail.com>
2016-09-29 18:00:00 +02:00
case 'N':
do_sync = false;
break;
Add pg_basebackup -z option for compression with default level
2011-05-30 00:02:02 +02:00
case 'z':
#ifdef HAVE_LIBZ
compresslevel = Z_DEFAULT_COMPRESSION;
#else
Pgindent run before 9.1 beta2.
2011-06-09 20:32:50 +02:00
compresslevel = 1; /* will be rejected below */
Add pg_basebackup -z option for compression with default level
2011-05-30 00:02:02 +02:00
#endif
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'Z':
compresslevel = atoi(optarg);
Fix pg_basebackup so that it accepts 0 as a valid compression level. The help message for pg_basebackup specifies that the numbers 0 through 9 are accepted as valid values of -Z option. But, previously -Z 0 was rejected as an invalid compression level. Per discussion, it's better to make pg_basebackup treat 0 as valid compression level meaning no compression, like pg_dump. Back-patch to all supported versions. Reported-By: Jeff Janes Reviewed-By: Amit Kapila Discussion: CAMkU=1x+GwjSayc57v6w87ij6iRGFWt=hVfM0B64b1_bPVKRqg@mail.gmail.com
2016-08-01 10:36:14 +02:00
if (compresslevel < 0 || compresslevel > 9)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
fprintf(stderr, _("%s: invalid compression level \"%s\"\n"),
progname, optarg);
exit(1);
}
break;
case 'c':
Use pg_strcasecmp instead of strcasecmp for portability Per buildfarm.
2011-01-23 17:35:02 +01:00
if (pg_strcasecmp(optarg, "fast") == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fastcheckpoint = true;
Use pg_strcasecmp instead of strcasecmp for portability Per buildfarm.
2011-01-23 17:35:02 +01:00
else if (pg_strcasecmp(optarg, "spread") == 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
fastcheckpoint = false;
else
{
fprintf(stderr, _("%s: invalid checkpoint argument \"%s\", must be \"fast\" or \"spread\"\n"),
progname, optarg);
exit(1);
}
break;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
case 'd':
connection_string = pg_strdup(optarg);
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'h':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbhost = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'p':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbport = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'U':
Standardize naming of malloc/realloc/strdup wrapper functions. We had a number of variants on the theme of "malloc or die", with the majority named like "pg_malloc", but by no means all. Standardize on the names pg_malloc, pg_malloc0, pg_realloc, pg_strdup. Get rid of pg_calloc entirely in favor of using pg_malloc0. This is an essentially cosmetic change, so no back-patch. (I did find a couple of places where psql and pg_dump were using plain malloc or strdup instead of the pg_ versions, but they don't look significant enough to bother back-patching.)
2012-10-02 21:35:10 +02:00
dbuser = pg_strdup(optarg);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
break;
case 'w':
dbgetpassword = -1;
break;
case 'W':
dbgetpassword = 1;
break;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
case 's':
Fix pg_basebackup/pg_receivexlog for floating point timestamps Since the replication protocol deals with TimestampTz, we need to care for the floating point case as well in the frontend tools. Fujii Masao, with changes from Magnus Hagander
2012-06-10 12:12:36 +02:00
standby_message_timeout = atoi(optarg) * 1000;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (standby_message_timeout < 0)
{
fprintf(stderr, _("%s: invalid status interval \"%s\"\n"),
progname, optarg);
exit(1);
}
break;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
case 'v':
verbose++;
break;
case 'P':
showprogress = true;
break;
default:
/*
* getopt_long already emitted a complaint
*/
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
/*
* Any non-option arguments?
*/
if (optind < argc)
{
fprintf(stderr,
_("%s: too many command-line arguments (first is \"%s\")\n"),
progname, argv[optind]);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Required arguments
*/
if (basedir == NULL)
{
fprintf(stderr, _("%s: no target directory specified\n"), progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Mutually exclusive arguments
*/
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (format == 'p' && compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
fprintf(stderr,
_("%s: only tar mode backups can be compressed\n"),
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Use an enum instead of two bools to indicate wal inclusion in base backups This makes the code easier to read as it becomes more explicit what the different allowed combinations really are. Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
if (format == 't' && includewal == STREAM_WAL && strcmp(basedir, "-") == 0)
Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me.
2016-12-21 12:27:37 +01:00
{
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: cannot stream write-ahead logs in tar mode to stdout\n"),
Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me.
2016-12-21 12:27:37 +01:00
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Fix incorrect comparison due to bad merge Noted by Fujii Masao
2017-01-16 18:20:57 +01:00
if (replication_slot && includewal != STREAM_WAL)
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
{
fprintf(stderr,
Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix.
2015-11-22 02:21:31 +01:00
_("%s: replication slots can only be used with WAL streaming\n"),
pg_basebackup: Add --slot option This option specifies a replication slot for WAL streaming (-X stream), so that there can be continuous replication slot use between WAL streaming during the base backup and the start of regular streaming replication. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2015-07-22 03:06:45 +02:00
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
Make pg_basebackup use temporary replication slots Temporary replication slots will be used by default when wal streaming is used and no slot name is specified with -S. If a slot name is specified, then a permanent slot with that name is used. If --no-slot is specified, then no permanent or temporary slot will be used. Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
if (no_slot)
{
if (replication_slot)
{
fprintf(stderr,
_("%s: --no-slot cannot be used with slot name\n"),
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
temp_replication_slot = false;
}
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
if (strcmp(xlog_dir, "") != 0)
{
if (format != 'p')
{
fprintf(stderr,
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
_("%s: WAL directory location can only be specified in plain mode\n"),
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/* clean up xlog directory name, check it's absolute */
canonicalize_path(xlog_dir);
if (!is_absolute_path(xlog_dir))
{
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
fprintf(stderr, _("%s: WAL directory location must be "
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
"an absolute path\n"), progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
#ifndef HAVE_LIBZ
Fix oversights in pg_basebackup's -z (compression) option. The short-form -z switch didn't work, for lack of telling getopt_long about it; and even if specified long-form, it failed to do anything, because the various tests elsewhere in the file would take Z_DEFAULT_COMPRESSION (which is -1) as meaning "don't compress". Per bug #6060 from Shigehiro Honda, though I editorialized on his patch a bit.
2011-06-15 19:15:05 +02:00
if (compresslevel != 0)
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
{
fprintf(stderr,
_("%s: this build does not support compression\n"),
progname);
exit(1);
}
#endif
/*
* Verify that the target directory exists, or create it. For plaintext
* backups, always require the directory. For tar backups, require it
* unless we are writing to stdout.
*/
if (format == 'p' || strcmp(basedir, "-") != 0)
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(basedir, &made_new_pgdata, &found_existing_pgdata);
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/* connection in replication mode to server */
conn = GetConnection();
if (!conn)
{
/* Error message already written in GetConnection() */
exit(1);
}
Replace "transaction log" with "write-ahead log" This makes documentation and error messages match the renaming of "xlog" to "wal" in APIs and file naming.
2017-05-12 17:49:56 +02:00
/* Create pg_wal symlink, if required */
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
if (strcmp(xlog_dir, "") != 0)
{
char *linkloc;
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
verify_dir_is_empty_or_create(xlog_dir, &made_new_xlogdir, &found_existing_xlogdir);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
/*
* Form name of the place where the symlink must go. pg_xlog has
* been renamed to pg_wal in post-10 clusters.
*/
linkloc = psprintf("%s/%s", basedir,
PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
"pg_xlog" : "pg_wal");
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
#ifdef HAVE_SYMLINK
if (symlink(xlog_dir, linkloc) != 0)
{
fprintf(stderr, _("%s: could not create symbolic link \"%s\": %s\n"),
progname, linkloc, strerror(errno));
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
disconnect_and_exit(1);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
}
#else
add missing newline
2014-12-15 20:49:41 +01:00
fprintf(stderr, _("%s: symlinks are not supported on this platform\n"));
Rename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-20 17:24:37 +02:00
disconnect_and_exit(1);
Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory. Haribabu kommi, slightly modified by me.
2013-11-27 06:00:16 +01:00
#endif
free(linkloc);
}
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
BaseBackup();
pg_basebackup: Clean created directories on failure Like initdb, clean up created data and xlog directories, unless the new -n/--noclean option is specified. Tablespace directories are not cleaned up, but a message is written about that. Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
2016-09-12 18:00:00 +02:00
success = true;
Add pg_basebackup tool for streaming base backups This tool makes it possible to do the pg_start_backup/ copy files/pg_stop_backup step in a single command. There are still some steps to be done before this is a complete backup solution, such as the ability to stream the required WAL logs, but it's still usable, and could do with some buildfarm coverage. In passing, make the checkpoint request optionally fast instead of hardcoding it. Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
2011-01-23 12:21:23 +01:00
return 0;
}