rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/bin/pg_basebackup/streamutil.c

501 lines
11 KiB
C
Raw Normal View History

Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
/*-------------------------------------------------------------------------
*
* streamutil.c - utility functions for pg_basebackup and pg_receivelog
*
* Author: Magnus Hagander <magnus@hagander.net>
*
Update copyright for 2014 Update all files in head, and files COPYRIGHT and legal.sgml in all back branches.
2014-01-07 22:05:30 +01:00
* Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
*
* IDENTIFICATION
* src/bin/pg_basebackup/streamutil.c
*-------------------------------------------------------------------------
*/
Make xlog_internal.h includable in frontend context. This makes unnecessary the ugly hack used to #include postgres.h in pg_basebackup. Based on Alvaro Herrera's patch
2012-12-13 13:59:13 +01:00
#include "postgres_fe.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
#include <stdio.h>
#include <string.h>
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
/* for ntohl/htonl */
#include <netinet/in.h>
#include <arpa/inet.h>
/* local includes */
#include "receivelog.h"
#include "streamutil.h"
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
#include "pqexpbuffer.h"
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
#include "common/fe_memutils.h"
#include "datatype/timestamp.h"
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
const char *progname;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
char *connection_string = NULL;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
char *dbhost = NULL;
char *dbuser = NULL;
char *dbport = NULL;
Introduce replication slots. Replication slots are a crash-safe data structure which can be created on either a master or a standby to prevent premature removal of write-ahead log segments needed by a standby, as well as (with hot_standby_feedback=on) pruning of tuples whose removal would cause replication conflicts. Slots have some advantages over existing techniques, as explained in the documentation. In a few places, we refer to the type of replication slots introduced by this patch as "physical" slots, because forthcoming patches for logical decoding will also have slots, but with somewhat different properties. Andres Freund and Robert Haas
2014-02-01 04:45:17 +01:00
char *replication_slot = NULL;
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
char *dbname = NULL;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
static char *dbpassword = NULL;
PGconn *conn = NULL;
Make pg_recievexlog by default loop on connection failures Avoids the need for an external script in the most common scenario. Behavior can be overridden using the -n/--noloop commandline parameter.
2012-05-27 11:05:24 +02:00
/*
* Connect to the server. Returns a valid PGconn pointer if connected,
* or NULL on non-permanent error. On permanent error, the function will
* call exit(1) directly.
*/
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
PGconn *
GetConnection(void)
{
PGconn *tmpconn;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
int argcount = 7; /* dbname, replication, fallback_app_name,
* host, user, port, password */
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
int i;
const char **keywords;
const char **values;
Prevent pg_basebackup when integer_datetimes flag doesn't match. Magnus Hagander, reviewed by Fujii Masao, with slight wording changes by me.
2012-05-22 16:02:47 +02:00
const char *tmpparam;
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
bool need_password;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
PQconninfoOption *conn_opts = NULL;
PQconninfoOption *conn_opt;
char *err_msg = NULL;
/*
* Merge the connection info inputs given in form of connection string,
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
* options and default values (dbname=replication, replication=true, etc.)
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
*/
i = 0;
if (connection_string)
{
conn_opts = PQconninfoParse(connection_string, &err_msg);
if (conn_opts == NULL)
{
Fix PQconninfoParse error message handling The returned error message already includes a newline, but the callers were adding their own when printing it out.
2013-07-16 02:04:14 +02:00
fprintf(stderr, "%s: %s", progname, err_msg);
pg_receivexlog - Exit on failure to parse In streamutil.c:GetConnection(), upgrade failure to parse the connection string to an exit(1) instead of simply returning NULL. Most callers already immediately exited, but pg_receivexlog would loop on this case, continually trying to re-parse the connection string (which can't be changed after pg_receivexlog has started). GetConnection() was already expected to exit(1) in some cases (eg: failure to allocate memory or if unable to determine the integer_datetimes flag), so this change shouldn't surprise anyone. Began looking at this due to the Coverity scanner complaining that we were leaking err_msg in this case- no longer an issue since we just exit(1) immediately.
2013-07-14 21:31:23 +02:00
exit(1);
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
}
for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
{
if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
argcount++;
}
keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
values = pg_malloc0((argcount + 1) * sizeof(*values));
for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
{
if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
{
keywords[i] = conn_opt->keyword;
values[i] = conn_opt->val;
i++;
}
}
}
else
{
keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
values = pg_malloc0((argcount + 1) * sizeof(*values));
}
keywords[i] = "dbname";
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
values[i] = dbname == NULL ? "replication" : dbname;
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
i++;
keywords[i] = "replication";
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
values[i] = dbname == NULL ? "true" : "database";
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
i++;
keywords[i] = "fallback_application_name";
values[i] = progname;
i++;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (dbhost)
{
keywords[i] = "host";
values[i] = dbhost;
i++;
}
if (dbuser)
{
keywords[i] = "user";
values[i] = dbuser;
i++;
}
if (dbport)
{
keywords[i] = "port";
values[i] = dbport;
i++;
}
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
/* If -W was given, force prompt for password, but only the first time */
need_password = (dbgetpassword == 1 && dbpassword == NULL);
Tweak streamutil.c further to avoid scan-build warning The previous change added a new scan-build warning about need_password assigned but not read.
2013-11-22 03:46:43 +01:00
do
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
/* Get a new password if appropriate */
if (need_password)
{
if (dbpassword)
free(dbpassword);
dbpassword = simple_prompt(_("Password: "), 100, false);
need_password = false;
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
/* Use (or reuse, on a subsequent connection) password if we have it */
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (dbpassword)
{
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
keywords[i] = "password";
values[i] = dbpassword;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
else
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
{
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
keywords[i] = NULL;
values[i] = NULL;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
tmpconn = PQconnectdbParams(keywords, values, true);
Fix memory and file descriptor leaks in pg_receivexlog/pg_basebackup When the internal loop mode was added, freeing memory and closing filedescriptors before returning became important, and a few cases in the code missed that. Fujii Masao
2012-07-12 13:31:19 +02:00
/*
* If there is too little memory even to allocate the PGconn object
* and PQconnectdbParams returns NULL, we call exit(1) directly.
*/
if (!tmpconn)
{
fprintf(stderr, _("%s: could not connect to server\n"),
progname);
exit(1);
}
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
/* If we need a password and -w wasn't given, loop back and get one */
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
if (PQstatus(tmpconn) == CONNECTION_BAD &&
PQconnectionNeedsPassword(tmpconn) &&
dbgetpassword != -1)
{
Fix memory and file descriptor leaks in pg_receivexlog/pg_basebackup When the internal loop mode was added, freeing memory and closing filedescriptors before returning became important, and a few cases in the code missed that. Fujii Masao
2012-07-12 13:31:19 +02:00
PQfinish(tmpconn);
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
need_password = true;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
}
Tweak streamutil.c further to avoid scan-build warning The previous change added a new scan-build warning about need_password assigned but not read.
2013-11-22 03:46:43 +01:00
while (need_password);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
if (PQstatus(tmpconn) != CONNECTION_OK)
{
fprintf(stderr, _("%s: could not connect to server: %s\n"),
progname, PQerrorMessage(tmpconn));
PQfinish(tmpconn);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
free(values);
free(keywords);
Add -d option to pg_basebackup and pg_receivexlog, for connection string. Without this, there's no way to pass arbitrary libpq connection parameters to these applications. It's a bit strange that the option is called -d/--dbname, when in fact you can *not* pass a database name in it, but it's consistent with other client applications where a connection string is also passed using -d. Original patch by Amit Kapila, heavily modified by me.
2013-02-25 13:48:27 +01:00
if (conn_opts)
PQconninfoFree(conn_opts);
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
return NULL;
}
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
/* Connection ok! */
free(values);
free(keywords);
if (conn_opts)
PQconninfoFree(conn_opts);
/*
* Ensure we have the same value of integer timestamps as the server we
* are connecting to.
*/
tmpparam = PQparameterStatus(tmpconn, "integer_datetimes");
if (!tmpparam)
{
fprintf(stderr,
_("%s: could not determine server setting for integer_datetimes\n"),
progname);
PQfinish(tmpconn);
exit(1);
}
Prevent pg_basebackup when integer_datetimes flag doesn't match. Magnus Hagander, reviewed by Fujii Masao, with slight wording changes by me.
2012-05-22 16:02:47 +02:00
#ifdef HAVE_INT64_TIMESTAMP
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
if (strcmp(tmpparam, "on") != 0)
Prevent pg_basebackup when integer_datetimes flag doesn't match. Magnus Hagander, reviewed by Fujii Masao, with slight wording changes by me.
2012-05-22 16:02:47 +02:00
#else
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
if (strcmp(tmpparam, "off") != 0)
Prevent pg_basebackup when integer_datetimes flag doesn't match. Magnus Hagander, reviewed by Fujii Masao, with slight wording changes by me.
2012-05-22 16:02:47 +02:00
#endif
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
{
fprintf(stderr,
pg_basebackup: stylistic adjustments The most user-visible part of this is to change the long options --statusint and --noloop to --status-interval and --no-loop, respectively, per discussion. Also, consistently enclose file names in double quotes, per our conventions; and consistently use the term "transaction log file" to talk about WAL segments. (Someday we may need to go over this terminology and make it consistent across the whole source code.) Finally, reflow the code to better fit in 80 columns, and have pgindent fix it up some more.
2012-07-31 16:11:11 +02:00
_("%s: integer_datetimes compile flag does not match server\n"),
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
progname);
PQfinish(tmpconn);
exit(1);
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Clean up password prompting logic in streamutil.c. The previous coding was fairly unreadable and drew double-free warnings from clang. I believe the double free was actually not reachable, because PQconnectionNeedsPassword is coded to not return true if a password was provided, so that the loop can't iterate more than twice. Nonetheless it seems worth rewriting. No back-patch since this is just cosmetic.
2013-11-15 23:27:41 +01:00
return tmpconn;
Implement streaming xlog for backup tools Add option for parallel streaming of the transaction log while a base backup is running, to get the logfiles before the server has removed them. Also add a tool called pg_receivexlog, which streams the transaction log into files, creating a log archive without having to wait for segments to complete, thus decreasing the window of data loss without having to waste space using archive_timeout. This works best in combination with archive_command - suggested usage docs etc coming later.
2011-10-26 20:13:33 +02:00
}
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
/*
* Run IDENTIFY_SYSTEM through a given connection and give back to caller
* some result information if requested:
* - Start LSN position
* - Current timeline ID
* - System identifier
* - Plugin name
*/
bool
RunIdentifySystem(PGconn *conn, char **sysid, TimeLineID *starttli,
XLogRecPtr *startpos, char **db_name)
{
PGresult *res;
uint32 hi, lo;
/* Check connection existence */
Assert(conn != NULL);
res = PQexec(conn, "IDENTIFY_SYSTEM");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
fprintf(stderr, _("%s: could not send replication command \"%s\": %s"),
progname, "IDENTIFY_SYSTEM", PQerrorMessage(conn));
return false;
}
if (PQntuples(res) != 1 || PQnfields(res) < 3)
{
fprintf(stderr,
_("%s: could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields\n"),
progname, PQntuples(res), PQnfields(res), 1, 3);
return false;
}
/* Get system identifier */
if (sysid != NULL)
*sysid = pg_strdup(PQgetvalue(res, 0, 0));
/* Get timeline ID to start streaming from */
if (starttli != NULL)
*starttli = atoi(PQgetvalue(res, 0, 1));
/* Get LSN start position if necessary */
if (startpos != NULL)
{
if (sscanf(PQgetvalue(res, 0, 2), "%X/%X", &hi, &lo) != 2)
{
fprintf(stderr,
_("%s: could not parse transaction log location \"%s\"\n"),
progname, PQgetvalue(res, 0, 2));
return false;
}
*startpos = ((uint64) hi) << 32 | lo;
}
/* Get database name, only available in 9.4 and newer versions */
if (db_name != NULL)
{
if (PQnfields(res) < 4)
fprintf(stderr,
_("%s: could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields\n"),
progname, PQntuples(res), PQnfields(res), 1, 4);
if (PQgetisnull(res, 0, 3))
*db_name = NULL;
else
*db_name = pg_strdup(PQgetvalue(res, 0, 3));
}
PQclear(res);
return true;
}
/*
* Create a replication slot for the given connection. This function
* returns true in case of success as well as the start position
* obtained after the slot creation.
*/
bool
CreateReplicationSlot(PGconn *conn, const char *slot_name, const char *plugin,
XLogRecPtr *startpos, bool is_physical)
{
PQExpBuffer query;
PGresult *res;
query = createPQExpBuffer();
Assert((is_physical && plugin == NULL) ||
(!is_physical && plugin != NULL));
Assert(slot_name != NULL);
/* Build query */
if (is_physical)
appendPQExpBuffer(query, "CREATE_REPLICATION_SLOT \"%s\" PHYSICAL",
slot_name);
else
appendPQExpBuffer(query, "CREATE_REPLICATION_SLOT \"%s\" LOGICAL \"%s\"",
slot_name, plugin);
res = PQexec(conn, query->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
fprintf(stderr, _("%s: could not send replication command \"%s\": %s"),
progname, query->data, PQerrorMessage(conn));
return false;
}
if (PQntuples(res) != 1 || PQnfields(res) != 4)
{
fprintf(stderr,
_("%s: could not create replication slot \"%s\": got %d rows and %d fields, expected %d rows and %d fields\n"),
progname, slot_name,
PQntuples(res), PQnfields(res), 1, 4);
return false;
}
/* Get LSN start position if necessary */
if (startpos != NULL)
{
uint32 hi, lo;
if (sscanf(PQgetvalue(res, 0, 1), "%X/%X", &hi, &lo) != 2)
{
fprintf(stderr,
_("%s: could not parse transaction log location \"%s\"\n"),
progname, PQgetvalue(res, 0, 1));
return false;
}
*startpos = ((uint64) hi) << 32 | lo;
}
PQclear(res);
return true;
}
/*
* Drop a replication slot for the given connection. This function
* returns true in case of success.
*/
bool
DropReplicationSlot(PGconn *conn, const char *slot_name)
{
PQExpBuffer query;
PGresult *res;
Assert(slot_name != NULL);
query = createPQExpBuffer();
/* Build query */
appendPQExpBuffer(query, "DROP_REPLICATION_SLOT \"%s\"",
slot_name);
res = PQexec(conn, query->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
fprintf(stderr, _("%s: could not send replication command \"%s\": %s"),
progname, query->data, PQerrorMessage(conn));
return false;
}
if (PQntuples(res) != 0 || PQnfields(res) != 0)
{
fprintf(stderr,
_("%s: could not drop replication slot \"%s\": got %d rows and %d fields, expected %d rows and %d fields\n"),
progname, slot_name,
PQntuples(res), PQnfields(res), 0, 0);
return false;
}
PQclear(res);
return true;
}
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
/*
* Frontend version of GetCurrentTimestamp(), since we are not linked with
Refactor replication connection code of various pg_basebackup utilities. Move some more code to manage replication connection command to streamutil.c. A later patch will introduce replication slot via pg_receivexlog and this avoid duplicating relevant code between pg_receivexlog and pg_recvlogical. Author: Michael Paquier, with some editing by me.
2014-10-01 17:22:21 +02:00
* backend code. The replication protocol always uses integer timestamps,
* regardless of the server setting.
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
*/
int64
feGetCurrentTimestamp(void)
{
int64 result;
struct timeval tp;
gettimeofday(&tp, NULL);
result = (int64) tp.tv_sec -
((POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY);
result = (result * USECS_PER_SEC) + tp.tv_usec;
return result;
}
/*
* Frontend version of TimestampDifference(), since we are not linked with
* backend code.
*/
void
feTimestampDifference(int64 start_time, int64 stop_time,
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
long *secs, int *microsecs)
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
{
int64 diff = stop_time - start_time;
if (diff <= 0)
{
*secs = 0;
*microsecs = 0;
}
else
{
*secs = (long) (diff / USECS_PER_SEC);
*microsecs = (int) (diff % USECS_PER_SEC);
}
}
/*
* Frontend version of TimestampDifferenceExceeds(), since we are not
* linked with backend code.
*/
bool
feTimestampDifferenceExceeds(int64 start_time,
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
int64 stop_time,
int msec)
Add pg_recvlogical, a tool to receive data logical decoding data. This is fairly basic at the moment, but it's at least useful for testing and debugging, and possibly more. Andres Freund
2014-03-18 17:19:57 +01:00
{
int64 diff = stop_time - start_time;
return (diff >= msec * INT64CONST(1000));
}
/*
* Converts an int64 to network byte order.
*/
void
fe_sendint64(int64 i, char *buf)
{
uint32 n32;
/* High order half first, since we're doing MSB-first */
n32 = (uint32) (i >> 32);
n32 = htonl(n32);
memcpy(&buf[0], &n32, 4);
/* Now the low order half */
n32 = (uint32) i;
n32 = htonl(n32);
memcpy(&buf[4], &n32, 4);
}
/*
* Converts an int64 from network byte order to native format.
*/
int64
fe_recvint64(char *buf)
{
int64 result;
uint32 h32;
uint32 l32;
memcpy(&h32, buf, 4);
memcpy(&l32, buf + 4, 4);
h32 = ntohl(h32);
l32 = ntohl(l32);
result = h32;
result <<= 32;
result |= l32;
return result;
}