Commit Graph

415 Commits

Author SHA1 Message Date
Tom Lane bdf46af748 Post-feature-freeze pgindent run.
Discussion: https://postgr.es/m/15719.1523984266@sss.pgh.pa.us
2018-04-26 14:47:16 -04:00
Noah Misch 6336b6dfc5 Correct pg_recvlogical server version test.
The predecessor test boiled down to "PQserverVersion(NULL) >= 100000",
which is always false.  No release includes that, so it could not have
reintroduced CVE-2018-1058.  Back-patch to 9.4, like the addition of the
predecessor in commit 8d2814f274.

Discussion: https://postgr.es/m/20180422215551.GB2676194@rfd.leadboat.com
2018-04-25 18:50:29 -07:00
Tom Lane f04d4ac919 Reindent Perl files with perltidy version 20170521.
Discussion: https://postgr.es/m/CABUevEzK3cNiHZQ18f5tK0guoT+cN_jWeVzhYYxY=r+1Q3SmoA@mail.gmail.com
2018-04-25 14:00:19 -04:00
Magnus Hagander d7754822c5 Silence some warnings in TAP tests
Author: Michael Paquier
2018-04-09 21:46:17 +02:00
Stephen Frost c37b3d08ca Allow group access on PGDATA
Allow the cluster to be optionally init'd with read access for the
group.

This means a relatively non-privileged user can perform a backup of the
cluster without requiring write privileges, which enhances security.

The mode of PGDATA is used to determine whether group permissions are
enabled for directory and file creates.  This method was chosen as it's
simple and works well for the various utilities that write into PGDATA.

Changing the mode of PGDATA manually will not automatically change the
mode of all the files contained therein.  If the user would like to
enable group access on an existing cluster then changing the mode of all
the existing files will be required.  Note that pg_upgrade will
automatically change the mode of all migrated files if the new cluster
is init'd with the -g option.

Tests are included for the backend and all the utilities which operate
on the PG data directory to ensure that the correct mode is set based on
the data directory permissions.

Author: David Steele <david@pgmasters.net>
Reviewed-By: Michael Paquier, with discussion amongst many others.
Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 17:45:39 -04:00
Stephen Frost da9b580d89 Refactor dir/file permissions
Consolidate directory and file create permissions for tools which work
with the PG data directory by adding a new module (common/file_perm.c)
that contains variables (pg_file_create_mode, pg_dir_create_mode) and
constants to initialize them (0600 for files and 0700 for directories).

Convert mkdir() calls in the backend to MakePGDirectory() if the
original call used default permissions (always the case for regular PG
directories).

Add tests to make sure permissions in PGDATA are set correctly by the
tools which modify the PG data directory.

Authors: David Steele <david@pgmasters.net>,
         Adam Brightwell <adam.brightwell@crunchydata.com>
Reviewed-By: Michael Paquier, with discussion amongst many others.
Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-04-07 17:45:39 -04:00
Magnus Hagander 032429701e Clean up intermetiate state in pg_basebackup tests
These tests accummulated almost a gigabyte of data during the test which
was then removed at the end. Instead, remove output that's no longer
needed between the individual tests, to keep the total disk usage down
lower.

Author: Michael Banck
2018-04-06 22:26:31 +02:00
Magnus Hagander f66c37b2f7 Fix typo
Author: Michael Banck
2018-04-06 22:23:35 +02:00
Magnus Hagander ee9e145531 Fix pg_bsaebackup checksum tests
Hopefully fix the fact that these checks are unstable, by introducing
the corruption in a separate table from pg_class, and also explicitly
disable autovacuum on those tables. Also make sure PostgreSQL is
stopped while the corruption is introduced to avoid possible caching
effects.

Author: Michael Banck
2018-04-04 11:37:55 +02:00
Tom Lane dddfc4cb2e Prevent accidental linking of system-supplied copies of libpq.so etc.
We were being careless in some places about the order of -L switches in
link command lines, such that -L switches referring to external directories
could come before those referring to directories within the build tree.
This made it possible to accidentally link a system-supplied library, for
example /usr/lib/libpq.so, in place of the one built in the build tree.
Hilarity ensued, the more so the older the system-supplied library is.

To fix, break LDFLAGS into two parts, a sub-variable LDFLAGS_INTERNAL
and the main LDFLAGS variable, both of which are "recursively expanded"
so that they can be incrementally adjusted by different makefiles.
Establish a policy that -L switches for directories in the build tree
must always be added to LDFLAGS_INTERNAL, while -L switches for external
directories must always be added to LDFLAGS.  This is sufficient to
ensure a safe search order.  For simplicity, we typically also put -l
switches for the respective libraries into those same variables.
(Traditional make usage would have us put -l switches into LIBS, but
cleaning that up is a project for another day, as there's no clear
need for it.)

This turns out to also require separating SHLIB_LINK into two variables,
SHLIB_LINK and SHLIB_LINK_INTERNAL, with a similar rule about which
switches go into which variable.  And likewise for PG_LIBS.

Although this change might appear to affect external users of pgxs.mk,
I think it doesn't; they shouldn't have any need to touch the _INTERNAL
variables.

In passing, tweak src/common/Makefile so that the value of CPPFLAGS
recorded in pg_config lacks "-DFRONTEND" and the recorded value of
LDFLAGS lacks "-L../../../src/common".  Both of those things are
mistakes, apparently introduced during prior code rearrangements,
as old versions of pg_config don't print them.  In general we don't
want anything that's specific to the src/common subdirectory to
appear in those outputs.

This is certainly a bug fix, but in view of the lack of field
complaints, I'm unsure whether it's worth the risk of back-patching.
In any case it seems wise to see what the buildfarm makes of it first.

Discussion: https://postgr.es/m/25214.1522604295@sss.pgh.pa.us
2018-04-03 16:26:05 -04:00
Magnus Hagander 4eb77d50c2 Validate page level checksums in base backups
When base backups are run over the replication protocol (for example
using pg_basebackup), verify the checksums of all data blocks if
checksums are enabled. If checksum failures are encountered, log them
as warnings but don't abort the backup.

This becomes the default behaviour in pg_basebackup (provided checksums
are enabled on the server), so add a switch (-k) to disable the checks
if necessary.

Author: Michael Banck
Reviewed-By: Magnus Hagander, David Steele
Discussion: https://postgr.es/m/20180228180856.GE13784@nighthawk.caipicrew.dd-dns.de
2018-04-03 13:47:16 +02:00
Tom Lane 0b11a674fb Fix a boatload of typos in C comments.
Justin Pryzby

Discussion: https://postgr.es/m/20180331105640.GK28454@telsasoft.com
2018-04-01 15:01:28 -04:00
Teodor Sigaev 44bd95846a Fix count of skipped test of basebackup on Windows
Commit 920a5e500a add tests which should be
skipped on Windows boxes, but patch doesn't contain right count of them.

David Steel
2018-03-27 17:40:56 +03:00
Teodor Sigaev 920a5e500a Skip temp tables from basebackup.
Do not store temp tables in basebackup, they will not be visible anyway, so,
there are not reasons to store them.

Author: David Steel
Reviewed by: me
Discussion: https://www.postgresql.org/message-id/flat/5ea4d26a-a453-c1b7-eff9-5a3ef8f8aceb@pgmasters.net
2018-03-27 16:14:40 +03:00
Teodor Sigaev 8694cc96b5 Exclude unlogged tables from base backups
Exclude unlogged tables from base backup entirely except init fork which marks
created unlogged table. The next question is do not backup temp table but
it's a story for separate patch.

Author: David Steele
Review by: Adam Brightwell, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/04791bab-cb04-ba43-e9c0-664a4c1ffb2c@pgmasters.net
2018-03-23 19:14:12 +03:00
Magnus Hagander 8d2814f274 Fix pg_recvlogical for pre-10 versions
In e170b8c8, protection against modified search_path was added. However,
PostgreSQL versions prior to 10 does not accept SQL commands over a
replication connection, so the protection would generate a syntax error.

Since we cannot run SQL commands on it, we are also not vulnerable to
the issue that e170b8c8 fixes, so we can just skip this command for
older versions.

Author: Michael Paquier <michael@paquier.xyz>
2018-03-18 13:08:25 +01:00
Noah Misch 582edc369c Empty search_path in Autovacuum and non-psql/pgbench clients.
This makes the client programs behave as documented regardless of the
connect-time search_path and regardless of user-created objects.  Today,
a malicious user with CREATE permission on a search_path schema can take
control of certain of these clients' queries and invoke arbitrary SQL
functions under the client identity, often a superuser.  This is
exploitable in the default configuration, where all users have CREATE
privilege on schema "public".

This changes behavior of user-defined code stored in the database, like
pg_index.indexprs and pg_extension_config_dump().  If they reach code
bearing unqualified names, "does not exist" or "no schema has been
selected to create in" errors might appear.  Users may fix such errors
by schema-qualifying affected names.  After upgrading, consider watching
server logs for these errors.

The --table arguments of src/bin/scripts clients have been lax; for
example, "vacuumdb -Zt pg_am\;CHECKPOINT" performed a checkpoint.  That
now fails, but for now, "vacuumdb -Zt 'pg_am(amname);CHECKPOINT'" still
performs a checkpoint.

Back-patch to 9.3 (all supported versions).

Reviewed by Tom Lane, though this fix strategy was not his first choice.
Reported by Arseniy Sharoglazov.

Security: CVE-2018-1058
2018-02-26 07:39:44 -08:00
Bruce Momjian 9d4649ca49 Update copyright for 2018
Backpatch-through: certain files through 9.3
2018-01-02 23:30:12 -05:00
Alvaro Herrera 5303ffe71b Fix typo 2017-12-29 18:33:32 -03:00
Peter Eisentraut 143b54d21d pg_basebackup: Fix progress messages when writing to a file
The progress messages print out \r to keep overwriting the same line on
the screen.  But this does not yield useful results when writing the
output to a file.  So in that case, print out \n instead.

Author: Martín Marqués <martin@2ndquadrant.com>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
2017-12-01 09:21:34 -05:00
Robert Haas eaedf0df71 Update typedefs.list and re-run pgindent
Discussion: http://postgr.es/m/CA+TgmoaA9=1RWKtBWpDaj+sF3Stgc8sHgf5z=KGtbjwPLQVDMA@mail.gmail.com
2017-11-29 09:24:24 -05:00
Tom Lane 0772c152b9 Mark some more functions as pg_attribute_noreturn().
Doing this suppresses Coverity warnings and might allow improved
code in some cases.  The prospects of that are not so bright as
to warrant back-patching, though.

Michael Paquier, per Coverity
2017-11-27 20:56:46 -05:00
Simon Riggs 98267ee83e Exclude pg_internal.init from BASE_BACKUP
Add docs to explain this for other backup mechanisms

Author: David Steele <david@pgmasters.net>
Reviewed-by: Petr Jelinek <petr.jelinek@2ndQuadrant.com> et al
2017-11-07 12:28:35 -05:00
Magnus Hagander 86bc521811 Fix comment
Author: Bernd Helmle <mailings@oopsware.de>
2017-11-05 11:48:20 -05:00
Peter Eisentraut 067a2259fd pg_basebackup: Fix comparison handling of tablespace mappings on Windows
A candidate path needs to be canonicalized before being checked against
the mappings, because the mappings are also canonicalized.  This is
especially relevant on Windows

Reported-by: nb <nbedxp@gmail.com>
Author: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Ashutosh Sharma <ashu.coek88@gmail.com>
2017-11-01 10:20:05 -04:00
Robert Haas 5f3971291f pg_receivewal: Add --no-sync option.
Michael Paquier, reviewed by Kuntal Ghosh and by me.  I did a little
wordsmithing on the documentation, too.

Discussion: http://postgr.es/m/CAB7nPqTuXuyEoVKcWcExh_b0uAjgWd_14KfGLrCTccBZ=VA0KA@mail.gmail.com
2017-10-29 12:46:55 +05:30
Andres Freund 0ba99c84e8 Replace most usages of ntoh[ls] and hton[sl] with pg_bswap.h.
All postgres internal usages are replaced, it's just libpq example
usages that haven't been converted. External users of libpq can't
generally rely on including postgres internal headers.

Note that this includes replacing open-coded byte swapping of 64bit
integers (using two 32 bit swaps) with a single 64bit swap.

Where it looked applicable, I have removed netinet/in.h and
arpa/inet.h usage, which previously provided the relevant
functionality. It's perfectly possible that I missed other reasons for
including those, the buildfarm will tell.

Author: Andres Freund
Discussion: https://postgr.es/m/20170927172019.gheidqy6xvlxb325@alap3.anarazel.de
2017-10-01 15:36:14 -07:00
Peter Eisentraut 3709ca1cf0 pg_basebackup: Add option to create replication slot
When requesting a particular replication slot, the new pg_basebackup
option -C/--create-slot creates it before starting to replicate from it.

Further refactor the slot creation logic to include the temporary slot
creation logic into the same function.  Add new arguments is_temporary
and preserve_wal to CreateReplicationSlot().  Print in --verbose mode
that a slot has been created.

Author: Michael Banck <michael.banck@credativ.de>
2017-09-27 08:49:47 -04:00
Peter Eisentraut fa41461205 Add some more pg_receivewal tests
Add some more tests for the --create-slot and --drop-slot options,
verifying that the right kind of slot was created and that the slot was
dropped.  While working on an unrelated patch for pg_basebackup, some of
this was temporarily broken without any tests noticing.
2017-09-26 16:41:20 -04:00
Peter Eisentraut 15a8010ed6 Sort pg_basebackup options better
The --slot option somehow ended up under options controlling the output,
and some other options were in a nonsensical place or were not moved
after recent renamings, so tidy all that up a bit.
2017-09-26 11:58:22 -04:00
Peter Eisentraut 74ca8f9b90 Fix pg_basebackup test to original intent
One test case was meant to check that pg_basebackup does not succeed
when a slot is specified with -S but WAL streaming is not selected,
which used to require specifying -X stream.  Since -X stream is the
default in PostgreSQL 10, this test case no longer covers that meaning,
but the pg_basebackup invocation happened to fail anyway for the
unrelated reason that the specified replication slot does not exist.  To
fix, move the test case to later in the file where the slot does exist,
and add -X none to the invocation so that it covers the originally meant
behavior.

extracted from a patch by Michael Banck <michael.banck@credativ.de>
2017-09-23 22:59:26 -04:00
Andres Freund fc49e24fa6 Make WAL segment size configurable at initdb time.
For performance reasons a larger segment size than the default 16MB
can be useful. A larger segment size has two main benefits: Firstly,
in setups using archiving, it makes it easier to write scripts that
can keep up with higher amounts of WAL, secondly, the WAL has to be
written and synced to disk less frequently.

But at the same time large segment size are disadvantageous for
smaller databases. So far the segment size had to be configured at
compile time, often making it unrealistic to choose one fitting to a
particularly load. Therefore change it to a initdb time setting.

This includes a breaking changes to the xlogreader.h API, which now
requires the current segment size to be configured.  For that and
similar reasons a number of binaries had to be taught how to recognize
the current segment size.

Author: Beena Emerson, editorialized by Andres Freund
Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael
    Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja
Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-19 22:03:48 -07:00
Peter Eisentraut 6d9fa52645 pg_receivewal: Add --endpos option
This is primarily useful for making tests of this utility more
deterministic, to avoid the complexity of starting pg_receivewal as a
deamon in TAP tests.

While this is less useful than the equivalent pg_recvlogical option,
users can as well use it for example to enforce WAL streaming up to a
end-of-backup position, to save only a minimal amount of WAL.

Use this new option to stream WAL data in a deterministic way within a
new set of TAP tests.

Author: Michael Paquier <michael.paquier@gmail.com>
2017-09-11 16:48:30 -04:00
Peter Eisentraut 8e67380126 Remove useless empty string initializations
This coding style probably stems from the days of shell scripts.

Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-09-08 12:37:05 -04:00
Peter Eisentraut 514731af58 pg_receivewal: Improve verbose mode
Some informational messages showed up even if verbose mode was not
used.  Move them to verbose mode.

Author: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
2017-08-15 20:25:37 -04:00
Peter Eisentraut f7668b2b35 Translation updates
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 1a0b5e655d7871506c2b1c7ba562c2de6b6a55de
2017-08-07 13:55:34 -04:00
Peter Eisentraut 26d40ada3f Message style improvements 2017-08-04 18:31:32 -04:00
Peter Eisentraut 5ff3d73813 Add new files to nls.mk and add translation markers 2017-08-02 22:45:48 -04:00
Tom Lane c95275fc20 Fix broken link-command-line ordering for libpgfeutils.
In the frontend Makefiles that pull in libpgfeutils, we'd generally
done it like this:

LDFLAGS += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)

That method is badly broken, as seen in bug #14742 from Chris Ruprecht.
The -L flag for src/fe_utils ends up being placed after whatever random
-L flags are in LDFLAGS already.  That puts us at risk of pulling in
libpgfeutils.a from some previous installation rather than the freshly
built one in src/fe_utils.  Also, the lack of an "override" is hazardous
if someone tries to specify some LDFLAGS on the make command line.

The correct way to do it is like this:

override LDFLAGS := -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(LDFLAGS)

so that libpgfeutils, along with libpq, libpgport, and libpgcommon, are
guaranteed to be pulled in from the build tree and not from any referenced
system directory, because their -L flags will appear first.

In some places we'd been even lazier and done it like this:

LDFLAGS += -L$(top_builddir)/src/fe_utils -lpgfeutils -lpq

which is subtly wrong in an additional way: on platforms where we can't
restrict the symbols exported by libpq.so, it allows libpgfeutils to
latch onto libpgport and libpgcommon symbols from libpq.so, rather than
directly from those static libraries as intended.  This carries hazards
like those explained in the comments for the libpq_pgport macro.

In addition to fixing the broken libpgfeutils usages, I tried to
standardize on using $(libpq_pgport) like so:

override LDFLAGS := $(libpq_pgport) $(LDFLAGS)

even where libpgfeutils is not in the picture.  This makes no difference
right now but will hopefully discourage future mistakes of the same ilk.
And it's more like the way we handle CPPFLAGS in libpq-using Makefiles.

In passing, just for consistency, make pgbench include PTHREAD_LIBS the
same way everyplace else does, ie just after LIBS rather than in some
random place in the command line.  This might have practical effect if
there are -L switches in that macro on some platform.

It looks to me like the MSVC build scripts are not affected by this
error, but someone more familiar with them than I might want to double
check.

Back-patch to 9.6 where libpgfeutils was introduced.  In 9.6, the hazard
this error creates is that a reinstallation might link to the prior
installation's copy of libpgfeutils.a and thereby fail to absorb a
minor-version bug fix.

Discussion: https://postgr.es/m/20170714125106.9231.13772@wrigleys.postgresql.org
2017-07-14 12:26:53 -04:00
Heikki Linnakangas 8046465c2e Fix pg_basebackup output to stdout on Windows.
When writing a backup to stdout with pg_basebackup on Windows, put stdout
to binary mode. Any CR bytes in the output will otherwise be output
incorrectly as CR+LF.

In the passing, standardize on using "_setmode" instead of "setmode", for
the sake of consistency. They both do the same thing, but according to
MSDN documentation, setmode is deprecated.

Fixes bug #14634, reported by Henry Boehlert. Patch by Haribabu Kommi.
Backpatch to all supported versions.

Discussion: https://www.postgresql.org/message-id/20170428082818.24366.13134@wrigleys.postgresql.org
2017-07-14 16:02:53 +03:00
Alvaro Herrera 6c774caf0e Translation updates
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: c5a8de3653bb1af6b0eb41cc6bf090c5522df52b
2017-07-10 11:53:55 -04:00
Magnus Hagander 4808d69955 Fix out of date comment
Author: Masahiko Sawada <sawada.mshk@gmail.com>
2017-07-07 15:08:55 +03:00
Peter Eisentraut 4260c05c6d Update code comments for pg_xlog -> pg_wal
Author: Michael Paquier <michael.paquier@gmail.com>
2017-06-30 14:40:50 -04:00
Tom Lane 382ceffdf7 Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.

By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis.  However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent.  That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.

This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.

This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.

Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 15:35:54 -04:00
Tom Lane c7b8998ebb Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.

Commit e3860ffa4d wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code.  The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there.  BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs.  So the
net result is that in about half the cases, such comments are placed
one tab stop left of before.  This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.

Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.

This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.

Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 15:19:25 -04:00
Peter Eisentraut 4e88fe8f8f Add missing serial comma 2017-06-14 14:43:54 -04:00
Peter Eisentraut 7ff9812f9a Put new command-line options in alphabetical order 2017-06-08 12:12:31 -04:00
Robert Haas 79c4fa0f62 Fix some cases of "the the" split across two lines.
Kevin Grittner observed that 2186b608b3
introduced a new occurence of this by copying existing text, and I
found a few more cases using grep.

Discussion: http://postgr.es/m/CADAecHWfG-K+YvocHCkrXV-ycm+eUOaaUVfYZNOnwf0pSmuQCw@mail.gmail.com
2017-06-06 12:24:44 -04:00
Magnus Hagander 2712da8b64 Generate pg_basebackup temporary slot name using backend pid
Using the client pid can easily be non-unique when used on different
hosts. Using the backend pid should be guaranteed unique, since the
temporary slot gets removed when the client disconnects so it will be
gone even if the pid is renewed.

Reported by Ludovic Vaugeois-Pepin
2017-05-31 21:00:37 +02:00
Bruce Momjian ce55481032 Post-PG 10 beta1 pgperltidy run 2017-05-17 19:01:23 -04:00
Bruce Momjian a6fd7b7a5f Post-PG 10 beta1 pgindent run
perltidy run not included.
2017-05-17 16:31:56 -04:00
Tom Lane 05b5feb60e Revert changes to pg_basebackup and pg_waldump usage() code.
Partially revert commit c079673dcb.
There were complaints that splitting switch descriptions would
complicate translation efforts.  There are probably ways to resolve
the formatting problem without doing that, but undo it while we're
discussing.
2017-05-17 13:04:03 -04:00
Tom Lane c079673dcb Preventive maintenance in advance of pgindent run.
Reformat various places in which pgindent will make a mess, and
fix a few small violations of coding style that I happened to notice
while perusing the diffs from a pgindent dry run.

There is one actual bug fix here: the need-to-enlarge-the-buffer code
path in icu_convert_case was obviously broken.  Perhaps it's unreachable
in our usage?  Or maybe this is just sadly undertested.
2017-05-16 20:36:35 -04:00
Peter Eisentraut 82d24bab75 Translation updates
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 398beeef4921df0956f917becd7b5669d2a8a5c4
2017-05-15 12:19:54 -04:00
Magnus Hagander b1c45afb01 Fix typo in comment
Michael Paquier
2017-05-15 11:08:02 +02:00
Peter Eisentraut d496a65790 Standardize "WAL location" terminology
Other previously used terms were "WAL position" or "log position".
2017-05-12 13:51:27 -04:00
Peter Eisentraut c1a7f64b4a Replace "transaction log" with "write-ahead log"
This makes documentation and error messages match the renaming of "xlog"
to "wal" in APIs and file naming.
2017-05-12 11:52:43 -04:00
Tom Lane d10c626de4 Rename WAL-related functions and views to use "lsn" not "location".
Per discussion, "location" is a rather vague term that could refer to
multiple concepts.  "LSN" is an unambiguous term for WAL locations and
should be preferred.  Some function names, view column names, and function
output argument names used "lsn" already, but others used "location",
as well as yet other terms such as "wal_position".  Since we've already
renamed a lot of things in this area from "xlog" to "wal" for v10,
we may as well incur a bit more compatibility pain and make these names
all consistent.

David Rowley, minor additional docs hacking by me

Discussion: https://postgr.es/m/CAKJS1f8O0njDKe8ePFQ-LK5-EjwThsDws6ohJ-+c6nWK+oUxtg@mail.gmail.com
2017-05-11 11:49:59 -04:00
Tom Lane 7834d20b57 Avoid slow shutdown of pg_basebackup.
pg_basebackup's child process did not pay any attention to the pipe
from its parent while waiting for input from the source server.
If no server data was arriving, it would only wake up and check the
pipe every standby_message_timeout or so.  This creates a problem
since the parent process might determine and send the desired stop
position only after the server has reached end-of-WAL and stopped
sending data.  In the src/test/recovery regression tests, the timing
is repeatably such that it takes nearly 10 seconds for the child
process to realize that it should shut down.  It's not clear how
often that would happen in real-world cases, but it sure seems like
a bug --- and if the user turns off standby_message_timeout or sets
it very large, the delay could be a lot worse.

To fix, expand the StreamCtl API to allow the pipe input FD to be
passed down to the low-level wait routine, and watch both sockets
when sleeping.

(Note: AFAICS this issue doesn't affect the Windows port, since
it doesn't rely on a pipe to transfer the stop position to the
child thread.)

Discussion: https://postgr.es/m/6456.1493263884@sss.pgh.pa.us
2017-04-27 18:27:02 -04:00
Peter Eisentraut 6275f5d28a Fix new warnings from GCC 7
This addresses the new warning types -Wformat-truncation
-Wformat-overflow that are part of -Wall, via -Wformat, in GCC 7.
2017-04-17 13:59:46 -04:00
Magnus Hagander b935eb7da3 Fix reversed check of return value from sync
While at it also update the comments in walmethods.h to make it less
likely for mistakes like this to appear in the future (thanks to Tom for
improvements to the comments).

And finally, in passing change the return type of walmethod.getlasterror
to being const, also per suggestion from Tom.
2017-04-12 13:46:38 +02:00
Magnus Hagander 7220c7b3e5 Write "waiting for checkpoint" on regular progress row
When reporting progress, make the "waiting for checkpoint" test be
overwritten by the file-based progress once it's completed. This is more
consistent with how we report the rest of the progress.

Suggested by Jeff Janes
2017-04-01 17:04:14 +02:00
Peter Eisentraut facde2a98f Clean up Perl code according to perlcritic
Fix all perlcritic warnings of severity level 5, except in
src/backend/utils/Gen_dummy_probes.pl, which is automatically generated.

Reviewed-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
2017-03-27 08:18:22 -04:00
Magnus Hagander 839da3c7d9 Update obsolete comment
Commit ff27db5d changed this function to no longer return the start
position, but forgot the comment.
2017-03-19 17:07:27 +01:00
Peter Eisentraut eb4da3e380 Add option to control snapshot export to CREATE_REPLICATION_SLOT
We used to export snapshots unconditionally in CREATE_REPLICATION_SLOT
in the replication protocol, but several upcoming patches want more
control over what happens.

Suppress snapshot export in pg_recvlogical, which neither needs nor can
use the exported snapshot.  Since snapshot exporting can fail this
improves reliability.

This also paves the way for allowing the creation of replication slots
on standbys, which cannot export snapshots because they cannot allocate
new XIDs.

Author: Petr Jelinek <petr.jelinek@2ndquadrant.com>
2017-03-14 17:34:22 -04:00
Peter Eisentraut a47b38c9ee Spelling fixes
From: Josh Soref <jsoref@gmail.com>
2017-03-14 12:58:39 -04:00
Peter Eisentraut be37c2120a Enable replication connections by default in pg_hba.conf
initdb now initializes a pg_hba.conf that allows replication connections
from the local host, same as it does for regular connections.  The
connecting user still needs to have the REPLICATION attribute or be a
superuser.

The intent is to allow pg_basebackup from the local host to succeed
without requiring additional configuration.

Michael Paquier <michael.paquier@gmail.com> and me
2017-03-09 08:39:44 -05:00
Robert Haas 19dc233c32 Add pg_current_logfile() function.
The syslogger will write out the current stderr and csvlog names, if
it's running and there are any, to a new file in the data directory
called "current_logfiles".  We take care to remove this file when it
might no longer be valid (but not at shutdown).  The function
pg_current_logfile() can be used to read the entries in the file.

Gilles Darold, reviewed and modified by Karl O.  Pinc, Michael
Paquier, and me.  Further review by Álvaro Herrera and Christoph Berg.
2017-03-03 11:43:11 +05:30
Peter Eisentraut 788af6f854 Move atooid() definition to a central place 2017-03-01 11:55:28 -05:00
Magnus Hagander 1513dbea7f Add missing progname prefix to some messages
Author: Michael Banck
2017-02-26 21:32:00 +01:00
Magnus Hagander 51e26c9c3d Clarify the role of checkpoint at the begininng of base backups
Output a message about checkpoint starting in verbose mode of
pg_basebackup, and make the documentation state more clearly that this
happens.

Author: Michael Banck
2017-02-26 21:31:54 +01:00
Tom Lane 9e3755ecb2 Remove useless duplicate inclusions of system header files.
c.h #includes a number of core libc header files, such as <stdio.h>.
There's no point in re-including these after having read postgres.h,
postgres_fe.h, or c.h; so remove code that did so.

While at it, also fix some places that were ignoring our standard pattern
of "include postgres[_fe].h, then system header files, then other Postgres
header files".  While there's not any great magic in doing it that way
rather than system headers last, it's silly to have just a few files
deviating from the general pattern.  (But I didn't attempt to enforce this
globally, only in files I was touching anyway.)

I'd be the first to say that this is mostly compulsive neatnik-ism,
but over time it might save enough compile cycles to be useful.
2017-02-25 16:12:55 -05:00
Tom Lane c29aff959d Consistently declare timestamp variables as TimestampTz.
Twiddle the replication-related code so that its timestamp variables
are declared TimestampTz, rather than the uninformative "int64" that
was previously used for meant-to-be-always-integer timestamps.
This resolves the int64-vs-TimestampTz declaration inconsistencies
introduced by commit 7c030783a, though in the opposite direction to
what was originally suggested.

This required including datatype/timestamp.h in a couple more places
than before.  I decided it would be a good idea to slim down that
header by not having it pull in <float.h> etc, as those headers are
no longer at all relevant to its purpose.  Unsurprisingly, a small number
of .c files turn out to have been depending on those inclusions, so add
them back in the .c files as needed.

Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us
Discussion: https://postgr.es/m/27694.1487456324@sss.pgh.pa.us
2017-02-23 15:57:08 -05:00
Tom Lane b9d092c962 Remove now-dead code for !HAVE_INT64_TIMESTAMP.
This is a basically mechanical removal of #ifdef HAVE_INT64_TIMESTAMP
tests and the negative-case controlled code.

Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us
2017-02-23 14:04:43 -05:00
Magnus Hagander 1a16af8b35 Fix help message for pg_basebackup -R
The recovery.conf file that's generated is specifically for replication,
and not needed (or wanted) for regular backup restore, so indicate that
in the message.
2017-02-18 13:45:52 +01:00
Fujii Masao 0dfa89ba29 Replace reference to "xlog-method" with "wal-method" in error message.
Commit 62e8b38 renamed "--xlog-method" option for pg_basebackup to
"--wal-method", but forgot to update the error message mentioning that option.
2017-02-15 01:26:44 +09:00
Robert Haas 62e8b38751 Rename command line options for ongoing xlog -> wal conversion.
initdb and pg_basebackup now have a --waldir option rather --xlogdir,
and pg_basebackup now has --wal-method rather than --xlog-method.
2017-02-09 16:42:51 -05:00
Robert Haas 85c11324ca Rename user-facing tools with "xlog" in the name to say "wal".
This means pg_receivexlog because pg_receivewal, pg_resetxlog
becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-02-09 16:23:46 -05:00
Robert Haas 806091c96f Remove all references to "xlog" from SQL-callable functions in pg_proc.
Commit f82ec32ac3 renamed the pg_xlog
directory to pg_wal.  To make things consistent, and because "xlog" is
terrible terminology for either "transaction log" or "write-ahead log"
rename all SQL-callable functions that contain "xlog" in the name to
instead contain "wal".  (Note that this may pose an upgrade hazard for
some users.)

Similarly, rename the xlog_position argument of the functions that
create slots to be called wal_position.

Discussion: https://www.postgresql.org/message-id/CA+Tgmob=YmA=H3DbW1YuOXnFVgBheRmyDkWcD9M8f=5bGWYEoQ@mail.gmail.com
2017-02-09 15:10:09 -05:00
Magnus Hagander cada1af31d Add compression support to pg_receivexlog
Author: Michael Paquier, review and small changes by me
2017-01-17 12:10:26 +01:00
Magnus Hagander fcf708623e Fix incorrect comparison due to bad merge
Noted by Fujii Masao
2017-01-16 18:20:57 +01:00
Magnus Hagander e7b020f786 Make pg_basebackup use temporary replication slots
Temporary replication slots will be used by default when wal streaming
is used and no slot name is specified with -S. If a slot name is
specified, then a permanent slot with that name is used. If --no-slot is
specified, then no permanent or temporary slot will be used.

Temporary slots are only used on 10.0 and newer, of course.
2017-01-16 13:56:43 +01:00
Magnus Hagander 534b6f3ef2 Use an enum instead of two bools to indicate wal inclusion in base backups
This makes the code easier to read as it becomes more explicit what the
different allowed combinations really are.

Suggested by Michael Paquier
2017-01-09 16:03:47 +01:00
Simon Riggs 3e353a7bc2 Add new TAP tests for pg_recvlogical
Craig Ringer, reviewed by Euler Taveira and Naoki Okano
2017-01-04 19:06:45 +00:00
Simon Riggs 7c030783a5 Add pg_recvlogical —-endpos=LSN
Allow pg_recvlogical to specify an ending LSN, complementing
the existing -—startpos=LSN option.

Craig Ringer, reviewed by Euler Taveira and Naoki Okano
2017-01-04 19:02:07 +00:00
Magnus Hagander 9a4d51077c Make wal streaming the default mode for pg_basebackup
Since streaming is now supported for all output formats, make this the
default as this is what most people want.

To get the old behavior, the parameter -X none can be specified to turn
it off.

This also removes the parameter -x for fetch, now requiring -X fetch to
be specified to use that.

Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs
2017-01-04 10:40:38 +01:00
Bruce Momjian 1d25779284 Update copyright via script for 2017 2017-01-03 13:48:53 -05:00
Magnus Hagander 3ea56fffd6 Don't rename .partial files in pg_receivexlog if an error occured
In 56c7d8d the behavior to keep .partial segments around
(considered corrupt) in case an connection failure occurs was
accidentally removed. This would lead to an incomplete segment
being considered complete.

Author: Michael Paquier
2016-12-27 10:37:11 +01:00
Fujii Masao ecbdc4c555 Forbid invalid combination of options in pg_basebackup.
Commit 56c7d8d455 allowed pg_basebackup
to stream WAL in tar mode. But there is the restriction that WAL
streaming in tar mode works only when the value - (dash) is not
specified as output directory. This means that the combination of
three options "-D -", "-F t" and "-X stream" is invalid. However,
previously, even when those options were specified at the same time,
pg_basebackup background process unexpectedly started streaming WAL.
And then it exited with an error.

This commit changes pg_basebackup so that it errors out on such
invalid combination of options at the beginning.

Reviewed by Magnus Hagander, and patch by me.
2016-12-21 20:27:37 +09:00
Magnus Hagander 8cb545bfd4 Add missing newline in message 2016-12-15 16:45:31 +01:00
Noah Misch 650b967076 Change qr/foo$/m to qr/foo\n/m, for Perl 5.8.8.
In each case, absence of a trailing newline would itself constitute a
PostgreSQL bug.  Therefore, this slightly enhances the changed tests.
This works around a bug that last appeared in Perl 5.8.8, fixing
src/test/modules/test_pg_dump when run against that version.  Commit
e7293e3271 worked around the bug, but the
subsequent addition of test_pg_dump introduced affected code.  As that
commit had shown, slight increases in pattern complexity can suppress
the bug.  This commit edits qr/foo$/m patterns too complex to encounter
the bug today, for style consistency and robustness against unrelated
pattern changes.  Back-patch to 9.6, where test_pg_dump was introduced.

As of this writing, a fresh MSYS installation includes an affected Perl
5.8.8.  The Perl 5.8.8 in Red Hat Enterprise Linux 5.11 carries a patch
that renders it unaffected, but the Perl 5.8.5 of Red Hat Enterprise
Linux 4.4 is affected.
2016-11-07 20:27:30 -05:00
Magnus Hagander a775406ec4 Fix memory leak in tar file padding
Spotted by Coverity, patch by Michael Paquier
2016-10-30 14:10:39 +01:00
Magnus Hagander 78d109150b Free walmethods before exiting
Not strictly necessary since we quite after, but could become important
in the future if we do restarts etc.

Michael Paquier with nitpicking from me
2016-10-25 19:00:12 +02:00
Magnus Hagander 8c46f0c9ce Don't fsync() files when --no-sync is specified
Michael Paquier
2016-10-25 19:00:01 +02:00
Magnus Hagander 2dde01ccbf Use ssize_t where signed results can happen
Noted by Alexander Korotkov
2016-10-24 20:10:18 +02:00
Magnus Hagander eade082b12 Rename walmethod fsync method to sync
Using the name fsync clashed with the #define we have on Windows that
redefines it to _commit. Naming it sync should remove that conflict.

Per all the Windows buildfarm members
2016-10-23 18:04:34 +02:00
Magnus Hagander a5c17c1dce Fix obviously too quickly applied fix to zlib issue 2016-10-23 16:07:31 +02:00
Magnus Hagander 9ae6713cdf Fix walmethods.c build without libz
Per numerous buildfarm manuals
2016-10-23 16:00:42 +02:00
Magnus Hagander d97a59a4c5 Remove extra comma at end of enum list
C99-specific feature, and wasn't intentional in the first place.

Per buildfarm member mylodon
2016-10-23 15:57:25 +02:00
Magnus Hagander 56c7d8d455 Allow pg_basebackup to stream transaction log in tar mode
This will write the received transaction log into a file called
pg_wal.tar(.gz) next to the other tarfiles instead of writing it to
base.tar. When using fetch mode, the transaction log is still written to
base.tar like before, and when used against a pre-10 server, the file
is named pg_xlog.tar.

To do this, implement a new concept of a "walmethod", which is
responsible for writing the WAL. Two implementations exist, one that
writes to a plain directory (which is also used by pg_receivexlog) and
one that writes to a tar file with optional compression.

Reviewed by Michael Paquier
2016-10-23 15:23:11 +02:00