Commit Graph

46462 Commits

Author SHA1 Message Date
Andres Freund
51ad5b9bd3 jit: Re-allow JIT compilation of execGrouping.c hashtable comparisons.
In the course of 5567d12ce0, 356687bd8 and 317ffdfeaa, I changed
BuildTupleHashTable[Ext]'s call to ExecBuildGroupingEqual to not pass
in the parent node, but NULL. Which in turn prevents the tuple
equality comparator from being JIT compiled.  While that fixes
bug #15486, it is not actually necessary after all of the above commits,
as we don't re-build the comparator when using the new
BuildTupleHashTableExt() interface (as the content of the hashtable
are reset, but the TupleHashTable itself is not).

Therefore re-allow jit compilation for callers that use
BuildTupleHashTableExt with a separate context for "metadata" and
content.

As in the previous commit, there's ongoing work to make this easier to
test to prevent such regressions in the future, but that
infrastructure is not going to be backpatchable.

The performance impact of not JIT compiling hashtable equality
comparators can be substantial e.g. for aggregation queries that
aggregate a lot of input rows to few output rows (when there are a lot
of output groups, there will be fewer comparisons).

Author: Andres Freund
Discussion: https://postgr.es/m/20190927072053.njf6prdl3vb7y7qb@alap3.anarazel.de
Backpatch: 11, just as 5567d12ce0
2019-09-29 16:27:08 -07:00
Andrew Dunstan
51e3005a07 Allow SSL TAP tests to run on Windows
Windows does not enforce key file permissions checks in libpq, and psql
can produce CRLF line endings on Windows.

Backpatch to Release 12 (CRLF) and Release 11 (permissions check)
2019-09-29 17:50:44 -04:00
Tom Lane
0baa55655e Improve stability of partition_prune regression test.
This test already knew that, to get stable test output, it had to hide
"loops" counts in EXPLAIN ANALYZE results.  But that's not nearly enough:
if we get a smaller number of workers than we planned for, then the
"Workers Launched" number will change, and so will all the rows and loops
counts up to the Gather node.  This has resulted in repeated failures in
the buildfarm, so adjust the test to filter out all these counts.

(Really, we wouldn't bother with EXPLAIN ANALYZE at all here, except
that currently the only way to verify that executor-time pruning has
happened is to look for '(never executed)' annotations.  Those are
stable and needn't be filtered out.)

Back-patch to v11 where the test was introduced.

Discussion: https://postgr.es/m/11952.1569536725@sss.pgh.pa.us
2019-09-28 13:33:34 -04:00
Tom Lane
240c38a488 Doc: clean up markup for jsonb_set and related functions.
The markup for optional parameters was neither correct nor consistent.
In passing, fix a spelling mistake.

Per report from Alex Macy.  Some of these mistakes are old, so
back-patch as appropriate.

Discussion: https://postgr.es/m/156953522258.1204.12736099368284950578@wrigleys.postgresql.org
2019-09-27 11:01:36 -04:00
Amit Kapila
84ced048d1 Fix oversight in commit 4429f6a9e3.
The test name and the following test cases suggest the index created
should be hash index, but it forgot to add 'using hash' in the test case.
This in itself won't improve code coverage as there were some other tests
which were covering the corresponding code.  However, it is better if the
added tests serve their actual purpose.

Reported-by: Paul A Jungwirth
Author: Paul A Jungwirth
Reviewed-by: Mahendra Singh
Backpatch-through: 9.4
Discussion: https://postgr.es/m/CA+renyV=Us-5XfMC25bNp-uWSj39XgHHmGE9Rh2cQKMegSj52g@mail.gmail.com
2019-09-27 08:11:46 +05:30
Michael Paquier
d01d4f2374 Fix failure with lock mode used for custom relation options
In-core relation options can use a custom lock mode since 47167b7, that
has lowered the lock available for some autovacuum parameters.  However
it forgot to consider custom relation options.  This causes failures
with ALTER TABLE SET when changing a custom relation option, as its lock
is not defined.  The existing APIs to define a custom reloption does not
allow to define a custom lock mode, so enforce its initialization to
AccessExclusiveMode which should be safe enough in all cases.  An
upcoming patch will extend the existing APIs to allow a custom lock mode
to be defined.

The problem can be reproduced with bloom indexes, so add a test there.

Reported-by: Nikolay Sharplov
Analyzed-by: Thomas Munro, Michael Paquier
Author: Michael Paquier
Reviewed-by: Kuntal Ghosh
Discussion: https://postgr.es/m/20190920013831.GD1844@paquier.xyz
Backpatch-through: 9.6
2019-09-25 10:08:30 +09:00
Tom Lane
7cd99d3947 Doc: clarify handling of duplicate elements in array containment tests.
The array <@ and @> operators do not worry about duplicates: if every
member of array X matches some element of array Y, then X is contained
in Y, even if several members of X get matched to the same Y member.
This was not explicitly stated in the docs though, so improve matters.

Discussion: https://postgr.es/m/156614120484.1310.310161642239149585@wrigleys.postgresql.org
2019-09-23 12:37:04 -04:00
Tom Lane
8873c5b201 Doc: in v11 release notes, remove item about citext_pattern_ops.
The entry for commit f24649976 claimed that citext_pattern_ops could be
used for LIKE index searches, but in fact it cannot.  That patch only
created the opclass and some related operators, without doing anything
to teach the planner about how to use it.  The opclass is basically
useless in this unfinished state, which is why nothing was added to the
main documentation about it, and really we shouldn't have said anything
in the release notes either.  So remove the entry.

Discussion: https://postgr.es/m/CAKqncch4eBt2c8ddNVxzcVh3fFhdk54QoDMzpKgpqor4gA-wcQ@mail.gmail.com
2019-09-23 10:58:19 -04:00
Tom Lane
7e7abed056 Fix failure to zero-pad the result of bitshiftright().
If the bitstring length is not a multiple of 8, we'd shift the
rightmost bits into the pad space, which must be zeroes --- bit_cmp,
for one, depends on that.  This'd lead to the result failing to
compare equal to what it should compare equal to, as reported in
bug #16013 from Daryl Waycott.

This is, if memory serves, not the first such bug in the bitstring
functions.  In hopes of making it the last one, do a bit more work
than minimally necessary to fix the bug:

* Add assertion checks to bit_out() and varbit_out() to complain if
they are given incorrectly-padded input.  This will improve the
odds that manual testing of any new patch finds problems.

* Encapsulate the padding-related logic in macros to make it
easier to use.

Also, remove unnecessary padding logic from bit_or() and bitxor().
Somebody had already noted that we need not re-pad the result of
bit_and() since the inputs are required to be the same length,
but failed to extrapolate that to the other two.

Also, move a comment block that once was near the head of varbit.c
(but people kept putting other stuff in front of it), to put it in
the header block.

Note for the release notes: if anyone has inconsistent data as a
result of saving the output of bitshiftright() in a table, it's
possible to fix it with something like
UPDATE mytab SET bitcol = ~(~bitcol) WHERE bitcol != ~(~bitcol);

This has been broken since day one, so back-patch to all supported
branches.

Discussion: https://postgr.es/m/16013-c2765b6996aacae9@postgresql.org
2019-09-22 17:46:00 -04:00
Tom Lane
24f64fba0a Update time zone data files to tzdata release 2019c.
DST law changes in Fiji and Norfolk Island.  Historical corrections
for Alberta, Austria, Belgium, British Columbia, Cambodia, Hong Kong,
Indiana (Perry County), Kaliningrad, Kentucky, Michigan, Norfolk
Island, South Korea, and Turkey.
2019-09-20 19:54:00 -04:00
Amit Kapila
2a98eefc1a Fix typo in commit 82fa3ff867.
Reported-By: Kuntal Ghosh (off-list)
Backpatch-through: 9.4, like 82fa3ff867
2019-09-20 07:55:47 +05:30
Alexander Korotkov
984b9ba1d1 Fix oversight in backpatch of 6cae9d2c10
During backpatch of 6cae9d2c10 Float8GetDatum() was accidentally removed.  This
commit turns it back.

Reported-by: Erik Rijkers
Discussion: https://postgr.es/m/6d51305e1159241cabee132f7efc7eff%40xs4all.nl
Author: Tom Lane
Backpatch-through: from 11 to 9.5
2019-09-19 23:36:01 +03:00
Alexander Korotkov
d6a90aac56 Improve handling of NULLs in KNN-GiST and KNN-SP-GiST
This commit improves subject in two ways:

 * It removes ugliness of 02f90879e7, which stores distance values and null
   flags in two separate arrays after GISTSearchItem struct.  Instead we pack
   both distance value and null flag in IndexOrderByDistance struct.  Alignment
   overhead should be negligible, because we typically deal with at most few
   "col op const" expressions in ORDER BY clause.
 * It fixes handling of "col op NULL" expression in KNN-SP-GiST.  Now, these
   expression are not passed to support functions, which can't deal with them.
   Instead, NULL result is implicitly assumed.  It future we may decide to
   teach support functions to deal with NULL arguments, but current solution is
   bugfix suitable for backpatch.

Reported-by: Nikita Glukhov
Discussion: https://postgr.es/m/826f57ee-afc7-8977-c44c-6111d18b02ec%40postgrespro.ru
Author: Nikita Glukhov
Reviewed-by: Alexander Korotkov
Backpatch-through: 9.4
2019-09-19 21:49:32 +03:00
Michael Paquier
3153328fa9 Doc: Fix incorrect mention to connection_object in CONNECT command of ECPG
This fixes an inconsistency with this parameter name not listed in the
command synopsis, and connection_name is the parameter name more
commonly used in the docs for ECPG commands.

Reported-by: Yusuke Egashita
Discussion: https://postgr.es/m/156870956796.1259.11456186889345212399@wrigleys.postgresql.org
Backpatch-through: 9.4
2019-09-19 13:19:04 +09:00
Amit Kapila
1831c6b184 Doc: document autovacuum interruption.
It's important users be able to know (without looking at the source code)
that running DDL or DDL-like commands can interrupt autovacuum which can
lead to a lot of dead tuples and hence slower database operations.

Reported-by: James Coleman
Author: James Coleman
Reviewed-by: Amit Kapila
Backpatch-through: 9.4
Discussion: https://postgr.es/m/CAAaqYe-XYyNwML1=f=gnd0qWg46PnvD=BDrCZ5-L94B887XVxQ@mail.gmail.com
2019-09-19 09:00:13 +05:30
Alvaro Herrera
79b562448c pg_upgrade/test.sh: Quote sed(1) argument
Lack of quotes results in failure to run the test under older Solaris.

Author: Marina Polyakova, Victor Wagner
Discussion: https://postgr.es/m/feba89f89e8925b3535cb7d72b9e05e1@postgrespro.ru
2019-09-18 11:26:45 -03:00
Etsuro Fujita
5b7d305e8d Doc: Update FDW documentation about direct foreign table modification.
1. Commit 7086be6e3 should have documented the limitation that the direct
   modification is disabled when WCO constraints are present, but didn't,
   which is definitely my fault.  Update the documentation (Postgres 9.6
   onwards).

2. Commit fc22b6623 should have documented the limitation that the direct
   modification is disabled when generated columns are defined, but
   didn't.  Update the documentation (Postgres 12 onwards).

Author: Etsuro Fujita
Discussion: https://postgr.es/m/CAPmGK14AYCPunLb6TRz1CQsW5Le01Z2ox8LSOKH0P-cOVDcQRA%40mail.gmail.com
2019-09-18 18:50:02 +09:00
Noah Misch
40ad420251 Replace xlc __fetch_and_add() with inline asm.
PostgreSQL has been unusable when built with xlc 13 and newer, which are
incompatible with our use of __fetch_and_add().  Back-patch to 9.5,
which introduced pg_atomic_fetch_add_u32().

Reviewed by Tom Lane.

Discussion: https://postgr.es/m/20190831071157.GA3251746@rfd.leadboat.com
2019-09-13 19:34:18 -07:00
Noah Misch
585fc561f8 Test pg_atomic_fetch_add_ with variable addend and 16-bit edge cases.
Back-patch to 9.5, which introduced these functions.

Reviewed by Tom Lane.

Discussion: https://postgr.es/m/20190831071157.GA3251746@rfd.leadboat.com
2019-09-13 19:33:46 -07:00
Alvaro Herrera
41f3d26269 logical decoding: process ASSIGNMENT during snapshot build
Most WAL records are ignored in early SnapBuild snapshot build phases.
But it's critical to process some of them, so that later messages have
the correct transaction state after the snapshot is completely built; in
particular, XLOG_XACT_ASSIGNMENT messages are critical in order for
sub-transactions to be correctly assigned to their parent transactions,
or at least one assert misbehaves, as reported by Ildar Musin.

Diagnosed-by: Masahiko Sawada
Author: Masahiko Sawada
Discussion: https://postgr.es/m/CAONYFtOv+Er1p3WAuwUsy1zsCFrSYvpHLhapC_fMD-zNaRWxYg@mail.gmail.com
2019-09-13 16:36:28 -03:00
Alvaro Herrera
82d0ae919f Fix under-parenthesized macro definitions
Lack of parens in the definitions could cause a statement using these
macros to have unexpected semantics.  In current code no bug is
apparent, but best to fix the definitions to avoid problems down the
line.

Reported-by: Tom Lane
Discussion: https://postgr.es/m/19795.1568400476@sss.pgh.pa.us
2019-09-13 16:26:55 -03:00
Peter Geoghegan
e87b00b5f6 Fix nbtree page split rmgr desc routine.
Include newitemoff in rmgr desc output for nbtree page split records.
In passing, correct an obsolete comment that claimed that newitemoff is
only logged for _L variant nbtree page split WAL records.

Both issues were oversights in commit 2c03216d83, which revamped the
WAL format.

Author: Peter Geoghegan
Backpatch: 9.5-, where the WAL format was revamped.
2019-09-12 15:45:05 -07:00
Tom Lane
64d926f2b0 Fix usage of whole-row variables in WCO and RLS policy expressions.
Since WITH CHECK OPTION was introduced, ExecInitModifyTable has
initialized WCO expressions with the wrong plan node as parent -- that is,
it passed its input subplan not the ModifyTable node itself.  Up to now
we thought this was harmless, but bug #16006 from Vinay Banakar shows it's
not: if the input node is a SubqueryScan then ExecInitWholeRowVar can get
confused into doing the wrong thing.  (The fact that ExecInitWholeRowVar
contains such logic is certainly a horrid kluge that doesn't deserve to
live, but figuring out another way to do that is a task for some other day.)

Andres had already noticed the wrong-parent mistake and fixed it in commit
148e632c0, but not being aware of any user-visible consequences, he quite
reasonably didn't back-patch.  This patch is simply a back-patch of
148e632c0, plus addition of a test case based on bug #16006.  I also added
the test case to v12/HEAD, even though the bug is already fixed there.

Back-patch to all supported branches.  9.4 lacks RLS policies so the
new test case doesn't work there, but I'm pretty sure a test could be
devised based on using a whole-row Var in a plain WITH CHECK OPTION
condition.  (I lack the cycles to do so myself, though.)

Andres Freund and Tom Lane

Discussion: https://postgr.es/m/16006-99290d2e4642cbd5@postgresql.org
Discussion: https://postgr.es/m/20181205225213.hiwa3kgoxeybqcqv@alap3.anarazel.de
2019-09-12 18:29:48 -04:00
Alvaro Herrera
728c49779b Improve documentation about our XML functionality
Add a section explaining how our XML features depart from current
versions of the SQL standard.  Update and clarify the descriptions
of some XML functions.

This is a backpatch for branches 10 and 11, taken from Tom's commit
12d46ac392 for 12, then edited to correctly describe behaviors that
are fixed in 12 but still broken in 10 and 11.

Chapman Flack, reviewed by Ryan Lambert

Discussion: https://postgr.es/m/5BD1284C.1010305@anastigmatix.net
Discussion: https://postgr.es/m/5C81F8C0.6090901@anastigmatix.net
Discussion: https://postgr.es/m/CAN-V+g-6JqUQEQZ55Q3toXEN6d5Ez5uvzL4VR+8KtvJKj31taw@mail.gmail.com
2019-09-12 17:31:31 -03:00
Amit Kapila
1fcb73fe72 Doc: Update PL/pgSQL sample function in plpgsql.sgml.
The example used to explain 'Looping Through Query Results' uses
pseudo-materialized views.  Replace it with a more up-to-date example
which does the same thing with actual materialized views, which have
been available since PostgreSQL 9.3.

In the passing, change '%' as format specifier instead of '%s' as is used
in other examples in plpgsql.sgml.

Reported-by: Ian Barwick
Author: Ian Barwick
Reviewed-by: Amit Kapila
Backpatch-through: 9.4
Discussion: https://postgr.es/m/9a70d393-7904-4918-c97c-649f6d114b6a@2ndquadrant.com
2019-09-12 13:37:43 +05:30
Michael Paquier
8da17f3f50 Expand properly list of TAP tests used for prove in vcregress.pl
Depending on the system used, t/*.pl may not be expanded into a list of
tests which can be consumed by prove when attempting to run TAP tests on
a given path.  Fix that by using glob() directly in the script, to make
sure that a complete list of tests is provided.  This has not proved to
be an issue with MSVC as the list was properly expanded, but it is on
Linux with perl's system().

This is extracted from a larger patch.

Author: Tom Lane
Discussion: https://postgr.es/m/6628.1567958876@sss.pgh.pa.us
Backpatch-through: 9.4
2019-09-11 11:07:29 +09:00
Tom Lane
b5784042ae Sync isolationtester's handling of notice/warning messages with HEAD.
Back-patch relevant parts of these commits:
30717637c Fix isolationtester race condition for notices sent before blocking
ebd499282 Don't drop NOTICE messages in isolation tests
a28e10e82 Indicate session name in isolationtester notices

This ensures that older versions of the isolationtester will handle
NOTICE/WARNING messages the same way as HEAD and v12 do.  While this
isn't fixing any critical problem right now, it seems like a prudent
change to prevent surprises (like we had yesterday...) with
back-patches of future isolation test changes.

Back-patch as far as 9.6.  Due to the significant changes we made in
isolationtester in 9.6, back-patching isolation tests further than
that is going to be risky anyway; besides, this patch doesn't apply
cleanly before that.

Discussion: https://postgr.es/m/E1i7IqC-0000Uc-5H@gemulon.postgresql.org
2019-09-10 12:45:32 -04:00
Tom Lane
8459f908c6 Be more careful about port selection in src/test/ldap/.
Don't just assume that the next port is free; it might not be, or
if we're really unlucky it might even be out of the TCP range.
Do it honestly with two get_free_port() calls instead.

This is surely a pretty low-probability problem, but I think it
explains a buildfarm failure seen today, so let's fix it.

Back-patch to v11 where this script was added.

Discussion: https://postgr.es/m/25124.1568052346@sss.pgh.pa.us
2019-09-09 14:21:52 -04:00
Andrew Dunstan
eec5144701 Prevent msys2 conversion of "cmd /c" switch to a file path
Modern versions of msys2 have changed the treatment of "cmd /c" so that
the runtime will try to convert the switch to a native file path. This
patch adds a setting to inhibit that behaviour.

Discussion: https://postgr.es/m/3227042f-cfcc-745a-57dd-fb8c471f8ddf@2ndQuadrant.com

Backpatch to all live branches.
2019-09-09 09:02:53 -04:00
Andrew Dunstan
b97a345d96 Always skip recovery SysV shared memory tests on Windows
These tests were disabled on git master in commit 8e5ce1c3f8. This does
the same thing on the back branches.

Discussion: https://postgr.es/m/a1c40fab-a38c-cb42-6879-125f834e837b@2ndQuadrant.com
2019-09-09 08:43:57 -04:00
Tom Lane
69f883fef1 Fix RelationIdGetRelation calls that weren't bothering with error checks.
Some of these are quite old, but that doesn't make them not bugs.
We'd rather report a failure via elog than SIGSEGV.

While at it, uniformly spell the error check as !RelationIsValid(rel)
rather than a bare rel == NULL test.  The machine code is the same
but it seems better to be consistent.

Coverity complained about this today, not sure why, because the
mistake is in fact old.
2019-09-08 17:00:53 -04:00
Alexander Korotkov
d807200b4a Fix handling of NULL distances in KNN-GiST
In order to implement NULL LAST semantic GiST previously assumed distance to
the NULL value to be Inf.  However, our distance functions can return Inf and
NaN for non-null values.  In such cases, NULL LAST semantic appears to be
broken.  This commit fixes that by introducing separate array of null flags for
distances.

Backpatch to all supported versions.

Discussion: https://postgr.es/m/CAPpHfdsNvNdA0DBS%2BwMpFrgwT6C3-q50sFVGLSiuWnV3FqOJuQ%40mail.gmail.com
Author: Alexander Korotkov
Backpatch-through: 9.4
2019-09-08 21:46:58 +03:00
Alexander Korotkov
749b04d1d8 Fix handling Inf and Nan values in GiST pairing heap comparator
Previously plain float comparison was used in GiST pairing heap.  Such
comparison doesn't provide proper ordering for value sets containing Inf and Nan
values.  This commit fixes that by usage of float8_cmp_internal().  Note, there
is remaining problem with NULL distances, which are represented as Inf in
pairing heap.  It would be fixes in subsequent commit.

Backpatch to all supported versions.

Reported-by: Andrey Borodin
Discussion: https://postgr.es/m/CAPpHfdsNvNdA0DBS%2BwMpFrgwT6C3-q50sFVGLSiuWnV3FqOJuQ%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Heikki Linnakangas
Backpatch-through: 9.4
2019-09-08 21:41:56 +03:00
Peter Eisentraut
7babff18dc doc: Fix awkward markup 2019-09-06 22:21:15 +02:00
Robert Haas
61c65cce40 When performing a base backup, check for read errors.
The old code didn't differentiate between a read error and a
concurrent truncation. fread reports both of these by returning 0;
you have to use feof() or ferror() to distinguish between them,
which this code did not do.

It might be a better idea to use read() rather than fread() here,
so that we can display a less-generic error message, but I'm not
sure that would qualify as a back-patchable bug fix, so just do
this much for now.

Jeevan Chalke, reviewed by Jeevan Ladhe and by me.

Discussion: http://postgr.es/m/CA+TgmobG4ywMzL5oQq2a8YKp8x2p3p1LOMMcGqpS7aekT9+ETA@mail.gmail.com
2019-09-06 08:53:13 -04:00
Michael Paquier
af28744288 Fix thinko when ending progress report for a backend
The logic ending progress reporting for a backend entry introduced by
b6fb647 causes callers of pgstat_progress_end_command() to do some extra
work when track_activities is enabled as the process fields are reset in
the backend entry even if no command were started for reporting.

This resets the fields only if a command is registered for progress
reporting, and only if track_activities is enabled.

Author: Masahiho Sawada
Discussion: https://postgr.es/m/CAD21AoCry_vJ0E-m5oxJXGL3pnos-xYGCzF95rK5Bbi3Uf-rpA@mail.gmail.com
Backpatch-through: 9.6
2019-09-04 15:46:49 +09:00
Michael Paquier
996c92b27d Delay fsyncs of pg_basebackup until the end of backup
Since the addition of fsync requests in bc34223 to make base backup data
consistent on disk once pg_basebackup finishes, each tablespace tar file
is individually flushed once completed, with an additional flush of the
parent directory when the base backup finishes.  While holding a
connection to the server, a fsync request taking a long time may cause a
failure of the base backup, which is annoying for any integration.  A
recent example of breakage can involve tcp_user_timeout, but
wal_sender_timeout can cause similar problems.

While reviewing the code, there was a second issue causing too many
fsync requests to be done for the same WAL data.  As recursive fsyncs
are done at the end of the backup for both the plain and tar formats
from the base target directory where everything is written, it is fine
to disable fsyncs when fetching or streaming WAL.

Reported-by: Ryohei Takahashi
Author: Michael Paquier
Reviewed-by: Ryohei Takahashi
Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com
Backpatch-through: 10
2019-09-04 13:24:06 +09:00
Michael Paquier
f967a1fda8 Fix memory leak with lower, upper and initcap with ICU-provided collations
The leak happens in str_tolower, str_toupper and str_initcap, which are
used in several places including their equivalent SQL-level functions,
and can only be triggered when using an ICU-provided collation when
converting the input string.

b615920 fixed a similar leak.  Backpatch down 10 where ICU collations
have been introduced.

Author: Konstantin Knizhnik
Discussion: https://postgr.es/m/94c0ad0a-cbc2-e4a3-7829-2bdeaf9146db@postgrespro.ru
Backpatch-through: 10
2019-09-03 12:31:08 +09:00
Tom Lane
5524ef5581 Handle corner cases correctly in psql's reconnection logic.
After an unexpected connection loss and successful reconnection,
psql neglected to resynchronize its internal state about the server,
such as server version.  Ordinarily we'd be reconnecting to the same
server and so this isn't really necessary, but there are scenarios
where we do need to update --- one example is where we have a list
of possible connection targets and they're not all alike.

Define "resynchronize" as including connection_warnings(), so that
this case acts the same as \connect.  This seems useful; for example,
if the server version did change, the user might wish to know that.
An attuned user might also notice that the new connection isn't
SSL-encrypted, for example, though this approach isn't especially
in-your-face about such changes.  Although this part is a behavioral
change, it only affects interactive sessions, so it should not break
any applications.

Also, in do_connect, make sure that we desynchronize correctly when
abandoning an old connection in non-interactive mode.

These problems evidently are the result of people patching only one
of the two places where psql deals with connection changes, so insert
some cross-referencing comments in hopes of forestalling future bugs
of the same ilk.

Lastly, in Windows builds, issue codepage mismatch warnings only at
startup, not during reconnections.  psql's codepage can't change
during a reconnect, so complaining about it again seems like useless
noise.

Peter Billen and Tom Lane.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/CAMTXbE8e6U=EBQfNSe01Ej17CBStGiudMAGSOPaw-ALxM-5jXg@mail.gmail.com
2019-09-02 14:02:46 -04:00
Tom Lane
232f645ae2 Doc: describe the "options" allowed in an ECPG connection target string.
These have been there a long time, but their format was never explained
in the docs.  Per complaint from Yusuke Egashira.

Discussion: https://postgr.es/m/848B1649C8A6274AA527C4472CA11EDD5FC70CBE@G01JPEXMBYT02
2019-08-31 14:05:32 -04:00
Etsuro Fujita
8c88d21ced Fix typo in regression test comment. 2019-08-29 18:45:02 +09:00
Heikki Linnakangas
1c99acc6eb Fix overflow check and comment in GIN posting list encoding.
The comment did not match what the code actually did for integers with
the 43rd bit set. You get an integer like that, if you have a posting
list with two adjacent TIDs that are more than 2^31 blocks apart.
According to the comment, we would store that in 6 bytes, with no
continuation bit on the 6th byte, but in reality, the code encodes it
using 7 bytes, with a continuation bit on the 6th byte as normal.

The decoding routine also handled these 7-byte integers correctly, except
for an overflow check that assumed that one integer needs at most 6 bytes.
Fix the overflow check, and fix the comment to match what the code
actually does. Also fix the comment that claimed that there are 17 unused
bits in the 64-bit representation of an item pointer. In reality, there
are 64-32-11=21.

Fitting any item pointer into max 6 bytes was an important property when
this was written, because in the old pre-9.4 format, item pointers were
stored as plain arrays, with 6 bytes for every item pointer. The maximum
of 6 bytes per integer in the new format guaranteed that we could convert
any page from the old format to the new format after upgrade, so that the
new format was never larger than the old format. But we hardly need to
worry about that anymore, and running into that problem during upgrade,
where an item pointer is expanded from 6 to 7 bytes such that the data
doesn't fit on a page anymore, is implausible in practice anyway.

Backpatch to all supported versions.

This also includes a little test module to test these large distances
between item pointers, without requiring a 16 TB table. It is not
backpatched, I'm including it more for the benefit of future development
of new posting list formats.

Discussion: https://www.postgresql.org/message-id/33bfc20a-5c86-f50c-f5a5-58e9925d05ff%40iki.fi
Reviewed-by: Masahiko Sawada, Alexander Korotkov
2019-08-28 12:57:10 +03:00
Thomas Munro
b9c4ccfefc Avoid catalog lookups in RelationAllowsEarlyPruning().
RelationAllowsEarlyPruning() performed a catalog scan, but is used
in two contexts where that was a bad idea:

1.  In heap_page_prune_opt(), which runs very frequently in some large
    scans.  This caused major performance problems in a field report
    that was easy to reproduce.

2.  In TestForOldSnapshot(), which runs while we hold a buffer content
    lock.  It's not clear if this was guaranteed to be free of buffer
    deadlock risk.

The check was introduced in commit 2cc41acd8 and defended against a
real problem: 9.6's hash indexes have no page LSN and so we can't
allow early pruning (ie the snapshot-too-old feature).  We can remove
the check from all later releases though: hash indexes are now logged,
and there is no way to create UNLOGGED indexes on regular logged
tables.

If a future release allows such a combination, it might need to put
a similar check in place, but it'll need some more thought.

Back-patch to 10.

Author: Thomas Munro
Reviewed-by: Tom Lane, who spotted the second problem
Discussion: https://postgr.es/m/CA%2BhUKGKT8oTkp5jw_U4p0S-7UG9zsvtw_M47Y285bER6a2gD%2Bg%40mail.gmail.com
Discussion: https://postgr.es/m/CAA4eK1%2BWy%2BN4eE5zPm765h68LrkWc3Biu_8rzzi%2BOYX4j%2BiHRw%40mail.gmail.com
2019-08-28 17:59:27 +12:00
Michael Paquier
f51006ea96 Disable timeouts when running pg_rewind with online source cluster
In this case, the transfer uses a libpq connection, which is subject to
the timeout parameters set at system level, and this can make the rewind
operation suddenly canceled which is not good for automation.  One
workaround to such issues would be to use PGOPTIONS to enforce the
wanted timeout parameters, but that's annoying, and for example pg_dump,
which can run potentially long-running queries disables all types of
timeouts.

lock_timeout and statement_timeout are the ones which can cause problems
now.  Note that pg_rewind does not use transactions, so disabling
idle_in_transaction_session_timeout is optional, but it feels safer to
do so for the future.

This is back-patched down to 9.5.  idle_in_transaction_session_timeout
is only present since 9.6.

Author: Alexander Kukushkin
Discussion: https://postgr.es/m/CAFh8B=krcVXksxiwVQh1SoY+ziJ-JC=6FcuoBL3yce_40Es5_g@mail.gmail.com
Backpatch-through: 9.5
2019-08-28 11:48:19 +09:00
Tom Lane
8635d73e2f Doc: improve documentation of pg_signal_backend default role.
Give it an explanatory para like the other default roles have.
Don't imply that it can send any signal whatever.

In passing, reorder the table entries and explanatory paras
for the default roles into some semblance of consistency.

Ian Barwick, tweaked a bit by me.

Discussion: https://postgr.es/m/89907e32-76f3-7282-a89c-ea19c722fe5d@2ndquadrant.com
2019-08-27 18:03:09 -04:00
Tom Lane
e352a005bc Doc: clarify behavior of standard aggregates for null inputs.
Section 4.2.7 says that unless otherwise specified, built-in
aggregates ignore rows in which any input is null.  This is
not true of the JSON aggregates, but it wasn't documented.
Fix that.

Of the other entries in table 9.55, some were explicit about
ignoring nulls, and some weren't; for consistency and
self-contained-ness, make them all say it explicitly.

Per bug #15884 from Tim Möhlmann.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/15884-c32d848f787fcae3@postgresql.org
2019-08-27 16:37:22 -04:00
Tom Lane
cf86803e8e Reject empty names and recursion in config-file include directives.
An empty file name or subdirectory name leads join_path_components() to
just produce the parent directory name, which leads to weird failures or
recursive inclusions.  Let's throw a specific error for that.  It takes
only slightly more code to detect all-blank names, so do so.

Also, detect direct recursion, ie a file calling itself.  As coded
this will also detect recursion via "include_dir '.'", which is
perhaps more likely than explicitly including the file itself.

Detecting indirect recursion would require API changes for guc-file.l
functions, which seems not worth it since extensions might call them.
The nesting depth limit will catch such cases eventually, just not
with such an on-point error message.

In passing, adjust the example usages in postgresql.conf.sample
to perhaps eliminate the problem at the source: there's no reason
for the examples to suggest that an empty value is valid.

Per a trouble report from Brent Bates.  Back-patch to 9.5; the
issue is old, but the code in 9.4 is enough different that the
patch doesn't apply easily, and it doesn't seem worth the trouble
to fix there.

Ian Barwick and Tom Lane

Discussion: https://postgr.es/m/8c8bcbca-3bd9-dc6e-8986-04a5abdef142@2ndquadrant.com
2019-08-27 14:44:26 -04:00
Michael Paquier
128e9b2cc9 Fix failure of --jobs with vacuumdb on Windows
FD_SETSIZE needs to be declared before winsock2.h, or it is possible to
run into buffer overflow issues when using --jobs.  This is similar to
pgbench's solution done in a23c641.

This has been introduced by 71d84ef, and older versions have been using
the default value of FD_SETSIZE, defined at 64.  While on it, add a
missing newline to the previously-added error message.

Per buildfarm member jacana, but this impacts all Windows animals
running the TAP tests.  I have reproduced the failure locally to check
the patch.

Author: Michael Paquier
Reviewed-by: Andrew Dunstan
Discussion: https://postgr.es/m/20190826054000.GE7005@paquier.xyz
Backpatch-through: 9.5
2019-08-27 09:11:43 +09:00
Andrew Dunstan
ad9b419f7c Adjust to latest Msys2 kernel release number
Previously 'uname -r' on Msys2 reported a kernele release starting with
2. The latest version starts with 3. In commit 1638623f we specifically
looked for one starting with 2. This is now changed to look for any
digit between 2 and 9.

backpatch to release 10.
2019-08-26 08:22:32 -04:00
Andrew Dunstan
3a1417c426 Treat MINGW and MSYS the same in pg_upgrade test script
On msys2, 'uname -s' reports a string starting MSYS instead on MINGW
as happens on msys1. Treat these both the same way. This reverts
608a710195 in favor of a more general solution.

Backpatch to all live branches.
2019-08-26 07:45:37 -04:00