Commit Graph

46255 Commits

Author SHA1 Message Date
Alvaro Herrera 3574c0ac05 Change pg_restore -f- to dump to stdout instead of to ./-
Starting with PostgreSQL 12, pg_restore refuses to run when neither -d
nor -f are specified (c.f. commit 413ccaa74d), and it also makes "-f -"
mean the old implicit behavior of dumping to stdout.  However, older
branches write to a file called ./- when invoked like that, making it
impossible to write pg_restore scripts that work across versions.  This
is a partial backpatch of the aforementioned commit to all older
supported branches, providing an upgrade path.

Discussion: https://postgr.es/m/20191006190839.GE18030@telsasoft.com
2019-11-05 01:23:39 -03:00
Michael Paquier 8f8ff09d49 Doc: Improve description around ALTER TABLE ATTACH PARTITION
This clarifies more how to use and how to take advantage of constraints
when attaching a new partition.

Author: Justin Pryzby
Reviewed-by: Amit Langote, Álvaro Herrera, Michael Paquier
Discussion: https://postgr.es/m/20191028001207.GB23808@telsasoft.com
Backpatch-through: 10
2019-11-05 10:18:01 +09:00
Tom Lane 078f5bc8e3 Stabilize pg_dump output order for similarly-named triggers and policies.
The code only compared two triggers' names and namespaces (the latter
being the owning table's schema).  This could result in falling back
to an OID-based sort of similarly-named triggers on different tables.
We prefer to avoid that, so add a comparison of the table names too.
(The sort order is thus table namespace, trigger name, table name,
which is a bit odd, but it doesn't seem worth contorting the code
to work around that.)

Likewise for policy objects, in 9.5 and up.

Complaint and fix by Benjie Gillam.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/CAMThMzEEt2mvBbPgCaZ1Ap1N-moGn=Edxmadddjq89WG4NpPtQ@mail.gmail.com
2019-11-04 16:25:05 -05:00
Peter Eisentraut 0782050bc2 Catch invalid typlens in a couple of places
Rearrange the logic in record_image_cmp() and record_image_eq() to
error out on unexpected typlens (either not supported there or
completely invalid due to corruption).  Barring corruption, this is
not possible today but it seems more future-proof and robust to fix
this.

Reported-by: Peter Geoghegan <pg@bowt.ie>
2019-11-04 09:20:33 +01:00
Tom Lane 88d03d73c2 Suppress warning from older compilers.
Commit 8af1624e3 introduced a warning about possibly returning
without a value, on compilers that don't realize that ereport(ERROR)
doesn't return.  Tweak the code to avoid that.

Per buildfarm.  Back-patch to 9.6, like the aforesaid commit.
2019-11-03 16:10:45 -05:00
Tom Lane 65cdf8bc1b Validate ispell dictionaries more carefully.
Using incorrect, or just mismatched, dictionary and affix files
could result in a crash, due to failure to cross-check offsets
obtained from the file.  Add necessary validation, as well as
some Asserts for future-proofing.

Per bug #16050 from Alexander Lakhin.  Back-patch to 9.6 where the
problem was introduced.

Arthur Zakirov, per initial investigation by Tomas Vondra

Discussion: https://postgr.es/m/16050-024ae722464ab604@postgresql.org
Discussion: https://postgr.es/m/20191013012610.2p2fp3zzpoav7jzf@development
2019-11-02 16:45:32 -04:00
Michael Paquier f10815c5bf Fix failure when creating cloned indexes for a partition
When using CREATE TABLE for a new partition, the partitioned indexes of
the parent are created automatically in a fashion similar to LIKE
INDEXES.  The new partition and its parent use a mapping for attribute
numbers for this operation, and while the mapping was correctly built,
its length was defined as the number of attributes of the newly-created
child, and not the parent.  If the parent includes dropped columns, this
could cause failures.

This is wrong since 8b08f7d which has introduced the concept of
partitioned indexes, so backpatch down to 11.

Reported-by: Wyatt Alt
Author: Michael Paquier
Reviewed-by: Amit Langote
Discussion: https://postgr.es/m/CAGem3qCcRmhbs4jYMkenYNfP2kEusDXvTfw-q+eOhM0zTceG-g@mail.gmail.com
Backpatch-through: 11
2019-11-02 14:17:12 +09:00
Michael Paquier 61f2383925 Fix race condition at backend exit when deleting element in syncrep queue
When a backend exits, it gets deleted from the syncrep queue if present.
The queue was checked without SyncRepLock taken in exclusive mode, so it
would have been possible for a backend to remove itself after a WAL
sender already did the job.  Fix this issue based on a suggestion from
Fujii Masao, by first checking the queue without the lock.  Then, if the
backend is present in the queue, take the lock and perform an additional
lookup check before doing the element deletion.

Author: Dongming Liu
Reviewed-by: Kyotaro Horiguchi, Fujii Masao, Michael Paquier
Discussion: https://postgr.es/m/a0806273-8bbb-43b3-bbe1-c45a58f6ae21.lingce.ldm@alibaba-inc.com
Backpatch-through: 9.4
2019-11-01 22:38:51 +09:00
Andres Freund 3b24cf732e pg_waldump: Fix --bkp-details to not issue spurious newlines for FPWs.
The additional newline seems to have accidentally been introduced in
2c03216d83, in 9.5. The newline is only issued when an FPW is
present for the block reference.

While there could be an argument that removing the newlines in the
back branches could cause a problem for somebody parsing the
pg_waldump output, the likelihood of that seems small enough. It seems
at least equally likely that the randomness of when newlines are
issued causes problems.

Author: Andres Freund
Discussion: https://postgr.es/m/20191029233341.4gnyau7e5v2lh5sc@alap3.anarazel.de
Backpatch: 9.5, like 2c03216d83.
2019-10-29 22:53:33 -07:00
Andres Freund af67aee691 pg_waldump: Fix small memory leak when rmgr->rm_identify returns NULL.
This got broken in 604f7956b9, shortly after rm_identify's
introduction.

Author: Andres Freund
Discussion: https://postgr.es/m/20191029233341.4gnyau7e5v2lh5sc@alap3.anarazel.de
Backpatch: 9.5, where rm_identify was introduced
2019-10-29 19:28:34 -07:00
Tom Lane 9f7a664fe3 Avoid failure when selecting a namespace node in XMLTABLE.
It appears that libxml2 doesn't bother to set the "children" field of
an XML_NAMESPACE_DECL node to null; that field just contains garbage.
In v10 and v11, this can result in a crash in XMLTABLE().  The rewrite
done in commit 251cf2e27 fixed this, somewhat accidentally, in v12.
We're not going to back-patch 251cf2e27, however.  The case apparently
doesn't have wide use, so rather than risk introducing other problems,
just add a safety check to throw an error.

Even though no bug manifests in v12/HEAD, add the relevant test case
there too, to prevent future regressions.

Chapman Flack (per private report)
2019-10-25 15:22:40 -04:00
Bruce Momjian 2930f163f6 docs: fix wording of REFRESH CONCURRENTLY manual page
Reported-by: Asim / apraveen@pivotal.io

Discussion: https://postgr.es/m/157076828181.26165.15231292023740913543@wrigleys.postgresql.org

Backpatch-through: 9.4
2019-10-23 20:29:02 -04:00
Michael Paquier 669feabfb7 Clean up properly error_context_stack in autovacuum worker on exception
Any callback set would have no meaning in the context of an exception.
As an autovacuum worker exits quickly in this context, this could be
only an issue within EmitErrorReport(), where the elog hook is for
example called.  That's unlikely to going to be a problem, but let's be
clean and consistent with other code paths handling exceptions.  This is
present since 2909419, which introduced autovacuum.

Author: Ashwin Agrawal
Reviewed-by: Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/CALfoeisM+_+dgmAdAOHAu0k-ZpEHHqSSG=GRf3pKJGm8OqWX0w@mail.gmail.com
Backpatch-through: 9.4
2019-10-23 10:25:50 +09:00
Tom Lane a05a04d0ed Deal with yet another issue related to "Norwegian (Bokmål)" locale.
It emerges that recent versions of Windows (at least 2016 Standard)
spell this locale name as "Norwegian Bokmål_Norway.1252", defeating
our mapping code that translates "Norwegian (Bokmål)_Norway" to
something that's all-ASCII (cf commits db29620d4 and aa1d2fc5e).
Add another mapping entry to handle this spelling.

Per bug #16068 from Robert Ford.  Like the previous patches,
back-patch to all supported branches.

Discussion: https://postgr.es/m/16068-4cb6eeaa7eb46d93@postgresql.org
2019-10-21 14:18:31 -04:00
Tom Lane 4e19bd41d9 Use CFLAGS_SL while probing linkability of libperl.
On recent Red Hat platforms (at least RHEL 8 and Fedora 30, maybe older),
configure's probe for libperl failed if the user forces CFLAGS to be -O0.
This is because some code in perl's inline.h fails to be optimized away
at -O0, and said code doesn't work if compiled without -fPIC.

To fix, add CFLAGS_SL to the compile flags used during the libperl probe.
This is a better simulation of the way that plperl is built, anyway,
so it might forestall other issues in future.

Per gripe from Kyotaro Horiguchi.  Back-patch to all supported branches,
since people might want to build older branches on these platforms.

Discussion: https://postgr.es/m/20191010.144533.263180400.horikyota.ntt@gmail.com
2019-10-21 13:52:25 -04:00
Tom Lane 99c51d5ed8 Select CFLAGS_SL at configure time, not in platform-specific Makefiles.
Move the platform-dependent logic that sets CFLAGS_SL from
src/makefiles/Makefile.foo to src/template/foo, so that the value
is determined at configure time and thus is available while running
configure's tests.

On a couple of platforms this might save a few microseconds of build
time by eliminating a test that make otherwise has to do over and over.
Otherwise it's pretty much a wash for build purposes; in particular,
this makes no difference to anyone who might be overriding CFLAGS_SL
via a make option.

This patch in itself does nothing with the value and thus should not
change any behavior, though you'll probably have to re-run configure
to get a correctly updated Makefile.global.  We'll use the new
configure variable in a follow-on patch.

Per gripe from Kyotaro Horiguchi.  Back-patch to all supported branches,
because the follow-on patch is a portability bug fix.

Discussion: https://postgr.es/m/20191010.144533.263180400.horikyota.ntt@gmail.com
2019-10-21 12:32:36 -04:00
Noah Misch af4477b00c For PowerPC instruction "addi", use constraint "b".
Without "b", a variant of the tas() code miscompiles on macOS 10.4.
This may also fix a compilation failure involving macOS 10.1.  Today's
compilers have been allocating acceptable registers with or without this
change, but this future-proofs the code by precisely conveying the
acceptable registers.  Back-patch to 9.4 (all supported versions).

Reviewed by Tom Lane.

Discussion: https://postgr.es/m/20191009063900.GA4066266@rfd.leadboat.com
2019-10-18 20:20:32 -07:00
Fujii Masao f7b70700bc Fix failure of archive recovery with recovery_min_apply_delay enabled.
recovery_min_apply_delay parameter is intended for use with streaming
replication deployments. However, the document clearly explains that
the parameter will be honored in all cases if it's specified. So it should
take effect even if in archive recovery. But, previously, archive recovery
with recovery_min_apply_delay enabled always failed, and caused assertion
failure if --enable-caasert is enabled.

The cause of this problem is that; the ownership of recoveryWakeupLatch
that recovery_min_apply_delay uses was taken only when standby mode
is requested. So unowned latch could be used in archive recovery, and
which caused the failure.

This commit changes recovery code so that the ownership of
recoveryWakeupLatch is taken even in archive recovery. Which prevents
archive recovery with recovery_min_apply_delay from failing.

Back-patch to v9.4 where recovery_min_apply_delay was added.

Author: Fujii Masao
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/CAHGQGwEyD6HdZLfdWc+95g=VQFPR4zQL4n+yHxQgGEGjaSVheQ@mail.gmail.com
2019-10-18 22:35:07 +09:00
Michael Paquier feed5ee475 Fix timeout handling in logical replication worker
The timestamp tracking the last moment a message is received in a
logical replication worker was initialized in each loop checking if a
message was received or not, causing wal_receiver_timeout to be ignored
in basically any logical replication deployments.  This also broke the
ping sent to the server when reaching half of wal_receiver_timeout.

This simply moves the initialization of the timestamp out of the apply
loop to the beginning of LogicalRepApplyLoop().

Reported-by: Jehan-Guillaume De Rorthais
Author: Julien Rouhaud
Discussion: https://postgr.es/m/CAOBaU_ZHESFcWva8jLjtZdCLspMj7vqaB2k++rjHLY897ZxbYw@mail.gmail.com
Backpatch-through: 10
2019-10-18 14:27:00 +09:00
Alvaro Herrera 45e4067c05 Fix minor bug in logical-replication walsender shutdown
Logical walsender should exit when it catches up with sending WAL during
shutdown; but there was a rare corner case when it failed to because of
a race condition that puts it back to wait for more WAL instead -- but
since there wasn't any, it'd not shut down immediately.  It would only
continue the shutdown when wal_sender_timeout terminates the sleep,
which causes annoying waits during shutdown procedure.  Restructure the
code so that we no longer forget to set WalSndCaughtUp in that case.

This was an oversight in commit c6c333436.

Backpatch all the way down to 9.4.

Author: Craig Ringer, Álvaro Herrera
Discussion: https://postgr.es/m/CAMsr+YEuz4XwZX_QmnX_-2530XhyAmnK=zCmicEnq1vLr0aZ-g@mail.gmail.com
2019-10-17 15:06:05 +02:00
Thomas Munro 6737111a7a When restoring GUCs in parallel workers, show an error context.
Otherwise it can be hard to see where an error is coming from, when
the parallel worker sets all the GUCs that it received from the
leader.  Bug #15726.  Back-patch to 9.5, where RestoreGUCState()
appeared.

Reported-by: Tiago Anastacio
Reviewed-by: Daniel Gustafsson, Tom Lane
Discussion: https://postgr.es/m/15726-6d67e4fa14f027b3%40postgresql.org
2019-10-17 13:58:58 +13:00
Thomas Munro 6f1e336de0 Fix bug that could try to freeze running multixacts.
Commits 801c2dc7 and 801c2dc7 made it possible for vacuum to
try to freeze a multixact that is still running.  That was
prevented by a check, but raised an error.  Repair.

Back-patch all the way.

Author: Nathan Bossart, Jeremy Schneider
Reported-by: Jeremy Schneider
Reviewed-by: Jim Nasby, Thomas Munro
Discussion: https://postgr.es/m/DAFB8AFF-2F05-4E33-AD7F-FF8B0F760C17%40amazon.com
2019-10-17 11:01:35 +13:00
Tomas Vondra d071a2539f Improve the check for pg_catalog.unknown data type in pg_upgrade
The pg_upgrade check for pg_catalog.unknown type when upgrading from 9.6
had a couple of issues with domains and composite types - it detected
even composite types unused in objects with storage. So for example this
was enough to trigger an unnecessary pg_upgrade failure:

  CREATE TYPE unknown_composite AS (u pg_catalog.unknown)

On the other hand, this only happened with composite types directly on
the pg_catalog.unknown data type, but not with a domain. So this was not
detected

  CREATE DOMAIN unknown_domain AS pg_catalog.unknown;
  CREATE TYPE unknown_composite_2 AS (u unknown_domain);

unlike the first example. These false positives and inconsistencies are
unfortunate, but what's worse we've failed to detected objects using the
pg_catalog.unknown type through a domain. So we missed cases like this

  CREATE TABLE t (u unknown_composite_2);

The consequence is clusters broken after a pg_upgrade.

This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 10, where the of pg_catalog.unknown data type was restricted.

Author: Tomas Vondra
Backpatch-to: 10-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
2019-10-16 13:26:26 +02:00
Tomas Vondra a970b6cdeb Improve the check for pg_catalog.line data type in pg_upgrade
The pg_upgrade check for pg_catalog.line data type when upgrading from
9.3 had a couple of issues with domains and composite types. Firstly, it
triggered false positives for composite types unused in objects with
storage. This was enough to trigger an unnecessary pg_upgrade failure:

  CREATE TYPE line_composite AS (l pg_catalog.line)

On the other hand, this only happened with composite types directly on
the pg_catalog.line data type, but not with a domain. So this was not
detected

  CREATE DOMAIN line_domain AS pg_catalog.line;
  CREATE TYPE line_composite_2 AS (l line_domain);

unlike the first example. These false positives and inconsistencies are
unfortunate, but what's worse we've failed to detected objects using the
pg_catalog.line data type through a domain. So we missed cases like this

  CREATE TABLE t (l line_composite_2);

The consequence is clusters broken after a pg_upgrade.

This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. 9.3 did not
support domains on composite types, but we can still have multi-level
composite types.

Backpatch all the way to 9.4, where the format for pg_catalog.line data
type changed.

Author: Tomas Vondra
Backpatch-to: 9.4-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
2019-10-16 13:26:22 +02:00
Michael Paquier b60febc6d7 Doc: Fix various inconsistencies
This fixes multiple areas of the documentation:
- COPY for its past compatibility section.
- SET ROLE mentioning INHERITS instead of INHERIT
- PREPARE referring to stmt_name, that is not present.
- Extension documentation about format name with upgrade scripts.

Backpatch down to 9.4 for the relevant parts.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/bf95233a-9943-b341-e2ff-a860c28af481@gmail.com
Backpatch-through: 9.4
2019-10-16 13:10:40 +09:00
Noah Misch e5b4926ef6 AIX: Stop adding option -qsrcmsg.
With xlc v16.1.0, it causes internal compiler errors.  With xlc versions
not exhibiting that bug, removing -qsrcmsg merely changes the compiler
error reporting format.  Back-patch to 9.4 (all supported versions).

Discussion: https://postgr.es/m/20191003064105.GA3955242@rfd.leadboat.com
2019-10-12 00:21:50 -07:00
Michael Paquier e34358c436 Flush logical mapping files with fd opened for read/write at checkpoint
The file descriptor was opened with read-only to fsync a regular file,
which would cause EBADFD errors on some platforms.

This is similar to the recent fix done by a586cc4b (which was broken by
me with 82a5649), except that I noticed this issue while monitoring the
backend code for similar mistakes.  Backpatch to 9.4, as this has been
introduced since logical decoding exists as of b89e151.

Author: Michael Paquier
Reviewed-by: Andres Freund
Discussion: https://postgr.es/m/20191006045548.GA14532@paquier.xyz
Backpatch-through: 9.4
2019-10-09 13:31:17 +09:00
Bruce Momjian e4ca62b948 docs: Improve A?synchronous Multimaster Replication descr.
The docs for sync and async multimaster replication were unclear about
when to use it, and when it has benefits;  this change clarifies that.

Reported-by: juha-pekka.eloranta@reaktor.fi

Discussion: https://postgr.es/m/156856543824.1274.12180817186798859836@wrigleys.postgresql.org

Backpatch-through: 9.4
2019-10-07 18:06:08 -04:00
Bruce Momjian f68d5bff5e docs: clarify that today/tomorrow/yesterday is at 00:00
This should help people clearly know that these days start at midnight.

Reported-by: David Harper

Discussion: https://postgr.es/m/156258047907.1181.11324468080514061996@wrigleys.postgresql.org

Backpatch-through: 9.4
2019-10-07 17:26:46 -04:00
Bruce Momjian f872f18173 doc: move mention of log_min_error_statement in a better spot
Previously it was mentioned in the lock_timeout docs in a confusing
location.

Reported-by: ivaylo.zlatanov@gmail.com

Discussion: https://postgr.es/m/157019615723.25307.15449102262106437404@wrigleys.postgresql.org

Backpatch-through: 9.4
2019-10-07 14:33:31 -04:00
Tom Lane 021065aac6 Check for too many postmaster children before spawning a bgworker.
The postmaster's code path for spawning a bgworker neglected to check
whether we already have the max number of live child processes.  That's
a bit hard to hit, since it would necessarily be a transient condition;
but if we do, AssignPostmasterChildSlot() fails causing a postmaster
crash, as seen in a report from Bhargav Kamineni.

To fix, invoke canAcceptConnections() in the bgworker code path, as we
do in the other code paths that spawn children.  Since we don't want
the same pmState tests in this case, add a child-process-type parameter
to canAcceptConnections() so that it can know what to do.

Back-patch to 9.5.  In principle the same hazard exists in 9.4, but the
code is enough different that this patch wouldn't quite fix it there.
Given the tiny usage of bgworkers in that branch it doesn't seem worth
creating a variant patch for it.

Discussion: https://postgr.es/m/18733.1570382257@sss.pgh.pa.us
2019-10-07 12:39:09 -04:00
Noah Misch eb74022d24 Report test_atomic_ops() failures consistently, via macros.
This prints the unexpected value in more failure cases, and it removes
forty-eight hand-maintained error messages.  Back-patch to 9.5, which
introduced these tests.

Reviewed (in an earlier version) by Andres Freund.

Discussion: https://postgr.es/m/20190915160021.GA24376@alvherre.pgsql
2019-10-05 10:05:08 -07:00
Tom Lane 4fd51980ed Avoid use of wildcard in pg_waldump's .gitignore.
This would be all right, maybe, if it didn't also match a file that
definitely should not be ignored.  We don't add rmgrs so often that
manual maintenance of this file list is impractical, so just write
out the list.

(I find the equivalent wildcard use in the Makefile pretty lazy and
unsafe as well, but will leave that alone until it actually causes a
problem.)

Per bug #16042 from Denis Stuchalin.

Discussion: https://postgr.es/m/16042-c174ee692ac21cbd@postgresql.org
2019-10-05 12:26:55 -04:00
Andres Freund 77b2d95b14 Disable one more set of tests from c884119950.
Discussion: https://postgr.es/m/20191004222437.45qmglpto43pd3jb@alap3.anarazel.de
Backpatch: 9.6-, just like c884119950 and 6e61d75f52
2019-10-05 08:11:09 -07:00
Andres Freund 5711a1828f Disable one set of tests from c884119950.
One of the upsert related tests is unstable (sometimes even hanging
until isolationtester's step timeout is reached). Based on preliminary
analysis that might be a problem outside of just that test, but not
really related to EPQ and triggers.  Disable for now, to get the
buildfarm greener again.

Discussion: https://postgr.es/m/20191004222437.45qmglpto43pd3jb@alap3.anarazel.de
Backpatch: 9.6-, just like c884119950.
2019-10-04 21:37:54 -07:00
Andres Freund 54b0feaf90 Add isolation tests for the combination of EPQ and triggers.
As evidenced by bug #16036 this area is woefully under-tested. Add
fairly extensive tests for the combination.

Backpatch back to 9.6 - before that isolationtester was not capable
enough. While we don't backpatch tests all the time, future fixes to
trigger.c would potentially look different enough in 12+ from the
earlier branches that introducing bugs during backpatching is more
likely than normal. Also, it's just a crucial and undertested area of
the code.

Author: Andres Freund
Discussion: https://postgr.es/m/16036-28184c90d952fb7f@postgresql.org
Backpatch: 9.6-, the earliest these tests work
2019-10-04 14:01:35 -07:00
Andrew Dunstan 3b9c227008 Handle spaces in OpenSSL install location for MSVC
First, make sure that the .exe name is quoted when trying to get the
version number. Also, don't quote the lib name for using in the project
files if it's already been quoted. This second change applies to all
libraries, not just OpenSSL.

This has clearly been broken forever, so backpatch to all live branches.
2019-10-04 15:39:19 -04:00
Tom Lane b8ddf0bdf7 Fix bitshiftright()'s zero-padding some more.
Commit 5ac0d9360 failed to entirely fix bitshiftright's habit of
leaving one-bits in the pad space that should be all zeroes,
because in a moment of sheer brain fade I'd concluded that only
the code path used for not-a-multiple-of-8 shift distances needed
to be fixed.  Of course, a multiple-of-8 shift distance can also
cause the problem, so we need to forcibly zero the extra bits
in both cases.

Per bug #16037 from Alexander Lakhin.  As before, back-patch to all
supported branches.

Discussion: https://postgr.es/m/16037-1d1ebca564db54f4@postgresql.org
2019-10-04 10:34:21 -04:00
Tom Lane e5ff975719 Avoid unnecessary out-of-memory errors during encoding conversion.
Encoding conversion uses the very simplistic rule that the output
can't be more than 4X longer than the input, and palloc's a buffer
of that size.  This results in failure to convert any string longer
than 1/4 GB, which is becoming an annoying limitation.

As a band-aid to improve matters, allow the allocated output buffer
size to exceed 1GB.  We still insist that the final result fit into
MaxAllocSize (1GB), though.  Perhaps it'd be safe to relax that
restriction, but it'd require close analysis of all callers, which
is daunting (not least because external modules might call these
functions).  For the moment, this should allow a 2X to 4X improvement
in the longest string we can convert, which is a useful gain in
return for quite a simple patch.

Also, once we have successfully converted a long string, repalloc
the output down to the actual string length, returning the excess
to the malloc pool.  This seems worth doing since we can usually
expect to give back several MB if we take this path at all.

This still leaves much to be desired, most notably that the assumption
that MAX_CONVERSION_GROWTH == 4 is very fragile, and yet we have no
guard code verifying that the output buffer isn't overrun.  Fixing
that would require significant changes in the encoding conversion
APIs, so it'll have to wait for some other day.

The present patch seems safely back-patchable, so patch all supported
branches.

Alvaro Herrera and Tom Lane

Discussion: https://postgr.es/m/20190816181418.GA898@alvherre.pgsql
Discussion: https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
2019-10-03 17:34:25 -04:00
Tom Lane 82d0a46ea3 Allow repalloc() to give back space when a large chunk is downsized.
Up to now, if you resized a large (>8K) palloc chunk down to a smaller
size, aset.c made no attempt to return any space to the malloc pool.
That's unpleasant if a really large allocation is resized to a
significantly smaller size.  I think no such cases existed when this
code was designed, and I'm not sure whether they're common even yet,
but an upcoming fix to encoding conversion will certainly create such
cases.  Therefore, fix AllocSetRealloc so that it gives realloc()
a chance to do something with the block.  This doesn't noticeably
increase complexity, we mostly just have to change the order in which
the cases are considered.

Back-patch to all supported branches.

Discussion: https://postgr.es/m/20190816181418.GA898@alvherre.pgsql
Discussion: https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
2019-10-03 13:56:26 -04:00
Andrew Gierth 0a445f2790 Selectively include window frames in expression walks/mutates.
query_tree_walker and query_tree_mutator were skipping the
windowClause of the query, without regard for the fact that the
startOffset and endOffset in a WindowClause node are expression trees
that need to be processed. This was an oversight in commit ec4be2ee6
from 2010 which added the expression fields; the main symptom is that
function parameters in window frame clauses don't work in inlined
functions.

Fix (as conservatively as possible since this needs to not break
existing out-of-tree callers) and add tests.

Backpatch all the way, since this has been broken since 9.0.

Per report from Alastair McKinley; fix by me with kibitzing and review
from Tom Lane.

Discussion: https://postgr.es/m/DB6PR0202MB2904E7FDDA9D81504D1E8C68E3800@DB6PR0202MB2904.eurprd02.prod.outlook.com
2019-10-03 11:14:30 +01:00
Michael Paquier b978de0eba Remove temporary WAL and history files at the end of archive recovery
cbc55da has reworked the order of some actions at the end of archive
recovery.  Unfortunately this overlooked the fact that the startup
process needs to remove RECOVERYXLOG (for temporary WAL segment newly
recovered from archives) and RECOVERYHISTORY (for temporary history
file) at this step, leaving the files around even after recovery ended.

Backpatch to 9.5, like the previous commit.

Author: Sawada Masahiko
Reviewed-by: Fujii Masao, Michael Paquier
Discussion: https://postgr.es/m/CAD21AoBO_eDQub6zojFnWtnmutRBWvYf7=cW4Hsqj+U_R26w3Q@mail.gmail.com
Backpatch-through: 9.5
2019-10-02 15:53:56 +09:00
Andrew Dunstan b81a82e395 Suppress another CR in program output
This one was exposed by a12c75a10.

Backpatch to release 11 where check_pg_config was introduced.
2019-09-30 15:51:03 -04:00
Andres Freund 51ad5b9bd3 jit: Re-allow JIT compilation of execGrouping.c hashtable comparisons.
In the course of 5567d12ce0, 356687bd8 and 317ffdfeaa, I changed
BuildTupleHashTable[Ext]'s call to ExecBuildGroupingEqual to not pass
in the parent node, but NULL. Which in turn prevents the tuple
equality comparator from being JIT compiled.  While that fixes
bug #15486, it is not actually necessary after all of the above commits,
as we don't re-build the comparator when using the new
BuildTupleHashTableExt() interface (as the content of the hashtable
are reset, but the TupleHashTable itself is not).

Therefore re-allow jit compilation for callers that use
BuildTupleHashTableExt with a separate context for "metadata" and
content.

As in the previous commit, there's ongoing work to make this easier to
test to prevent such regressions in the future, but that
infrastructure is not going to be backpatchable.

The performance impact of not JIT compiling hashtable equality
comparators can be substantial e.g. for aggregation queries that
aggregate a lot of input rows to few output rows (when there are a lot
of output groups, there will be fewer comparisons).

Author: Andres Freund
Discussion: https://postgr.es/m/20190927072053.njf6prdl3vb7y7qb@alap3.anarazel.de
Backpatch: 11, just as 5567d12ce0
2019-09-29 16:27:08 -07:00
Andrew Dunstan 51e3005a07 Allow SSL TAP tests to run on Windows
Windows does not enforce key file permissions checks in libpq, and psql
can produce CRLF line endings on Windows.

Backpatch to Release 12 (CRLF) and Release 11 (permissions check)
2019-09-29 17:50:44 -04:00
Tom Lane 0baa55655e Improve stability of partition_prune regression test.
This test already knew that, to get stable test output, it had to hide
"loops" counts in EXPLAIN ANALYZE results.  But that's not nearly enough:
if we get a smaller number of workers than we planned for, then the
"Workers Launched" number will change, and so will all the rows and loops
counts up to the Gather node.  This has resulted in repeated failures in
the buildfarm, so adjust the test to filter out all these counts.

(Really, we wouldn't bother with EXPLAIN ANALYZE at all here, except
that currently the only way to verify that executor-time pruning has
happened is to look for '(never executed)' annotations.  Those are
stable and needn't be filtered out.)

Back-patch to v11 where the test was introduced.

Discussion: https://postgr.es/m/11952.1569536725@sss.pgh.pa.us
2019-09-28 13:33:34 -04:00
Tom Lane 240c38a488 Doc: clean up markup for jsonb_set and related functions.
The markup for optional parameters was neither correct nor consistent.
In passing, fix a spelling mistake.

Per report from Alex Macy.  Some of these mistakes are old, so
back-patch as appropriate.

Discussion: https://postgr.es/m/156953522258.1204.12736099368284950578@wrigleys.postgresql.org
2019-09-27 11:01:36 -04:00
Amit Kapila 84ced048d1 Fix oversight in commit 4429f6a9e3.
The test name and the following test cases suggest the index created
should be hash index, but it forgot to add 'using hash' in the test case.
This in itself won't improve code coverage as there were some other tests
which were covering the corresponding code.  However, it is better if the
added tests serve their actual purpose.

Reported-by: Paul A Jungwirth
Author: Paul A Jungwirth
Reviewed-by: Mahendra Singh
Backpatch-through: 9.4
Discussion: https://postgr.es/m/CA+renyV=Us-5XfMC25bNp-uWSj39XgHHmGE9Rh2cQKMegSj52g@mail.gmail.com
2019-09-27 08:11:46 +05:30
Michael Paquier d01d4f2374 Fix failure with lock mode used for custom relation options
In-core relation options can use a custom lock mode since 47167b7, that
has lowered the lock available for some autovacuum parameters.  However
it forgot to consider custom relation options.  This causes failures
with ALTER TABLE SET when changing a custom relation option, as its lock
is not defined.  The existing APIs to define a custom reloption does not
allow to define a custom lock mode, so enforce its initialization to
AccessExclusiveMode which should be safe enough in all cases.  An
upcoming patch will extend the existing APIs to allow a custom lock mode
to be defined.

The problem can be reproduced with bloom indexes, so add a test there.

Reported-by: Nikolay Sharplov
Analyzed-by: Thomas Munro, Michael Paquier
Author: Michael Paquier
Reviewed-by: Kuntal Ghosh
Discussion: https://postgr.es/m/20190920013831.GD1844@paquier.xyz
Backpatch-through: 9.6
2019-09-25 10:08:30 +09:00
Tom Lane 7cd99d3947 Doc: clarify handling of duplicate elements in array containment tests.
The array <@ and @> operators do not worry about duplicates: if every
member of array X matches some element of array Y, then X is contained
in Y, even if several members of X get matched to the same Y member.
This was not explicitly stated in the docs though, so improve matters.

Discussion: https://postgr.es/m/156614120484.1310.310161642239149585@wrigleys.postgresql.org
2019-09-23 12:37:04 -04:00