Commit Graph

37365 Commits

Author SHA1 Message Date
Peter Eisentraut 124966c1a3 Put some psql documentation pieces back into alphabetical order 2021-05-21 17:10:09 +02:00
Amit Kapila 6d0eb38557 Fix deadlock for multiple replicating truncates of the same table.
While applying the truncate change, the logical apply worker acquires
RowExclusiveLock on the relation being truncated. This allowed truncate on
the relation at a time by two apply workers which lead to a deadlock. The
reason was that one of the workers after updating the pg_class tuple tries
to acquire SHARE lock on the relation and started to wait for the second
worker which has acquired RowExclusiveLock on the relation. And when the
second worker tries to update the pg_class tuple, it starts to wait for
the first worker which leads to a deadlock. Fix it by acquiring
AccessExclusiveLock on the relation before applying the truncate change as
we do for normal truncate operation.

Author: Peter Smith, test case by Haiying Tang
Reviewed-by: Dilip Kumar, Amit Kapila
Backpatch-through: 11
Discussion: https://postgr.es/m/CAHut+PsNm43p0jM+idTvWwiGZPcP0hGrHMPK9TOAkc+a4UpUqw@mail.gmail.com
2021-05-21 07:54:27 +05:30
Tom Lane f21fadafaf Avoid detoasting failure after COMMIT inside a plpgsql FOR loop.
exec_for_query() normally tries to prefetch a few rows at a time
from the query being iterated over, so as to reduce executor
entry/exit overhead.  Unfortunately this is unsafe if we have
COMMIT or ROLLBACK within the loop, because there might be
TOAST references in the data that we prefetched but haven't
yet examined.  Immediately after the COMMIT/ROLLBACK, we have
no snapshots in the session, meaning that VACUUM is at liberty
to remove recently-deleted TOAST rows.

This was originally reported as a case triggering the "no known
snapshots" error in init_toast_snapshot(), but even if you miss
hitting that, you can get "missing toast chunk", as illustrated
by the added isolation test case.

To fix, just disable prefetching in non-atomic contexts.  Maybe
there will be performance complaints prompting us to work harder
later, but it's not clear at the moment that this really costs
much, and I doubt we'd want to back-patch any complicated fix.

In passing, adjust that error message in init_toast_snapshot()
to be a little clearer about the likely cause of the problem.

Patch by me, based on earlier investigation by Konstantin Knizhnik.

Per bug #15990 from Andreas Wicht.  Back-patch to v11 where
intra-procedure COMMIT was added.

Discussion: https://postgr.es/m/15990-eee2ac466b11293d@postgresql.org
2021-05-20 18:32:37 -04:00
Andrew Dunstan bdbb2ce7d5
Install PostgresVersion.pm
A lamentable oversight on my part meant that when PostgresVersion.pm was
added in commit 4c4eaf3d19 provision to install it was not added to the
Makefile, so it was not installed along with the other perl modules.
2021-05-20 15:11:17 -04:00
Tom Lane 6d59a218c3 Clean up cpluspluscheck violation.
"typename" is a C++ keyword, so pg_upgrade.h fails to compile in C++.
Fortunately, there seems no likely reason for somebody to need to
do that.  Nonetheless, it's project policy that all .h files should
pass cpluspluscheck, so rename the argument to fix that.

Oversight in 57c081de0; back-patch as that was.  (The policy requiring
pg_upgrade.h to pass cpluspluscheck only goes back to v12, but it
seems best to keep this code looking the same in all branches.)
2021-05-20 13:03:08 -04:00
Andrew Dunstan 8bdd6f563a
Use a more portable way to get the version string in PostgresNode
Older versions of perl on Windows don't like the list form of pipe open,
and perlcritic doesn't like the string form of open, so we avoid both
with a simpler formulation using qx{}.

Per complaint from Amit Kapila.
2021-05-20 08:07:08 -04:00
Tom Lane 413c1ef98e Avoid creating testtablespace directories where not wanted.
Recently we refactored things so that pg_regress makes the
"testtablespace" subdirectory used by the core regression tests,
instead of doing that in the makefiles.  That had the undesirable
side effect of making such a subdirectory in every directory that
has "input" or "output" test files.  Since these subdirectories
remain empty, git doesn't complain about them, but nonetheless
they're clutter.

To fix, invent an explicit --make-testtablespace-dir switch,
so that pg_regress only makes the subdirectory when explicitly
told to.

Discussion: https://postgr.es/m/2854388.1621284789@sss.pgh.pa.us
2021-05-19 14:04:01 -04:00
Dean Rasheed 0f516d039d Fix pgbench permute tests.
One of the tests for the pgbench permute() function added by
6b258e3d68 fails on some 32-bit platforms, due to variations in the
floating point computations in getrand(). The remaining tests give
sufficient coverage, so just remove the failing test.

Reported by Christoph Berg. Analysis by Thomas Munro and Tom Lane.
Based on patch by Fabien Coelho.

Discussion: https://postgr.es/m/YKQnUoYV63GRJBDD@msg.df7cb.de
2021-05-19 12:50:58 +01:00
Fujii Masao 167bd48049 Make standby promotion reset the recovery pause state to 'not paused'.
If a promotion is triggered while recovery is paused, the paused state ends
and promotion continues. But previously in that case
pg_get_wal_replay_pause_state() returned 'paused' wrongly while a promotion
was ongoing.

This commit changes a standby promotion so that it marks the recovery
pause state as 'not paused' when it's triggered, to fix the issue.

Author: Fujii Masao
Reviewed-by: Dilip Kumar, Kyotaro Horiguchi
Discussion: https://postgr.es/m/f706876c-4894-0ba5-6f4d-79803eeea21b@oss.nttdata.com
2021-05-19 13:48:19 +09:00
Amit Kapila 0a442a408b Fix 020_messages.pl test.
We were not waiting for a publisher to catch up with the subscriber after
creating a subscription. Now, it can happen that apply worker starts
replication even after we have disabled the subscription in the test. This
will make the test expect that there is no active slot whereas there
exists one. Fix this symptom by allowing the publisher to wait for
catching up with the subscription.

It is not a good idea to ensure if the slot is still active by checking
for walsender existence as we release the slot after we clean up the
walsender related memory. Fix that by checking the slot status in
pg_replication_slots.

Also, it is better to avoid repeated enabling/disabling of the
subscription.

Finally, we make autovacuum off for this test to avoid any empty
transaction appearing in the test while consuming changes.

Reported-by: as per buildfarm
Author: Vignesh C
Reviewed-by: Amit Kapila, Michael Paquier
Discussion: https://postgr.es/m/CAA4eK1+uW1UGDHDz-HWMHMen76mKP7NJebOTZN4uwbyMjaYVww@mail.gmail.com
2021-05-19 08:54:46 +05:30
Fujii Masao d8735b8b46 Fix issues in pg_stat_wal.
1) Previously there were both pgstat_send_wal() and pgstat_report_wal()
   in order to send WAL activity to the stats collector. With the former being
   used by wal writer, the latter by most other processes. They were a bit
   redundant and so this commit merges them into pgstat_send_wal() to
   simplify the code.

2) Previously WAL global statistics counters were calculated and then
   compared with zero-filled buffer in order to determine whether any WAL
   activity has happened since the last submission. These calculation and
   comparison were not cheap. This was regularly exercised even in read-only
   workloads. This commit fixes the issue by making some WAL activity
   counters directly be checked to determine if there's WAL activity stats
   to send.

3) Previously pgstat_report_stat() did not check if there's WAL activity
   stats to send as part of the "Don't expend a clock check if nothing to do"
   check at the top. It's probably rare to have pending WAL stats without
   also passing one of the other conditions, but for safely this commit
   changes pgstat_report_stats() so that it checks also some WAL activity
   counters at the top.

This commit also adds the comments about the design of WAL stats.

Reported-by: Andres Freund
Author: Masahiro Ikeda
Reviewed-by: Kyotaro Horiguchi, Atsushi Torikoshi, Andres Freund, Fujii Masao
Discussion: https://postgr.es/m/20210324232224.vrfiij2rxxwqqjjb@alap3.anarazel.de
2021-05-19 11:38:34 +09:00
Michael Paquier 694da1983e Add --no-toast-compression to pg_dumpall
This is an oversight from bbe0a81d, where the equivalent option exists
in pg_dump.  This is useful to be able to reset the compression methods
cluster-wide when restoring the data based on default_toast_compression.

Reviewed-by: Daniel Gustafsson, Tom Lane
Discussion: https://postgr.es/m/YKHC+qCJvzCRVCpY@paquier.xyz
2021-05-19 09:38:48 +09:00
David Rowley 2ded19fa3a Fix typo and outdated information in README.barrier
README.barrier didn't seem to get the memo when atomics were added. Fix
that.

Author: Tatsuo Ishii, David Rowley
Discussion: https://postgr.es/m/20210516.211133.2159010194908437625.t-ishii%40sraoss.co.jp
Backpatch-through: 9.6, oldest supported release
2021-05-18 09:54:56 +12:00
Peter Eisentraut 6292b83074 Translation updates
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 9bbd9c3714d0c76daaa806588b1fbf744aa60496
2021-05-17 14:30:27 +02:00
Alvaro Herrera 354f32d01d
Unbreak EXEC_BACKEND build
Per buildfarm
2021-05-15 15:17:15 -04:00
Alvaro Herrera cafde58b33
Allow compute_query_id to be set to 'auto' and make it default
Allowing only on/off meant that all either all existing configuration
guides would become obsolete if we disabled it by default, or that we
would have to accept a performance loss in the default config if we
enabled it by default.  By allowing 'auto' as a middle ground, the
performance cost is only paid by those who enable pg_stat_statements and
similar modules.

I only edited the release notes to comment-out a paragraph that is now
factually wrong; further edits are probably needed to describe the
related change in more detail.

Author: Julien Rouhaud <rjuju123@gmail.com>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20210513002623.eugftm4nk2lvvks3@nol
2021-05-15 14:13:09 -04:00
Tom Lane 30d8bad494 Be more careful about barriers when releasing BackgroundWorkerSlots.
ForgetBackgroundWorker lacked any memory barrier at all, while
BackgroundWorkerStateChange had one but unaccountably did
additional manipulation of the slot after the barrier.  AFAICS,
the rule must be that the barrier is immediately before setting
or clearing slot->in_use.

It looks like back in 9.6 when ForgetBackgroundWorker was first
written, there might have been some case for not needing a
barrier there, but I'm not very convinced of that --- the fact
that the load of bgw_notify_pid is in the caller doesn't seem
to guarantee no memory ordering problem.  So patch 9.6 too.

It's likely that this doesn't fix any observable bug on Intel
hardware, but machines with weaker memory ordering rules could
have problems here.

Discussion: https://postgr.es/m/4046084.1620244003@sss.pgh.pa.us
2021-05-15 12:21:06 -04:00
Peter Geoghegan 8f72bbac3e Harden nbtree deduplication posting split code.
Add a defensive "can't happen" error to code that handles nbtree posting
list splits (promote an existing assertion).  This avoids a segfault in
the event of an insertion of a newitem that is somehow identical to an
existing non-pivot tuple in the index.  An nbtree index should never
have two index tuples with identical TIDs.

This scenario is not particular unlikely in the event of any kind of
corruption that leaves the index in an inconsistent state relative to
the heap relation that is indexed.  There are two known reports of
preventable hard crashes.  Doing nothing seems unacceptable given the
general expectation that nbtree will cope reasonably well with corrupt
data.

Discussion: https://postgr.es/m/CAH2-Wz=Jr_d-dOYEEmwz0-ifojVNWho01eAqewfQXgKfoe114w@mail.gmail.com
Backpatch: 13-, where nbtree deduplication was introduced.
2021-05-14 15:08:02 -07:00
Tom Lane c3c35a733c Prevent infinite insertion loops in spgdoinsert().
Formerly we just relied on operator classes that assert longValuesOK
to eventually shorten the leaf value enough to fit on an index page.
That fails since the introduction of INCLUDE-column support (commit
09c1c6ab4), because the INCLUDE columns might alone take up more
than a page, meaning no amount of leaf-datum compaction will get
the job done.  At least with spgtextproc.c, that leads to an infinite
loop, since spgtextproc.c won't throw an error for not being able
to shorten the leaf datum anymore.

To fix without breaking cases that would otherwise work, add logic
to spgdoinsert() to verify that the leaf tuple size is decreasing
after each "choose" step.  Some opclasses might not decrease the
size on every single cycle, and in any case, alignment roundoff
of the tuple size could obscure small gains.  Therefore, allow
up to 10 cycles without additional savings before throwing an
error.  (Perhaps this number will need adjustment, but it seems
quite generous right now.)

As long as we've developed this logic, let's back-patch it.
The back branches don't have INCLUDE columns to worry about, but
this seems like a good defense against possible bugs in operator
classes.  We already know that an infinite loop here is pretty
unpleasant, so having a defense seems to outweigh the risk of
breaking things.  (Note that spgtextproc.c is actually the only
known opclass with longValuesOK support, so that this is all moot
for known non-core opclasses anyway.)

Per report from Dilip Kumar.

Discussion: https://postgr.es/m/CAFiTN-uxP_soPhVG840tRMQTBmtA_f_Y8N51G7DKYYqDh7XN-A@mail.gmail.com
2021-05-14 15:07:34 -04:00
Tom Lane eb7a6b9229 Fix query-cancel handling in spgdoinsert().
Knowing that a buggy opclass could cause an infinite insertion loop,
spgdoinsert() intended to allow its loop to be interrupted by query
cancel.  However, that never actually worked, because in iterations
after the first, we'd be holding buffer lock(s) which would cause
InterruptHoldoffCount to be positive, preventing servicing of the
interrupt.

To fix, check if an interrupt is pending, and if so fall out of
the insertion loop and service the interrupt after we've released
the buffers.  If it was indeed a query cancel, that's the end of
the matter.  If it was a non-canceling interrupt reason, make use
of the existing provision to retry the whole insertion.  (This isn't
as wasteful as it might seem, since any upper-level index tuples we
already created should be usable in the next attempt.)

While there's no known instance of such a bug in existing release
branches, it still seems like a good idea to back-patch this to
all supported branches, since the behavior is fairly nasty if a
loop does happen --- not only is it uncancelable, but it will
quickly consume memory to the point of an OOM failure.  In any
case, this code is certainly not working as intended.

Per report from Dilip Kumar.

Discussion: https://postgr.es/m/CAFiTN-uxP_soPhVG840tRMQTBmtA_f_Y8N51G7DKYYqDh7XN-A@mail.gmail.com
2021-05-14 13:29:39 -04:00
Tom Lane e47f93f981 Refactor CHECK_FOR_INTERRUPTS() to add flexibility.
Split up CHECK_FOR_INTERRUPTS() to provide an additional macro
INTERRUPTS_PENDING_CONDITION(), which just tests whether an
interrupt is pending without attempting to service it.  This is
useful in situations where the caller knows that interrupts are
blocked, and would like to find out if it's worth the trouble
to unblock them.

Also add INTERRUPTS_CAN_BE_PROCESSED(), which indicates whether
CHECK_FOR_INTERRUPTS() can be relied on to clear the pending interrupt.

This commit doesn't actually add any uses of the new macros,
but a follow-on bug fix will do so.  Back-patch to all supported
branches to provide infrastructure for that fix.

Alvaro Herrera and Tom Lane

Discussion: https://postgr.es/m/20210513155351.GA7848@alvherre.pgsql
2021-05-14 13:29:39 -04:00
Peter Eisentraut 09ae329957 Message style improvements 2021-05-14 10:26:41 +02:00
David Rowley 6cb93beddd Convert misleading while loop into an if condition
This seems to be leftover from ea15e1867 and from when we used to evaluate
SRFs at each node.

Since there is an unconditional "return" at the end of the loop body, only
1 loop is ever possible, so we can just change this into an if condition.

There is no actual bug being fixed here so no back-patch. It seems fine to
just fix this anomaly in master only.

Author: Greg Nancarrow
Discussion: https://postgr.es/m/CAJcOf-d7T1q0az-D8evWXnsuBZjigT04WkV5hCAOEJQZRWy28w@mail.gmail.com
2021-05-14 12:26:11 +12:00
Peter Geoghegan fbe9b80610 Fix autovacuum log output heap truncation issue.
The percentage of blocks from the table value reported by autovacuum log
output (following commit 5100010ee4) should never exceed 100% because
it describes the state of the table back when lazy_vacuum() was called.
The value could nevertheless exceed 100% in the event of heap relation
truncation.  We failed to compensate for how truncation affects
rel_pages.

Fix the faulty accounting by using the original rel_pages value instead
of the current/final rel_pages value.

Reported-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20210423204306.5osfpkt2ggaedyvy@alap3.anarazel.de
2021-05-13 16:07:17 -07:00
Peter Eisentraut 5a73a9e3b5 pg_amcheck: Message style and formatting improvements 2021-05-13 08:23:13 +02:00
Michael Paquier 1906cc07d9 Make saner the tab completion of INSERT and DELETE in psql
When specified directly as DML queries, INSERT was not getting always
completed to "INSERT INTO", same for DELETE with "DELETE FROM".  This
makes the completion behavior more consistent for both commands, saving
a few keystrokes.

Commands on policies, triggers, grant/revoke, etc. require only DELETE
as completion keyword.

Author: Haiying Tang
Reviewed-by: Dilip Kumar, Julien Rouhaud
Discussion: https://postgr.es/m/OS0PR01MB61135AE2B07CCD1AB8C6A0F6FB549@OS0PR01MB6113.jpnprd01.prod.outlook.com
2021-05-13 09:48:28 +09:00
Alvaro Herrera db16c65647
Rename the logical replication global "wrconn"
The worker.c global wrconn is only meant to be used by logical apply/
tablesync workers, but there are other variables with the same name. To
reduce future confusion rename the global from "wrconn" to
"LogRepWorkerWalRcvConn".

While this is just cosmetic, it seems better to backpatch it all the way
back to 10 where this code appeared, to avoid future backpatching
issues.

Author: Peter Smith <smithpb2250@gmail.com>
Discussion: https://postgr.es/m/CAHut+Pu7Jv9L2BOEx_Z0UtJxfDevQSAUW2mJqWU+CtmDrEZVAg@mail.gmail.com
2021-05-12 19:13:54 -04:00
Tom Lane 7dde98728a Double-space commands in system_constraints.sql/system_functions.sql.
Previously, any error reported by the backend while reading
system_constraints.sql would report the entire file, not just the
particular command it was working on.  (Ask me how I know.)  Likewise,
there were chunks of system_functions.sql that would be read as one
command, which would be annoying if anything failed there.

The issue for system_constraints.sql is an oversight in commit
dfb75e478.  I didn't try to trace down where the poor formatting
in system_functions.sql started, but it's certainly contrary to
the advice at the head of that file.
2021-05-12 18:41:39 -04:00
Tom Lane 1447244286 Do pre-release housekeeping on catalog data.
Run renumber_oids.pl to move high-numbered OIDs down, as per pre-beta
tasks specified by RELEASE_CHANGES.  For reference, the command was
./renumber_oids.pl --first-mapped-oid 8000 --target-oid 6150
2021-05-12 13:36:06 -04:00
Tom Lane def5b065ff Initial pgindent and pgperltidy run for v14.
Also "make reformat-dat-files".

The only change worthy of note is that pgindent messed up the formatting
of launcher.c's struct LogicalRepWorkerId, which led me to notice that
that struct wasn't used at all anymore, so I just took it out.
2021-05-12 13:14:10 -04:00
Michael Paquier e6ccd1ce16 Simplify one use of ScanKey in pg_subscription.c
The section of the code in charge of returning all the relations
associated to a subscription only need one ScanKey, but allocated two of
them.  This code was introduced as a copy-paste from a different area on
the same file by 7c4f524, making the result confusing to follow.

Author: Peter Smith
Reviewed-by: Tom Lane, Julien Rouhaud, Bharath Rupireddy
Discussion: https://postgr.es/m/CAHut+PsLKe+rN3FjchoJsd76rx2aMsFTB7CTFxRgUP05p=kcpQ@mail.gmail.com
2021-05-12 14:54:02 +09:00
Peter Eisentraut ec6e70c79f Refactor some error messages for easier translation 2021-05-12 07:42:51 +02:00
Etsuro Fujita a363bc6da9 Fix EXPLAIN ANALYZE for async-capable nodes.
EXPLAIN ANALYZE for an async-capable ForeignScan node associated with
postgres_fdw is done just by using instrumentation for ExecProcNode()
called from the node's callbacks, causing the following problems:

1) If the remote table to scan is empty, the node is incorrectly
   considered as "never executed" by the command even if the node is
   executed, as ExecProcNode() isn't called from the node's callbacks at
   all in that case.
2) The command fails to collect timings for things other than
   ExecProcNode() done in the node, such as creating a cursor for the
   node's remote query.

To fix these problems, add instrumentation for async-capable nodes, and
modify postgres_fdw accordingly.

My oversight in commit 27e1f1456.

While at it, update a comment for the AsyncRequest struct in execnodes.h
and the documentation for the ForeignAsyncRequest API in fdwhandler.sgml
to match the code in ExecAsyncAppendResponse() in nodeAppend.c, and fix
typos in comments in nodeAppend.c.

Per report from Andrey Lepikhov, though I didn't use his patch.

Reviewed-by: Andrey Lepikhov
Discussion: https://postgr.es/m/2eb662bb-105d-fc20-7412-2f027cc3ca72%40postgrespro.ru
2021-05-12 14:00:00 +09:00
Tom Lane e135743ef0 Reduce runtime of privileges.sql test under CLOBBER_CACHE_ALWAYS.
Several queries in the privileges regression test cause the planner
to apply the plpgsql function "leak()" to every element of the
histogram for atest12.b.  Since commit 0c882e52a increased the size
of that histogram to 10000 entries, the test invokes that function
over 100000 times, which takes an absolutely unreasonable amount of
time in clobber-cache-always mode.

However, there's no real reason why that has to be a plpgsql
function: for the purposes of this test, all that matters is that
it not be marked leakproof.  So we can replace the plpgsql
implementation with a direct call of int4lt, which has the same
behavior and is orders of magnitude faster.  This is expected to
cut several hours off the buildfarm cycle time for CCA animals.
It has some positive impact in normal builds too, though that's
probably lost in the noise.

Back-patch to v13 where 0c882e52a came in.

Discussion: https://postgr.es/m/575884.1620626638@sss.pgh.pa.us
2021-05-11 20:59:58 -04:00
Fujii Masao d780d7c088 Change data type of counters in BufferUsage and WalUsage from long to int64.
Previously long was used as the data type for some counters in BufferUsage
and WalUsage. But long is only four byte, e.g., on Windows, and it's entirely
possible to wrap a four byte counter. For example, emitting more than
four billion WAL records in one transaction isn't actually particularly rare.

To avoid the overflows of those counters, this commit changes the data type
of them from long to int64.

Suggested-by: Andres Freund
Author: Masahiro Ikeda
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/20201221211650.k7b53tcnadrciqjo@alap3.anarazel.de
Discussion: https://postgr.es/m/af0964ac-7080-1984-dc23-513754987716@oss.nttdata.com
2021-05-12 09:56:34 +09:00
Andrew Dunstan 0bf62931ca
Tweak generation of Gen_dummy_probes.pl
Use a static prolog file instead of generating the prolog from the
existing perl script. Also, support generation of the file in a vpath
build.

Discussion: https://postgr.es/m/700620.1620662868@sss.pgh.pa.us
2021-05-11 20:02:02 -04:00
Tom Lane 0b85fa93e4 Fix vcregress.pl's ancient misspelling of --max-connections.
I copied the existing spelling of "--max_connections", but
that's just wrong :-(.  Evidently setting $ENV{MAX_CONNECTIONS}
has never actually worked in this script.  Given the lack of
complaints, it's probably not worth back-patching a fix.

Per buildfarm.

Discussion: https://postgr.es/m/899209.1620759506@sss.pgh.pa.us
2021-05-11 19:17:07 -04:00
Tom Lane 1df3555acc Get rid of the separate serial_schedule list of tests.
Having to maintain two lists of regression test scripts has proven
annoyingly error-prone.  We can achieve the effect of the
serial_schedule by running the parallel_schedule with
"--max_connections=1"; so do that and remove serial_schedule.

This causes cosmetic differences in the progress output, but it
doesn't seem worth restructuring pg_regress to avoid that.

Discussion: https://postgr.es/m/899209.1620759506@sss.pgh.pa.us
2021-05-11 17:52:13 -04:00
Tom Lane 6303a57309 Replace opr_sanity test's binary_coercible() function with C code.
opr_sanity's binary_coercible() function has always been meant
to match the parser's notion of binary coercibility, but it also
has always been a rather poor approximation of the parser's
real rules (as embodied in IsBinaryCoercible()).  That hasn't
bit us so far, but it's predictable that it will eventually.

It also now emerges that implementing this check in plpgsql
performs absolutely horribly in clobber-cache-always testing.
(Perhaps we could do something about that, but I suspect it just
means that plpgsql is exploiting catalog caching to the hilt.)

Hence, let's replace binary_coercible() with a C shim that directly
invokes IsBinaryCoercible(), eliminating both the semantic hazard
and the performance issue.

Most of regress.c's C functions are declared in create_function_1,
but we can't simply move that to before opr_sanity/type_sanity
since those tests would complain about the resulting shell types.
I chose to split it into create_function_0 and create_function_1.
Since create_function_0 now runs as part of a parallel group while
create_function_1 doesn't, reduce the latter to create just those
functions that opr_sanity and type_sanity would whine about.

To make room for create_function_0 in the second parallel group
of tests, move tstypes to the third parallel group.

In passing, clean up some ordering deviations between
parallel_schedule and serial_schedule.

Discussion: https://postgr.es/m/292305.1620503097@sss.pgh.pa.us
2021-05-11 14:28:11 -04:00
Peter Eisentraut 6d177e2813 Fix typo 2021-05-11 09:06:49 +02:00
Michael Paquier 9ca40dcd4d Add support for LZ4 build in MSVC scripts
Since its introduction in bbe0a81, compression of table data supports
LZ4, but nothing had been done within the MSVC scripts to allow users to
build the code with this library.

This commit closes the gap by extending the MSVC scripts to be able to
build optionally with LZ4.  Getting libraries that can be used for
compilation and execution is possible as LZ4 can be compiled down to
MSVC 2010 using its source tarball.  MinGW may require extra efforts to
be able to work, and I have been able to test this only with MSVC, still
this is better than nothing to give users a way to test the feature on
Windows.

Author: Dilip Kumar
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/YJPdNeF68XpwDDki@paquier.xyz
2021-05-11 10:43:05 +09:00
Tom Lane 049e1e2edb Fix mishandling of resjunk columns in ON CONFLICT ... UPDATE tlists.
It's unusual to have any resjunk columns in an ON CONFLICT ... UPDATE
list, but it can happen when MULTIEXPR_SUBLINK SubPlans are present.
If it happens, the ON CONFLICT UPDATE code path would end up storing
tuples that include the values of the extra resjunk columns.  That's
fairly harmless in the short run, but if new columns are added to
the table then the values would become accessible, possibly leading
to malfunctions if they don't match the datatypes of the new columns.

This had escaped notice through a confluence of missing sanity checks,
including

* There's no cross-check that a tuple presented to heap_insert or
heap_update matches the table rowtype.  While it's difficult to
check that fully at reasonable cost, we can easily add assertions
that there aren't too many columns.

* The output-column-assignment cases in execExprInterp.c lacked
any sanity checks on the output column numbers, which seems like
an oversight considering there are plenty of assertion checks on
input column numbers.  Add assertions there too.

* We failed to apply nodeModifyTable's ExecCheckPlanOutput() to
the ON CONFLICT UPDATE tlist.  That wouldn't have caught this
specific error, since that function is chartered to ignore resjunk
columns; but it sure seems like a bad omission now that we've seen
this bug.

In HEAD, the right way to fix this is to make the processing of
ON CONFLICT UPDATE tlists work the same as regular UPDATE tlists
now do, that is don't add "SET x = x" entries, and use
ExecBuildUpdateProjection to evaluate the tlist and combine it with
old values of the not-set columns.  This adds a little complication
to ExecBuildUpdateProjection, but allows removal of a comparable
amount of now-dead code from the planner.

In the back branches, the most expedient solution seems to be to
(a) use an output slot for the ON CONFLICT UPDATE projection that
actually matches the target table, and then (b) invent a variant of
ExecBuildProjectionInfo that can be told to not store values resulting
from resjunk columns, so it doesn't try to store into nonexistent
columns of the output slot.  (We can't simply ignore the resjunk columns
altogether; they have to be evaluated for MULTIEXPR_SUBLINK to work.)
This works back to v10.  In 9.6, projections work much differently and
we can't cheaply give them such an option.  The 9.6 version of this
patch works by inserting a JunkFilter when it's necessary to get rid
of resjunk columns.

In addition, v11 and up have the reverse problem when trying to
perform ON CONFLICT UPDATE on a partitioned table.  Through a
further oversight, adjust_partition_tlist() discarded resjunk columns
when re-ordering the ON CONFLICT UPDATE tlist to match a partition.
This accidentally prevented the storing-bogus-tuples problem, but
at the cost that MULTIEXPR_SUBLINK cases didn't work, typically
crashing if more than one row has to be updated.  Fix by preserving
resjunk columns in that routine.  (I failed to resist the temptation
to add more assertions there too, and to do some minor code
beautification.)

Per report from Andres Freund.  Back-patch to all supported branches.

Security: CVE-2021-32028
2021-05-10 11:02:29 -04:00
Tom Lane f02b9085ad Prevent integer overflows in array subscripting calculations.
While we were (mostly) careful about ensuring that the dimensions of
arrays aren't large enough to cause integer overflow, the lower bound
values were generally not checked.  This allows situations where
lower_bound + dimension overflows an integer.  It seems that that's
harmless so far as array reading is concerned, except that array
elements with subscripts notionally exceeding INT_MAX are inaccessible.
However, it confuses various array-assignment logic, resulting in a
potential for memory stomps.

Fix by adding checks that array lower bounds aren't large enough to
cause lower_bound + dimension to overflow.  (Note: this results in
disallowing cases where the last subscript position would be exactly
INT_MAX.  In principle we could probably allow that, but there's a lot
of code that computes lower_bound + dimension and would need adjustment.
It seems doubtful that it's worth the trouble/risk to allow it.)

Somewhat independently of that, array_set_element() was careless
about possible overflow when checking the subscript of a fixed-length
array, creating a different route to memory stomps.  Fix that too.

Security: CVE-2021-32027
2021-05-10 10:44:38 -04:00
Peter Eisentraut 6206454bda Translation updates
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 1c361d3ac016b61715d99f2055dee050397e3f13
2021-05-10 14:36:21 +02:00
Peter Eisentraut fa8fbadb93 Emit dummy statements for probes.d probes when disabled
When building without --enable-dtrace, emit dummy

    do {} while (0)

statements for the stubbed-out TRACE_POSTGRESQL_foo() macros
instead of empty macros that totally elide the original probe
statement.

This fixes the

    warning: suggest braces around empty body in an ‘if’ statement [-Wempty-body]

introduced by b94409a02f.

Author: Craig Ringer <craig.ringer@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/20210504221531.cfvpmmdfsou6eitb%40alap3.anarazel.de
2021-05-10 11:40:03 +02:00
Peter Eisentraut 3c55410030 Remove unused function arguments
Was present in original commit
198b3716db but apparently never used.
2021-05-10 10:02:33 +02:00
Michael Paquier 829daab4bb Fix typos in operatorcmds.c
Author: Kyotaro Horiguchi, Justin Pryzby
Discussion: https://postgr.es/m/20210428.173633.1525059946206239295.horikyota.ntt@gmail.com
2021-05-10 15:45:54 +09:00
Thomas Munro c2dc19342e Revert recovery prefetching feature.
This set of commits has some bugs with known fixes, but at this late
stage in the release cycle it seems best to revert and resubmit next
time, along with some new automated test coverage for this whole area.

Commits reverted:

dc88460c: Doc: Review for "Optionally prefetch referenced data in recovery."
1d257577: Optionally prefetch referenced data in recovery.
f003d9f8: Add circular WAL decoding buffer.
323cbe7c: Remove read_page callback from XLogReader.

Remove the new GUC group WAL_RECOVERY recently added by a55a9847, as the
corresponding section of config.sgml is now reverted.

Discussion: https://postgr.es/m/CAOuzzgrn7iKnFRsB4MHp3UisEQAGgZMbk_ViTN4HV4-Ksq8zCg%40mail.gmail.com
2021-05-10 16:06:09 +12:00
Michael Paquier 63db0ac3f9 Add more TAP tests for pg_dump with attribute compression
Some relations with LZ4 used as the toast compression methods have been
left around in the main regression test suite to stress pg_upgrade, but
pg_dump, that includes tests much more picky in terms of output
generated, had no actual coverage with this feature.

Similarly to collations, tests only working with LZ4 are tracked with an
additional flag, and this uses TestLib::check_pg_config() to check if
the build supports LZ4 or not.  This stresses more scenarios with
tables, materialized views and pg_dump --no-toast-compression.

Author: Dilip Kumar
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/CAFiTN-twgPmohG7qj1HXhySq16h123y5OowsQR+5h1YeZE9fmQ@mail.gmail.com
2021-05-10 11:12:07 +09:00
Tom Lane 8dc3d68cbe Improve comments about USE_VALGRIND in pg_config_manual.h.
These comments left the impression that USE_VALGRIND isn't really
essential for valgrind testing.  But that's wrong, as I learned
the hard way today.

Discussion: https://postgr.es/m/512778.1620588546@sss.pgh.pa.us
2021-05-09 19:33:24 -04:00