Commit Graph

4500 Commits

Author SHA1 Message Date
Tom Lane 991a3df227 Fix filtering of "cloned" outer-join quals some more.
We've had multiple issues with the clause_is_computable_at logic that
I introduced in 2489d76c4: it's been known to accept more than one
clone of the same qual at the same plan node, and also to accept no
clones at all.  It's looking impractical to get it 100% right on the
basis of the currently-stored information, so fix it by introducing a
new RestrictInfo field "incompatible_relids" that explicitly shows
which outer joins a given clone mustn't be pushed above.

In principle we could populate this field in every RestrictInfo, but
that would cost space and there doesn't presently seem to be a need
for it in general.  Also, while deconstruct_distribute_oj_quals can
easily fill the field with the remaining members of the commutative
join set that it's considering, computing it in the general case
seems again pretty complicated.  So for now, just fill it for
clone quals.

Along the way, fix a bug that may or may not be only latent:
equivclass.c was generating replacement clauses with is_pushed_down
and has_clone/is_clone markings that didn't match their
required_relids.  This led me to conclude that leaving the clone flags
out of make_restrictinfo's purview wasn't such a great idea after all,
so add them.

Per report from Richard Guo.

Discussion: https://postgr.es/m/CAMbWs48EYi_9-pSd0ORes1kTmTeAjT4Q3gu49hJtYCbSn2JyeA@mail.gmail.com
2023-05-25 10:28:33 -04:00
Tom Lane a2eb99a01e Expand some more uses of "deleg" to "delegation" or "delegated".
Complete the task begun in 9c0a0e2ed: we don't want to use the
abbreviation "deleg" for GSS delegation in any user-visible places.
(For consistency, this also changes most internal uses too.)

Abhijit Menon-Sen and Tom Lane

Discussion: https://postgr.es/m/949048.1684639317@sss.pgh.pa.us
2023-05-21 10:55:18 -04:00
Bruce Momjian 9c0a0e2ed9 rename "gss_accept_deleg" to "gss_accept_delegation".
This is more consistent with existing GUC spelling.

Discussion: https://postgr.es/m/ZGdnEsGtNj7+fZoa@momjian.us
2023-05-20 21:32:54 -04:00
Tom Lane 0245f8db36 Pre-beta mechanical code beautification.
Run pgindent, pgperltidy, and reformat-dat-files.

This set of diffs is a bit larger than typical.  We've updated to
pg_bsd_indent 2.1.2, which properly indents variable declarations that
have multi-line initialization expressions (the continuation lines are
now indented one tab stop).  We've also updated to perltidy version
20230309 and changed some of its settings, which reduces its desire to
add whitespace to lines to make assignments etc. line up.  Going
forward, that should make for fewer random-seeming changes to existing
code.

Discussion: https://postgr.es/m/20230428092545.qfb3y5wcu4cm75ur@alvherre.pgsql
2023-05-19 17:24:48 -04:00
Michael Paquier e7bff46e50 pageinspect: Fix gist_page_items() with included columns
Non-leaf pages of GiST indexes contain key attributes, leaf pages
contain both key and non-key attributes, and gist_page_items() ignored
the handling of non-key attributes.  This caused a few problems when
using gist_page_items() on a GiST index with INCLUDE:
- On a non-leaf page, the function would crash.
- On a leaf page, the function would work, but miss to display all the
values for included attributes.

This commit fixes gist_page_items() to handle such cases in a more
appropriate way, and now displays the values of key and non-key
attributes for each item separately in a style consistent with what
ruleutils.c would generate for the attribute list, depending on the page
type dealt with.  In a way similar to how a record is displayed, values
would be double-quoted for key or non-key attributes if required.

ruleutils.c did not provide a routine able to control if non-key
attributes should be displayed, so an extended() routine for index
definitions is added to work around the leaf and non-leaf page
differences.

While on it, this commit fixes a third problem related to the amount of
data reported for key attributes.  The code originally relied on
BuildIndexValueDescription() (used for error reports on constraints)
that would not print all the data stored in the index but the index
opclass's input type, so this limited the amount of information
available.  This switch makes gist_page_items() much cheaper as there is
no need to run ACL checks for each item printed, which is not an issue
anyway as superuser rights are required to execute the functions of
pageinspect.  Opclasses whose data cannot be displayed can rely on
gist_page_items_bytea().

The documentation of this function was slightly incorrect for the
output results generated on HEAD and v15, so adjust it on these
branches.

Author: Alexander Lakhin, Michael Paquier
Discussion: https://postgr.es/m/17884-cb8c326522977acb@postgresql.org
Backpatch-through: 14
2023-05-19 12:37:58 +09:00
Tomas Vondra 428c0cae92 Show empty BRIN ranges in brin_page_items
Commit 3581cbdcd6 added a flag to identify empty BRIN ranges. This adds
the new flag to brin_page_items() output.

This is kept as a separate commit as it should not be backpatched.

Reviewed-by: Justin Pryzby, Matthias van de Meent, Alvaro Herrera
Discussion: https://postgr.es/m/402430e4-7d9d-6cf1-09ef-464d80afff3b@enterprisedb.com
2023-05-19 02:00:21 +02:00
Tom Lane 1d369c9e90 Ensure Soundex difference() function handles empty input sanely.
fuzzystrmatch's difference() function assumes that _soundex()
always initializes its output buffer fully.  This was not so for
the case of a string containing no alphabetic characters, resulting
in unstable output and Valgrind complaints.

Fix by using memset() to fill the whole buffer in the early-exit
case.  Also make some cosmetic improvements (I didn't care for the
random switches between "instr[0]" and "*instr" notation).

Report and diagnosis by Alexander Lakhin (bug #17935).
Back-patch to all supported branches.

Discussion: https://postgr.es/m/17935-b99316aa79c18513@postgresql.org
2023-05-16 10:53:42 -04:00
Tom Lane 8d525d7b95 Adjust sepgsql expected output for 681d9e462 et al.
Security: CVE-2023-2454
2023-05-08 11:24:47 -04:00
Noah Misch 681d9e4621 Replace last PushOverrideSearchPath() call with set_config_option().
The two methods don't cooperate, so set_config_option("search_path",
...) has been ineffective under non-empty overrideStack.  This defect
enabled an attacker having database-level CREATE privilege to execute
arbitrary code as the bootstrap superuser.  While that particular attack
requires v13+ for the trusted extension attribute, other attacks are
feasible in all supported versions.

Standardize on the combination of NewGUCNestLevel() and
set_config_option("search_path", ...).  It is newer than
PushOverrideSearchPath(), more-prevalent, and has no known
disadvantages.  The "override" mechanism remains for now, for
compatibility with out-of-tree code.  Users should update such code,
which likely suffers from the same sort of vulnerability closed here.
Back-patch to v11 (all supported versions).

Alexander Lakhin.  Reported by Alexander Lakhin.

Security: CVE-2023-2454
2023-05-08 06:14:07 -07:00
Alvaro Herrera 95f5a7a42c
Fix sepgsql expected test output
Commit f75cec4fff changed the order in which the relations are
permission-checked in RI_Initial_Check, which the sepgsql test is
sensitive to.  Adapt.

Discussion: https://postgr.es/m/3468125.1683238309@sss.pgh.pa.us
2023-05-05 11:10:25 +02:00
Michael Paquier 8961cb9a03 Fix typos in comments
The changes done in this commit impact comments with no direct
user-visible changes, with fixes for incorrect function, variable or
structure names.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/e8c38840-596a-83d6-bd8d-cebc51111572@gmail.com
2023-05-02 12:23:08 +09:00
Tom Lane df38157d94 In hstore_plpython, avoid crashing when return value isn't a mapping.
Python 3 changed the behavior of PyMapping_Check(), breaking the
test in plpython_to_hstore() that verifies whether a function result
to be transformed is acceptable.  A backwards-compatible fix is to
first verify that the object doesn't pass PySequence_Check().

Perhaps accidentally, our other uses of PyMapping_Check() already
follow uses of PySequence_Check(), so that no other bugs were
created by this change.

Per bug #17908 from Alexander Lakhin.  Back-patch to all supported
branches.

Dmitry Dolgov and Tom Lane

Discussion: https://postgr.es/m/17908-3f19a125d56a11d6@postgresql.org
2023-04-27 11:55:06 -04:00
Michael Paquier 806fad7573 Fix buffer refcount leak with FDW bulk inserts
The leak would show up when using batch inserts with foreign tables
included in a partition tree, as the slots used in the batch were not
reset once processed.  In order to fix this problem, some
ExecClearTuple() are added to clean up the slots used once a batch is
filled and processed, mapping with the number of slots currently in use
as tracked by the counter ri_NumSlots.

This buffer refcount leak has been introduced in b676ac4 with the
addition of the executor facility to improve bulk inserts for FDWs, so
backpatch down to 14.

Alexander has provided the patch (slightly modified by me).  The test
for postgres_fdw comes from me, based on the test case that the author
has sent in the report.

Author: Alexander Pyhalov
Discussion: https://postgr.es/m/b035780a740efd38dc30790c76927255@postgrespro.ru
Backpatch-through: 14
2023-04-25 09:42:19 +09:00
Alexander Korotkov 8bbd0cce92 Validate ltree siglen GiST option to be int-aligned
Unaligned siglen could lead to an unaligned access to subsequent key fields.

Backpatch to 13, where opclass options were introduced.

Reported-by: Alexander Lakhin
Bug: 17847
Discussion: https://postgr.es/m/17847-171232970bea406b%40postgresql.org
Reviewed-by: Tom Lane, Pavel Borisov, Alexander Lakhin
Backpatch-through: 13
2023-04-23 14:30:09 +03:00
Michael Paquier 0ecb87e1fa Remove io prefix from pg_stat_io columns
a9c70b46 added the statistics view pg_stat_io which contained columns
"io_context" and "io_object".  Given that the columns are in the
pg_stat_io view, the "io" prefix is somewhat redundant, so remove it.

The code variables referring to these fields are kept unchanged so as
they can keep their context about I/O.

Bump catalog version.

Author: Melanie Plageman
Reviewed-by: Kyotaro Horiguchi, Fabrízio de Royes Mello
Discussion: https://postgr.es/m/CAAKRu_aAQoJWrvT2BYYQvJChFKra_O-5ra3jhzKJZqWsTR1CPQ@mail.gmail.com
2023-04-21 07:21:50 +09:00
David Rowley 3f58a4e296 Fix various typos and incorrect/outdated name references
Author: Alexander Lakhin
Discussion: https://postgr.es/m/699beab4-a6ca-92c9-f152-f559caf6dc25@gmail.com
2023-04-19 13:50:33 +12:00
David Rowley eef231e816 Fix some typos and some incorrectly duplicated words
Author: Justin Pryzby
Reviewed-by: David Rowley
Discussion: https://postgr.es/m/ZD3D1QxoccnN8A1V@telsasoft.com
2023-04-18 14:03:49 +12:00
David Rowley b4dbf3e924 Fix various typos
This fixes many spelling mistakes in comments, but a few references to
invalid parameter names, function names and option names too in comments
and also some in string constants

Also, fix an #undef that was undefining the incorrect definition

Author: Alexander Lakhin
Reviewed-by: Justin Pryzby
Discussion: https://postgr.es/m/d5f68d19-c0fc-91a9-118d-7c6a5a3f5fad@gmail.com
2023-04-18 13:23:23 +12:00
Tom Lane 1c54b93a8c Avoid using non-ASCII commentary in daitch_mokotoff.c.
Commit d6b5dee42 failed to suppress that warning from MSVC,
so let's just get rid of all non-ASCII glyphs in the comments.
The comment for iso8859_1_to_ascii_upper[] is not essential,
and the other cases can be handled by spelling out the
Unicode character names.

(I'm now really in the dark as to why MSVC doesn't complain
about predicate.c's comment.  But whatever.)

Discussion: https://postgr.es/m/1546512.1681495035@sss.pgh.pa.us
2023-04-16 12:57:32 -04:00
Tom Lane d6b5dee42d Remove some non-ASCII symbols from a comment.
Buildfarm member hamerkop is unhappy that daitch_mokotoff.c "contains
a character that cannot be represented in the current code page
(932). Save the file in Unicode format to prevent data loss".  There
are a lot of non-ASCII characters in comments, but I do not see any
outside a comment, so this is just cosmetic.  Nonetheless the warning
is fairly scary and people might waste time trying to understand the
cause, so we ought to silence it.

We have only one other occurrence of a non-ASCII character in .c or .h
files in the tree: predicate.c mentions "Uwe Röhm".  It's far from
clear why hamerkop isn't complaining about that too; but it isn't,
so maybe there is some special provision for accented Roman letters.
We can remove the non-letter symbols from the comment for
iso8859_1_to_ascii_upper without any real loss of usefulness,
so let's try that first to see if it silences the warning.

Discussion: https://postgr.es/m/1546512.1681495035@sss.pgh.pa.us
2023-04-15 11:20:46 -04:00
Stephen Frost 6633cfb216 De-Revert "Add support for Kerberos credential delegation"
This reverts commit 3d03b24c3 (Revert Add support for Kerberos
credential delegation) which was committed on the grounds of concern
about portability, but on further review and discussion, it's clear that
we are better off explicitly requiring MIT Kerberos as that appears to
be the only GSSAPI library currently that's under proper maintenance
and ongoing development.  The API used for storing credentials was added
to MIT Kerberos over a decade ago while for the other libraries which
appear to be mainly based on Heimdal, which exists explicitly to be a
re-implementation of MIT Kerberos, the API never made it to a released
version (even though it was added to the Heimdal git repo over 5 years
ago..).

This post-feature-freeze change was approved by the RMT.

Discussion: https://postgr.es/m/ZDDO6jaESKaBgej0%40tamriel.snowman.net
2023-04-13 08:55:07 -04:00
Alvaro Herrera 21e6bf4ddd
Revert "Adjust contrib/sepgsql regression test expected outputs."
This reverts commit 76c111a7f166; should have been included in
9ce04b50e1.

Noted by Joe Conway
2023-04-12 20:22:17 +02:00
Robert Haas 8e82db97b0 basebackup_to_shell: Check for a NULL return from OpenPipeStream.
Per complaint from Peter Eisentraut.

Discussion: http://postgr.es/m/4f1707cc-2432-da35-64a2-5c2a8d92a388@enterprisedb.com
2023-04-12 11:37:13 -04:00
Stephen Frost 3d03b24c35 Revert "Add support for Kerberos credential delegation"
This reverts commit 3d4fa227bc.

Per discussion and buildfarm, this depends on APIs that seem to not
be available on at least one platform (NetBSD).  Should be certainly
possible to rework to be optional on that platform if necessary but bit
late for that at this point.

Discussion: https://postgr.es/m/3286097.1680922218@sss.pgh.pa.us
2023-04-08 07:21:35 -04:00
Thomas Munro 6db75edb2e Update contrib/trgm_regexp's memory management.
While no code change was necessary for this code to keep working, we
don't need to use PG_TRY()/PG_FINALLY() with explicit clean-up while
working with regexes anymore.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com
2023-04-08 22:09:17 +12:00
Thomas Munro faeedbcefd Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.
In order to have the option to use O_DIRECT/FILE_FLAG_NO_BUFFERING in a
later commit, we need the addresses of user space buffers to be well
aligned.  The exact requirements vary by OS and file system (typically
sectors and/or memory pages).  The address alignment size is set to
4096, which is enough for currently known systems: it matches modern
sectors and common memory page size.  There is no standard governing
O_DIRECT's requirements so we might eventually have to reconsider this
with more information from the field or future systems.

Aligning I/O buffers on memory pages is also known to improve regular
buffered I/O performance.

Three classes of I/O buffers for regular data pages are adjusted:
(1) Heap buffers are now allocated with the new palloc_aligned() or
MemoryContextAllocAligned() functions introduced by commit 439f6175.
(2) Stack buffers now use a new struct PGIOAlignedBlock to respect
PG_IO_ALIGN_SIZE, if possible with this compiler.  (3) The buffer
pool is also aligned in shared memory.

WAL buffers were already aligned on XLOG_BLCKSZ.  It's possible for
XLOG_BLCKSZ to be configured smaller than PG_IO_ALIGNED_SIZE and thus
for O_DIRECT WAL writes to fail to be well aligned, but that's a
pre-existing condition and will be addressed by a later commit.

BufFiles are not yet addressed (there's no current plan to use O_DIRECT
for those, but they could potentially get some incidental speedup even
in plain buffered I/O operations through better alignment).

If we can't align stack objects suitably using the compiler extensions
we know about, we disable the use of O_DIRECT by setting PG_O_DIRECT to
0.  This avoids the need to consider systems that have O_DIRECT but
can't align stack objects the way we want; such systems could in theory
be supported with more work but we don't currently know of any such
machines, so it's easier to pretend there is no O_DIRECT support
instead.  That's an existing and tested class of system.

Add assertions that all buffers passed into smgrread(), smgrwrite() and
smgrextend() are correctly aligned, unless PG_O_DIRECT is 0 (= stack
alignment tricks may be unavailable) or the block size has been set too
small to allow arrays of buffers to be all aligned.

Author: Thomas Munro <thomas.munro@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/CA+hUKGK1X532hYqJ_MzFWt0n1zt8trz980D79WbjwnT-yYLZpg@mail.gmail.com
2023-04-08 16:34:50 +12:00
Tom Lane 2bfbad9c42 Remove useless dependencies in daitch_mokotoff_header.pl.
Actually, the correct fix for this is "we don't need this at all",
because this program isn't dealing in any non-ASCII data.  The
dependency on Data::Dumper seems to be a leftover too.

Discussion: https://postgr.es/m/3287943.1680922997@sss.pgh.pa.us
2023-04-07 23:23:25 -04:00
Stephen Frost 3d4fa227bc Add support for Kerberos credential delegation
Support GSSAPI/Kerberos credentials being delegated to the server by a
client.  With this, a user authenticating to PostgreSQL using Kerberos
(GSSAPI) credentials can choose to delegate their credentials to the
PostgreSQL server (which can choose to accept them, or not), allowing
the server to then use those delegated credentials to connect to
another service, such as with postgres_fdw or dblink or theoretically
any other service which is able to be authenticated using Kerberos.

Both postgres_fdw and dblink are changed to allow non-superuser
password-less connections but only when GSSAPI credentials have been
delegated to the server by the client and GSSAPI is used to
authenticate to the remote system.

Authors: Stephen Frost, Peifeng Qiu
Reviewed-By: David Christensen
Discussion: https://postgr.es/m/CO1PR05MB8023CC2CB575E0FAAD7DF4F8A8E29@CO1PR05MB8023.namprd05.prod.outlook.com
2023-04-07 21:58:04 -04:00
Tom Lane edc627ae27 Pacify perlcritic.
Discussion: https://postgr.es/m/3271512.1680916423@sss.pgh.pa.us
2023-04-07 21:33:05 -04:00
Tom Lane 76c111a7f1 Adjust contrib/sepgsql regression test expected outputs.
Per buildfarm, the log output has changed as a consequence of
commit e056c557a changing the catalog accesses performed in
some commands.

Discussion: https://postgr.es/m/20230407211950.aojbdirwdrxjeyzb@awork3.anarazel.de
2023-04-07 18:17:22 -04:00
Tom Lane a290378a37 Add support for Daitch-Mokotoff Soundex in contrib/fuzzystrmatch.
This modernized version of Soundex works significantly better than
the original, particularly for non-English names.

Dag Lem, reviewed by quite a few people along the way

Discussion: https://postgr.es/m/yger1atbgfy.fsf@sid.nimrod.no
2023-04-07 17:32:26 -04:00
Daniel Gustafsson 664d757531 Refactor background psql TAP functions
This breaks out the background and interactive psql functionality into a
new class, PostgreSQL::Test::BackgroundPsql.  Sessions are still initiated
via PostgreSQL::Test::Cluster, but once started they can be manipulated by
the new helper functions which intend to make querying easier.  A sample
session for a command which can be expected to finish at a later time can
be seen below.

  my $session = $node->background_psql('postgres');
  $bsession->query_until(qr/start/, q(
    \echo start
	CREATE INDEX CONCURRENTLY idx ON t(a);
  ));
  $bsession->quit;

Patch by Andres Freund with some additional hacking by me.

Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
2023-04-07 22:14:20 +02:00
Tom Lane f3fa31327e Add pg_buffercache_usage_counts() to contrib/pg_buffercache.
It was pointed out that pg_buffercache_summary()'s report of
the overall average usage count isn't that useful, and what
would be more helpful in many cases is to report totals for
each possible usage count.  Add a new function to do it like
that.  Since pg_buffercache 1.4 is already new for v16,
we don't need to create a new extension version; we'll just
define this as part of 1.4.

Nathan Bossart

Discussion: https://postgr.es/m/20230130233040.GA2800702@nathanxps13
2023-04-07 14:25:53 -04:00
Etsuro Fujita 983ec23007 postgres_fdw: Add support for parallel abort.
postgres_fdw aborts remote (sub)transactions opened on remote server(s)
in a local (sub)transaction one by one when the local (sub)transaction
aborts.  This patch allows it to abort the remote (sub)transactions in
parallel to improve performance.  This is enabled by the server option
"parallel_abort".  The default is false.

Etsuro Fujita, reviewed by David Zhang.

Discussion: http://postgr.es/m/CAPmGK15FuPVGx3TGHKShsbPKKtF1y58-ZLcKoxfN-nqLj1dZ%3Dg%40mail.gmail.com
2023-04-06 17:30:00 +09:00
Andres Freund acab1b0914 Convert many uses of ReadBuffer[Extended](P_NEW) to ExtendBufferedRel()
A few places are not converted. Some because they are tackled in later
commits (e.g. hio.c, xlogutils.c), some because they are more
complicated (e.g. brin_pageops.c).  Having a few users of ReadBuffer(P_NEW) is
good anyway, to ensure the backward compat path stays working.

Discussion: https://postgr.es/m/20221029025420.eplyow6k7tgu6he3@awork3.anarazel.de
2023-04-05 18:57:29 -07:00
Michael Paquier 1d477a907e Fix row tracking in pg_stat_statements with extended query protocol
pg_stat_statements relies on EState->es_processed to count the number of
rows processed by ExecutorRun().  This proves to be a problem under the
extended query protocol when the result of a query is fetched through
more than one call of ExecutorRun(), as es_processed is reset each time
ExecutorRun() is called.  This causes pg_stat_statements to report the
number of rows calculated in the last execute fetch, rather than the
global sum of all the rows processed.

As pquery.c tells, this is a problem when a portal does not use
holdStore.  For example, DMLs with RETURNING would report a correct
tuple count as these do one execution cycle when the query is first
executed to fill in the portal's store with one ExecutorRun(), feeding
on the portal's store for each follow-up execute fetch depending on the
fetch size requested by the client.

The fix proposed for this issue is simple with the addition of an extra
counter in EState that's preserved across multiple ExecutorRun() calls,
incremented with the value calculated in es_processed.  This approach is
not back-patchable, unfortunately.

Note that libpq does not currently give any way to control the fetch
size when using the extended v3 protocol, meaning that in-core testing
is not possible yet.  This issue can be easily verified with the JDBC
driver, though, with *autocommit disabled*.  Hence, having in-core tests
requires more features, left for future discussion:
- At least two new libpq routines splitting PQsendQueryGuts(), one for
the bind/describe and a second for a series of execute fetches with a
custom fetch size, likely in a fashion similar to what JDBC does.
- A psql meta-command for the execute phase.  This part is not strictly
mandatory, still it could be handy.

Reported-by: Andrew Dunstan (original discovery by Simon Siggs)
Author: Sami Imseih
Reviewed-by: Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/EBE6C507-9EB6-4142-9E4D-38B1673363A7@amazon.com
Discussion: https://postgr.es/m/c90890e7-9c89-c34f-d3c5-d5c763a34bd8@dunslane.net
2023-04-06 09:29:03 +09:00
Daniel Gustafsson fb6fad6ef1 Fix function reference in comment
Commit a61b1f748 renamed ExecCheckRTEPerms to ExecCheckPermissions
as part of a larger body of work, but missed this comment.  Fix by
updating the referenced function name to make the comment the same
as other occurrences.

Author: Koshi Shibagaki <shibagaki.koshi@fujitsu.com>
Discussion: https://postgr.es/m/OS3PR01MB653359ACBE8DBBE29EE2BC71FA909@OS3PR01MB6533.jpnprd01.prod.outlook.com
2023-04-05 09:06:32 +02:00
Noah Misch ab73291d26 Fix copy-pasto in contrib/auth_delay/meson.build variable name. 2023-04-02 09:31:10 -07:00
Andres Freund 61b313e47e Pass down table relation into more index relation functions
This is done in preparation for logical decoding on standby, which needs to
include whether visibility affecting WAL records are about a (user) catalog
table. Which is only known for the table, not the indexes.

It's also nice to be able to pass the heap relation to GlobalVisTestFor() in
vacuumRedirectAndPlaceholder().

Author: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
Discussion: https://postgr.es/m/21b700c3-eecf-2e05-a699-f8c78dd31ec7@gmail.com
2023-04-01 20:18:29 -07:00
Peter Geoghegan df4f3ab517 Add show_data option to pg_get_wal_block_info.
Allow users to opt out of returning FPI data and block data from
pg_get_wal_block_info as an optimization.  Testing has shown that this
can make function execution over twice as fast in some cases.

When pg_get_wal_block_info is called with "show_data := false", it
always returns NULL values for its block_data and block_fpi_data bytea
output parameters.  Nothing else changes.  In particular, the function
will still return the usual per-block summary of block data/FPI space
overhead.  Use of "show_data := false" is therefore feasible with all
queries that don't specifically require these raw binary strings.

Follow-up to recent work in commit 122376f0.  There still hasn't been a
stable release with the pg_get_wal_block_info function, so no bump in
the pg_walinspect extension version.

Per suggestion from Melanie Plageman.

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAAKRu_bJvbcYBRj2cN6G2xV7B7-Ja+pjTO1nEnEhRR8OXYiABA@mail.gmail.com
Discussion: https://postgr.es/m/CAH2-Wzm9shOkEDM10_+qOZkRSQhKVxwBFiehH6EHWQQRd_rDPw@mail.gmail.com
2023-03-31 14:02:52 -07:00
Peter Geoghegan 122376f028 Show record information in pg_get_wal_block_info.
Expand the output parameters in pg_walinspect's pg_get_wal_block_info
function to return additional information that was previously only
available from pg_walinspect's pg_get_wal_records_info function.  Some
of the details are attributed to individual block references, rather
than aggregated into whole-record values, since the function returns one
row per block reference per WAL record (unlike pg_get_wal_records_info,
which always returns one row per WAL record).

This structure is much easier to work with when writing queries that
track how individual blocks changed over time, or when attributing costs
to individual blocks (not WAL records) is useful.

This is the second time that pg_get_wal_block_info has been enhanced in
recent weeks.  Commit 9ecb134a expanded on the original version of the
function added in commit c31cf1c0 (where it first appeared under the
name pg_get_wal_fpi_info).  There still hasn't been a stable release
since commit c31cf1c0, so no bump in the pg_walinspect extension
version.

Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Kyotaro HORIGUCHI <horikyota.ntt@gmail.com>
Discussion: https://postgr.es/m/CALj2ACVRK5=Z+2ZVsjgTTSkfEnQzCuwny7iigpG7g1btk4Ws2A@mail.gmail.com
2023-03-30 12:26:12 -07:00
Robert Haas b7cea58822 amcheck: In verify_heapam, allows tuples with xmin 0.
Commit e88754a196 caused that case
to be reported as corruption, but Peter Geoghegan pointed out that
it can legitimately happen in the case of a speculative insertion
that aborts, so we'd better not flag it as corruption after all.

Back-patch to v14, like the commit that introduced the issue.

Discussion: http://postgr.es/m/CAH2-WzmEabzcPTxSY-NXKH6Qt3FkAPYHGQSe2PtvGgj17ZQkCw@mail.gmail.com
2023-03-28 16:16:53 -04:00
Peter Geoghegan bade01cb4d Fix recent pg_walinspect fpi_length bug.
Commit 0276ae42dd taught pg_walinspect's pg_get_wal_record_info()
function to output NULLs rather than empty strings for its record
description and block_ref output parameters.  However, it inadvertently
moved the function call that sets fpi_length until after it was already
set.  As a result, pg_get_wal_record_info() always output spurious
fpi_length values of 0.

Fix by switching the order back (but keep the behavioral change).

Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-WzkJmgSYkt6-smQ+57SxSmov+EKqFZdSimFewosoL_JKoA@mail.gmail.com
2023-03-28 10:53:48 -07:00
Peter Geoghegan 637dce2254 pg_walinspect: Adjust memory context name.
Correct the name of the memory context used by the
pg_get_wal_block_info() SQL-callable function.

Oversight in commit 9ecb134a93.
2023-03-27 16:14:32 -07:00
Robert Haas c87aff065c amcheck: Generalize one of the recently-added update chain checks.
Commit bbc1376b39 checked that if
a redirected line pointer pointed to a tuple, the tuple should be
marked both HEAP_ONLY_TUPLE and HEAP_UPDATED. But Andres Freund
pointed out that *any* tuple that is marked HEAP_ONLY_TUPLE should
be marked HEAP_UPDATED, not just one that is the target of a
redirected line pointer. Do that instead.

To see why this is better, consider a redirect line pointer A
which points to a heap-only tuple B which points (via CTID)
to another heap-only tuple C. With the old code, we'd complain
if B was not marked HEAP_UPDATED, but with this change, we'll
complain if either B or C is not marked HEAP_UPDATED.

(Note that, with or without this commit, if either B or C were
not marked HEAP_ONLY_TUPLE, we would also complain about that.)

Discussion: http://postgr.es/m/CA%2BTgmobLypZx%3DcOH%2ByY1GZmCruaoucHm77A6y_-Bo%3Dh-_3H28g%40mail.gmail.com
2023-03-27 13:37:16 -04:00
Robert Haas 80d5e3a615 amcheck: Tighten up validation of redirect line pointers.
Commit bbc1376b39 added a new lp_valid[]
array which records whether or not a line pointer was thought to be
valid, but entries could sometimes get set to true in cases where that
wasn't actually safe. Fix that.

Suppose A is a redirect line pointer and B is the other line pointer
to which it points.  The old code could mishandle this situation in a
couple of different ways.  First, if B was unused, we'd complain about
corruption but still set lp_valid[A] = true, causing later code
to try to access the B as if it were pointing to a tuple. Second,
if B was dead, we wouldn't complain about corruption at all, which is
an oversight, and would also set lp_valid[A] = true, which would
again confuse later code. Fix all that.

In the case where B is a redirect, the old code was correct, but
refactor things a bit anyway so that all of these cases are handled
more symmetrically. Also add an Assert() and some comments.

Andres Freund and Robert Haas

Discussion: http://postgr.es/m/20230323172607.y3lejpntjnuis5vv%40awork3.anarazel.de
2023-03-27 13:36:11 -04:00
Michael Paquier 0276ae42dd Improve a few things in pg_walinspect
This improves a few things in pg_walinspect:
- Return NULL rather than empty strings in pg_get_wal_records_info() for
the block references and the record description if there is no
information provided by the fallback.  This point has been raised by
Peter Geoghegan.
- Add a check on XLogRecHasAnyBlockRefs() for pg_get_wal_block_info(),
to directly skip records that have no block references.  This speeds up
the function a bit, depending on the number of records that have no
block references.

Author: Bharath Rupireddy
Reviewed-by: Kyotaro Horiguchi, Michael Paquier
Discussion: https://postgr.es/m/CALj2ACWL9RG8sGJHinggRNBTxgRWJTSxCkB+cE6=t3Phh=Ey+A@mail.gmail.com
2023-03-27 13:15:04 +09:00
Robert Haas e88754a196 amcheck: Fix verify_heapam for tuples where xmin or xmax is 0.
In such cases, get_xid_status() doesn't set its output parameter (the
third argument), so we shouldn't fall through to code which will test
the value of that parameter. There are five existing calls to
get_xid_status(), three of which seem to already handle this case
properly.  This commit tries to fix the other two.

If we're checking xmin and find that it is invalid (i.e. 0) just
report that as corruption, similar to what's already done in the
three cases that seem correct. If we're checking xmax and find
that's invalid, that's fine: it just means that the tuple hasn't
been updated or deleted.

Thanks to Andres Freund and valgrind for finding this problem, and
also to Andres for having a look at the patch.  This bug seems to go
all the way back to where verify_heapam was first introduced, but
wasn't detected until recently, possibly because of the new test cases
added for update chain verification.  Back-patch to v14, where this
code showed up.

Discussion: http://postgr.es/m/CA+TgmoZAYzQZqyUparXy_ks3OEOfLD9-bEXt8N-2tS1qghX9gQ@mail.gmail.com
2023-03-24 10:56:10 -04:00
Robert Haas 949e2e7c4f amcheck: Fix a few bugs in new update chain validation.
We shouldn't set successor[whatever] to an offset number that is less
than FirstOffsetNumber or more than maxoff. We already avoided that
for redirects, but not for CTID links. Allowing bad offset numbers
into the successor[] array causes core dumps.

We shouldn't use HeapTupleHeaderIsHotUpdated() because it checks
stuff other than the status of the infomask2 bit HEAP_HOT_UPDATED.
We only care about the status of that bit, not the other stuff
that HeapTupleHeaderIsHotUpdated() checks. This mistake can cause
verify_heapam() to report corruption when none is present.

The first hunk of this patch was written by me. The other two were
written by Andres Freund. This could probably do with more review
before commit, but I'd like to try to get the buildfarm green again
sooner rather than later.

Discussion: http://postgr.es/m/20230322204552.s6cv3ybqkklhhybb@awork3.anarazel.de
2023-03-23 13:03:03 -04:00
Michael Paquier 4f1882b960 Improve a bit the tests of pg_walinspect
This commit improves the tests of pg_walinspect on a few things:
- Remove aggregates for queries that should fail.  If the code is
reworked in such a way that the behavior of these queries is changed,
we would get more input from them, written this way.
- Expect at least one record reported in the valid queries doing scans
across ranges, rather than zero records.
- Adjust a few comments, for consistency.

Author: Bharath Rupireddy
Discussion: https://postgr.es/m/CALj2ACVaoXW3nJD9zq8E66BEf-phgJfFcKRVJq9GXkuX0b3ULQ@mail.gmail.com
2023-03-23 11:50:35 +09:00