Commit Graph

39553 Commits

Author SHA1 Message Date
Tom Lane 6ad86feecb Add CHECK_FOR_INTERRUPTS in ExecInsert's speculative insertion loop.
Ordinarily the functions called in this loop ought to have plenty
of CFIs themselves; but we've now seen a case where no such CFI is
reached, making the loop uninterruptible.  Even though that's from
a recently-introduced bug, it seems prudent to install a CFI at
the loop level in all branches.

Per discussion of bug #17558 from Andrew Kesper (an actual fix for
that bug will follow).

Discussion: https://postgr.es/m/17558-3f6599ffcf52fd4a@postgresql.org
2022-08-04 14:10:06 -04:00
Tom Lane cc11647991 Add proper regression test for the recent SRFs-in-pathkeys problem.
Remove the test case added by commit fac1b470a, which never actually
worked to expose the problem it claimed to test.  Replace it with
a case that does expose the problem, and also covers the SRF-not-
at-the-top deficiency repaired in 1aa8dad41.

Richard Guo, with some editorialization by me

Discussion: https://postgr.es/m/17564-c7472c2f90ef2da3@postgresql.org
2022-08-04 11:11:33 -04:00
Daniel Gustafsson f8f20203c2 Rephrase comments to make them clearer
The use of "we" when referring to the active backend might be
misunderstood, so rephrase to make it clearer who is performing
the actions discussed in the comment.

Author: Junwang Zhao <zhjwpku@gmail.com>
Reviewed-by: Erikjan Rijkers <er@xs4all.nl>
Reviewed-by: Robert Treat <rob@xzilla.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAEG8a3LRSMqkvjiURiJoSi4aGWORpiXUmUfQQK5PaD6WfPzu3w@mail.gmail.com
2022-08-04 16:30:06 +02:00
John Naylor bcabbfc6a9 Fix formatting and comment typos
Justin Pryzby

Discussion: https://www.postgresql.org/message-id/20220801181136.GJ15006%40telsasoft.com
2022-08-04 16:41:29 +07:00
Michael Paquier 245e14e28b Fix inconsistent comments for some function declarations in headers
Some of the headers list a couple of function prototypes located in a
different file than what is referred to.  This fixes a couple of
places where this inconsistency exists.

Author: Richard Guo
Discussion: https://postgr.es/m/CAMbWs4__RdcSNXPa7L62Ozvo_Q4LvT60o3Bnp8yrQ_m9y5CKvg@mail.gmail.com
2022-08-04 17:36:21 +09:00
John Naylor 56f2c7b58b Support SSE2 intrinsics where available
SSE2 vector instructions are part of the spec for the 64-bit x86
architecture. Until now we have relied on the compiler to autovectorize
in some limited situations, but some useful coding idioms can only be
expressed explicitly via compiler intrinsics. To this end, add a header
that defines USE_SSE2 where available. While x86-only for now, we can
add other architectures in the future. This will also be the intended
place for helper functions that use vector operations.

Reviewed by Nathan Bossart and Masahiko Sawada

Discussion: https://www.postgresql.org/message-id/CAFBsxsE2G_H_5Wbw%2BNOPm70-BK4xxKf86-mRzY%3DL2sLoQqM%2B-Q%40mail.gmail.com
2022-08-04 13:49:18 +07:00
Tom Lane 1aa8dad41f Fix incorrect tests for SRFs in relation_can_be_sorted_early().
Commit fac1b470a thought we could check for set-returning functions
by testing only the top-level node in an expression tree.  This is
wrong in itself, and to make matters worse it encouraged others
to make the same mistake, by exporting tlist.c's special-purpose
IS_SRF_CALL() as a widely-visible macro.  I can't find any evidence
that anyone's taken the bait, but it was only a matter of time.

Use expression_returns_set() instead, and stuff the IS_SRF_CALL()
genie back in its bottle, this time with a warning label.  I also
added a couple of cross-reference comments.

After a fair amount of fooling around, I've despaired of making
a robust test case that exposes the bug reliably, so no test case
here.  (Note that the test case added by fac1b470a is itself
broken, in that it doesn't notice if you remove the code change.
The repro given by the bug submitter currently doesn't fail either
in v15 or HEAD, though I suspect that may indicate an unrelated bug.)

Per bug #17564 from Martijn van Oosterhout.  Back-patch to v13,
as the faulty patch was.

Discussion: https://postgr.es/m/17564-c7472c2f90ef2da3@postgresql.org
2022-08-03 17:33:42 -04:00
Tom Lane 1da0850f0e Reduce test runtime of src/test/modules/snapshot_too_old.
The sto_using_cursor and sto_using_select tests were coded to exercise
every permutation of their test steps, but AFAICS there is no value in
exercising more than one.  This matters because each permutation costs
about six seconds, thanks to the "pg_sleep(6)".  Perhaps we could
reduce that, but the useless permutations seem worth getting rid of
in any case.  (Note that sto_using_hash_index got it right already.)

While here, clean up some other sloppiness such as an unused table.

This doesn't make too much difference in interactive testing, since the
wasted time is typically masked by parallelization with other tests.
However, the buildfarm runs this as a serial step, which means we can
expect to shave ~40 seconds from every buildfarm run.  That makes it
worth back-patching.

Discussion: https://postgr.es/m/2515192.1659454702@sss.pgh.pa.us
2022-08-03 11:14:55 -04:00
Amit Kapila 0c20dd33db Add wait_for_subscription_sync for TAP tests.
The TAP tests for logical replication in src/test/subscription are using
the following code in many places to make sure that the subscription is
synchronized with the publisher:

  $node_publisher->wait_for_catchup('tap_sub');
  $node_subscriber->poll_query_until('postgres',
    qq[SELECT count(1) = 0
       FROM pg_subscription_rel
       WHERE srsubstate NOT IN ('r', 's')]);

The new function wait_for_subscription_sync() can be used to replace the
above code. This eliminates duplicated code and makes it easier to write
future tests.

Author: Masahiko Sawada
Reviewed by: Amit Kapila, Shi yu
Discussion: https://postgr.es/m/CAD21AoC-fvAkaKHa4t1urupwL8xbAcWRePeETvshvy80f6WV1A@mail.gmail.com
2022-08-03 15:31:17 +05:30
David Rowley 9fc1776dda Remove unused fields from ExprEvalStep
These were added recently by 1349d2790.

Reported-by: Zhihong Yu
Discussion: https://postgr.es/m/CALNJ-vTi+YDuAWKp4Z_Dv=mrz=aq81qTg0D7wzc8y7rS_+i_cw@mail.gmail.com
2022-08-03 09:46:02 +12:00
Tom Lane ec62ce55a8 Change type "char"'s I/O format for non-ASCII characters.
Previously, a byte with the high bit set was just transmitted
as-is by charin() and charout().  This is problematic if the
database encoding is multibyte, because the result of charout()
won't be validly encoded, which breaks various stuff that
expects all text strings to be validly encoded.  We've
previously decided to enforce encoding validity rather than try
to individually harden each place that might have a problem with
such strings, so it's time to do something about "char".

To fix, represent high-bit-set characters as \ooo (backslash
and three octal digits), following the ancient "escape" format
for bytea.  charin() will continue to accept the old way as well,
though that is only reachable in single-byte encodings.

Add some test cases just so there is coverage for this code.
We'll otherwise leave this question undocumented as it was before,
because we don't really want to encourage end-user use of "char".

For the moment, back-patch into v15 so that this change appears
in 15beta3.  If there's not great pushback we should consider
absorbing this change into the older branches.

Discussion: https://postgr.es/m/2318797.1638558730@sss.pgh.pa.us
2022-08-02 10:29:35 -04:00
David Rowley 1349d2790b Improve performance of ORDER BY / DISTINCT aggregates
ORDER BY / DISTINCT aggreagtes have, since implemented in Postgres, been
executed by always performing a sort in nodeAgg.c to sort the tuples in
the current group into the correct order before calling the transition
function on the sorted tuples.  This was not great as often there might be
an index that could have provided pre-sorted input and allowed the
transition functions to be called as the rows come in, rather than having
to store them in a tuplestore in order to sort them once all the tuples
for the group have arrived.

Here we change the planner so it requests a path with a sort order which
supports the most amount of ORDER BY / DISTINCT aggregate functions and
add new code to the executor to allow it to support the processing of
ORDER BY / DISTINCT aggregates where the tuples are already sorted in the
correct order.

Since there can be many ORDER BY / DISTINCT aggregates in any given query
level, it's very possible that we can't find an order that suits all of
these aggregates.  The sort order that the planner chooses is simply the
one that suits the most aggregate functions.  We take the most strictly
sorted variation of each order and see how many aggregate functions can
use that, then we try again with the order of the remaining aggregates to
see if another order would suit more aggregate functions.  For example:

SELECT agg(a ORDER BY a),agg2(a ORDER BY a,b) ...

would request the sort order to be {a, b} because {a} is a subset of the
sort order of {a,b}, but;

SELECT agg(a ORDER BY a),agg2(a ORDER BY c) ...

would just pick a plan ordered by {a} (we give precedence to aggregates
which are earlier in the targetlist).

SELECT agg(a ORDER BY a),agg2(a ORDER BY b),agg3(a ORDER BY b) ...

would choose to order by {b} since two aggregates suit that vs just one
that requires input ordered by {a}.

Author: David Rowley
Reviewed-by: Ronan Dunklau, James Coleman, Ranier Vilela, Richard Guo, Tom Lane
Discussion: https://postgr.es/m/CAApHDvpHzfo92%3DR4W0%2BxVua3BUYCKMckWAmo-2t_KiXN-wYH%3Dw%40mail.gmail.com
2022-08-02 23:11:45 +12:00
Amit Kapila 6b24d3f9cc Move common catalog cache access routines to lsyscache.c
In passing, move pg_relation_is_publishable next to similar functions.

Suggested-by: Alvaro Herrera
Author: Amit Kapila
Reviewed-by: Hou Zhijie
Discussion: https://postgr.es/m/CAHut+PupQ5UW9A9ut0Yjt21J9tHhx958z5L0k8-9hTYf_NYqxA@mail.gmail.com
2022-08-02 10:47:22 +05:30
John Naylor c689baa158 Fix comment in pg_db_role_setting.h
Noted by Japin Li

Discussion: https://www.postgresql.org/message-id/MEYP282MB16691ACEDBC94161CF4BA1CCB69A9%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
2022-08-02 11:49:37 +07:00
Amit Kapila 7bf91ec0f3 Remove duplicated wait for subscription sync from 007_ddl.pl.
An oversight in 8f2e2bbf14.

Author: Masahiko Sawada
Reviewed by: Amit Kapila
Backpatch-through: 15, where it was introduced
Discussion: https://postgr.es/m/CAD21AoC-fvAkaKHa4t1urupwL8xbAcWRePeETvshvy80f6WV1A@mail.gmail.com
2022-08-02 09:30:46 +05:30
David Rowley b592422095 Relax overly strict rules in select_outer_pathkeys_for_merge()
The select_outer_pathkeys_for_merge function made an attempt to build the
merge join pathkeys in the same order as query_pathkeys.  This was done as
it may have led to no sort being required for an ORDER BY or GROUP BY
clause in the upper planner.  However, this restriction seems overly
strict as it required that we match the query_pathkeys entirely or we
don't bother putting the merge join pathkeys in that order.

Here we relax this rule so that we use a prefix of the query_pathkeys
providing that prefix matches all of the join quals.  This may provide the
upper planner with partially sorted input which will allow the use of
incremental sorts instead of full sorts.

Author: David Rowley
Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/CAApHDvrtZu0PHVfDPFM4Yx3jNR2Wuwosv+T2zqa7LrhhBr2rRg@mail.gmail.com
2022-08-02 11:02:46 +12:00
David Rowley 3592e0ff98 Have ExecFindPartition cache the last found partition
Here we add code which detects when ExecFindPartition() continually finds
the same partition and add a caching layer to improve partition lookup
performance for such cases.

Both RANGE and LIST partitioned tables traditionally require a binary
search for the set of Datums that a partition needs to be found for. This
binary search is commonly visible in profiles when bulk loading into a
partitioned table.  Here we aim to reduce the overhead of bulk-loading
into partitioned tables for cases where many consecutive tuples belong to
the same partition and make the performance of this operation closer to
what it is with a traditional non-partitioned table.

When we find the same partition 16 times in a row, the next search will
result in us simply just checking if the current set of values belongs to
the last found partition.  For LIST partitioning we record the index into
the PartitionBoundInfo's datum array.  This allows us to check if the
current Datum is the same as the Datum that was last looked up.  This
means if any given LIST partition supports storing multiple different
Datum values, then the caching only works when we find the same value as
we did the last time.  For RANGE partitioning we simply check if the given
Datums are in the same range as the previously found partition.

We store the details of the cached partition in PartitionDesc (i.e.
relcache) so that the cached values are maintained over multiple
statements.

No caching is done for HASH partitions.  The majority of the cost in HASH
partition lookups are in the hashing function(s), which would also have to
be executed if we were to try to do caching for HASH partitioned tables.
Since most of the cost is already incurred, we just don't bother.  We also
don't do any caching for LIST partitions when we continually find the
values being looked up belong to the DEFAULT partition.  We've no
corresponding index in the PartitionBoundInfo's datum array for this case.
We also don't cache when we find the given values match to a LIST
partitioned table's NULL partition.  This is so cheap that there's no
point in doing any caching for this.  We also don't cache for a RANGE
partitioned table's DEFAULT partition.

There have been a number of different patches submitted to improve
partition lookups. Hou, Zhijie submitted a patch to detect when the value
belonging to the partition key column(s) were constant and added code to
cache the partition in that case.  Amit Langote then implemented an idea
suggested by me to remember the last found partition and start to check if
the current values work for that partition.  The final patch here was
written by me and was done by taking many of the ideas I liked from the
patches in the thread and redesigning other aspects.

Discussion: https://postgr.es/m/OS0PR01MB571649B27E912EA6CC4EEF03942D9%40OS0PR01MB5716.jpnprd01.prod.outlook.com
Author: Amit Langote, Hou Zhijie, David Rowley
Reviewed-by: Amit Langote, Hou Zhijie
2022-08-02 09:55:27 +12:00
Tom Lane 83f1793d60 Check maximum number of columns in function RTEs, too.
I thought commit fd96d14d9 had plugged all the holes of this sort,
but no, function RTEs could produce oversize tuples too, either
via long coldeflists or just from multiple functions in one RTE.
(I'm pretty sure the other variants of base RTEs aren't a problem,
because they ultimately refer to either a table or a sub-SELECT,
whose widths are enforced elsewhere.  But we explicitly allow join
RTEs to be overwidth, as long as you don't try to form their
tuple result.)

Per further discussion of bug #17561.  As before, patch all branches.

Discussion: https://postgr.es/m/17561-80350151b9ad2ad4@postgresql.org
2022-08-01 12:22:35 -04:00
Michael Paquier 8b1ec7d295 Fix error reporting after ioctl() call with pg_upgrade --clone
errno was not reported correctly after attempting to clone a file,
leading to incorrect error reports.  While scanning through the code, I
have not noticed any similar mistakes.

Error introduced in 3a769d8.

Author: Justin Pryzby
Discussion: https://postgr.es/m/20220731134135.GY15006@telsasoft.com
Backpatch-through: 12
2022-08-01 16:38:23 +09:00
Michael Paquier 7ff358b76a Append -X to direct invocation of psql in new test for BASE_BACKUP
Per buildfarm member wrasse, that looks to open a transaction when it
loads its .psqlrc, causing the test to fail.

Oversight in ad34146.
2022-08-01 09:58:19 +09:00
Michael Paquier ad341469b4 Add more TAP tests with BASE_BACKUP and pg_backup_start/stop
This commit adds some test coverage for ee79647 (prevent BASE_BACKUP
from running in the middle of another base backup) and b24b2be
(BASE_BACKUP cancellation followed by pg_backup_start), caused by the
interactions of replication and SQL commands in a logical replication
connection in a WAL sender.

The second test uses a design close to what has been introduced in
0475a97f, where BASE_BACKUP is throttled to give enough room for a
cancellation, though this time we rely on psql with multiple -c
switches to keep a connection around for the second query.

Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/Ys/NCI4Eo9300GnQ@paquier.xyz
2022-08-01 09:16:11 +09:00
Tom Lane 3451a57f77 Remove test_oat_hooks.c's nodetag_to_string().
In the short time this function has existed, it's already proven to be
a nontrivial maintenance burden, since it has to be updated whenever a
node tag is added or removed.  Although in principle we could now
automate that, I see little justification for having such functionality
here at all.  The function is only being applied to utility statements,
for which we already have infrastructure for obtaining string names.
Moreover, that infrastructure produces already-familiar-to-users names,
unlike nodetag_to_string().

So, remove this function and use the existing infrastructure instead.
That saves over a thousand lines of largely-unreachable code.

Back-patch to v15 where this code came in.  Although it seems unlikely
that v15's nodetag list will change anymore, we might as well keep the
two branches looking and acting alike; otherwise back-patching any
test-results changes in this area will be painful.

Discussion: https://postgr.es/m/843818.1659218928@sss.pgh.pa.us
2022-07-31 16:58:25 -04:00
Andrew Dunstan 7781f4e3e7 Add --schema and --exclude-schema options to vacuumdb.
These two new options can be used to either process all tables in
specific schemas or to skip processing all tables in specific
schemas.  This change also refactors the handling of invalid
combinations of command-line options to a new helper function.

Author: Gilles Darold
Reviewed-by: Justin Pryzby, Nathan Bossart and Michael Paquier.
Discussion: https://postgr.es/m/929fbf3c-24b8-d454-811f-1d5898ab3e91%40migops.com
2022-07-31 16:46:13 -04:00
Tom Lane 4ddfbd2a8e Fix trim_array() for zero-dimensional array argument.
The code tried to access ARR_DIMS(v)[0] and ARR_LBOUND(v)[0]
whether or not those values exist.  This made the range check
on the "n" argument unstable --- it might or might not fail, and
if it did it would report garbage for the allowed upper limit.
These bogus accesses would probably annoy Valgrind, and if you
were very unlucky even lead to SIGSEGV.

Report and fix by Martin Kalcher.  Back-patch to v14 where this
function was added.

Discussion: https://postgr.es/m/baaeb413-b8a8-4656-5757-ef347e5ec11f@aboutsource.net
2022-07-31 13:43:17 -04:00
Michael Paquier 43231423da Feed ObjectAddress to event triggers for ALTER TABLE ATTACH/DETACH
These flavors of ALTER TABLE were already shaped to report the
ObjectAddress of the partition attached or detached, but this data was
not added to what is collected for event triggers.  The tests of
test_ddl_deparse are updated to show the modification in the data
reported.

Author: Hou Zhijie
Reviewed-by: Álvaro Herrera, Amit Kapila, Hayato Kuroda, Michael Paquier
Discussion: https://postgr.es/m/OS0PR01MB571626984BD099DADF53F38394899@OS0PR01MB5716.jpnprd01.prod.outlook.com
2022-07-31 13:04:43 +09:00
Michael Paquier 07ff701dbd Expand tests of test_ddl_deparse/ for ALTER TABLE
This module is expanded to track the description of the objects changed
in the subcommands of ALTER TABLE by reworking the function
get_altertable_subcmdtypes() (now named get_altertable_subcmdinfo) used
in the event trigger of the test.  It now returns a set of rows made of
(subcommand type, object description) instead of a text array with only
the information about the subcommand type.

The tests have been lacking a lot of the subcommands added to
AlterTableType over the years.  All the missing subcommands are added,
and the code is now structured so as the addition of a new subcommand
is detected by removing the default clause used in the switch for the
subcommand types.

The coverage of the module is increased from roughly 30% to 50%.  More
could be done but this is already a nice improvement.

Author: Michael Paquier, Hou Zhijie
Reviewed-by: Álvaro Herrera, Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/OS0PR01MB571626984BD099DADF53F38394899@OS0PR01MB5716.jpnprd01.prod.outlook.com
2022-07-31 11:48:14 +09:00
Tom Lane 6a1f082aba Improve regression test coverage of GiST index building.
Add a test case that exercises the "buffering build" code path.
This covers almost all the non-error-case lines in gistbuild.c
and gistbuildbuffers.c.

Matheus Alcantara, based on earlier work by Pavel Borisov

Discussion: https://postgr.es/m/3z8Fde-IHbW57a7bEZtaf19f4YOCWu67IZoWJoGW18rKD9R16ZHHchf4d7KFI3Yg7-0N4NonFuwKEgh98HjMCZYoVx7KOioPo6Wn2nZRpf4=@pm.me
2022-07-30 16:22:24 -04:00
Tom Lane d8e34fa7a1 Fix incorrect is-this-the-topmost-join tests in parallel planning.
Two callers of generate_useful_gather_paths were testing the wrong
thing when deciding whether to call that function: they checked for
being at the top of the current join subproblem, rather than being at
the actual top join.  This'd result in failing to construct parallel
paths for a sub-join for which they might be useful.

While set_rel_pathlist() isn't actively broken, it seems best to
make its identical-in-intention test for this be like the other two.

This has been wrong all along, but given the lack of field complaints
I'm hesitant to back-patch into stable branches; we usually prefer
to avoid non-bug-fix changes in plan choices in minor releases.
It seems not too late for v15 though.

Richard Guo, reviewed by Antonin Houska and Tom Lane

Discussion: https://postgr.es/m/CAMbWs4-mH8Zf87-w+3P2J=nJB+5OyicO28ia9q_9o=Lamf_VHg@mail.gmail.com
2022-07-30 13:05:15 -04:00
Tom Lane d10fad96c6 Adjust new pg_read_file() test cases for more portability.
It's allowed for an installation to remove postgresql.auto.conf,
so don't rely on that being present.  Instead probe whether we can
read postmaster.pid.  (If you've removed that, you broke the data
directory's multiple-postmaster interlock, not to mention pg_ctl.)
Per gripe from Michael Paquier.

Discussion: https://postgr.es/m/YuSZTsoBMObyY+vT@paquier.xyz
2022-07-30 11:17:07 -04:00
Robert Haas 212bdc0cbc Revise test case added in 4374699639.
Instead of using command_ok() to run psql, use safe_psql(). wrasse
isn't happy, and it be because of failure to pass -X to the psql
invocation, which safe_psql() will do automatically.

Since safe_psql() returns standard output instead of writing it to
a file, this requires some changes to the incantation for running
'diff'.

Test against the 'regression' database rather than 'postgres' so
we test more than just one table. That also means we need to record
the horizons later, after the test does "VACUUM FULL pg_largeobject".

Add an ORDER BY clause to the horizon query for stability.

Patch by me, reviewed by Tom Lane.

Discussion: http://postgr.es/m/CA+TgmoaGBbpzgu3=du1f9zDUbkfycO0y=_uWrLFy=KKEqXWeLQ@mail.gmail.com
2022-07-29 23:24:39 -04:00
Andrew Dunstan b998196bb5 Fix new recovery test for log_error_verbosity=verbose case
The new test is from commit 9e4f914b5e.

With this setting messages have SQL error numbers included, so that
needs to be provided for in the pattern looked for.
2022-07-29 17:54:19 -04:00
Robert Haas 4374699639 Fix brown paper bag bug in bbe08b8869.
We must issue the TRUNCATE command first and update relfrozenxid
and relminmxid afterward; otherwise, TRUNCATE overwrites the
previously-set values.

Add a test case like I should have done the first time.

Per buildfarm report from TestUpgradeXversion.pm, by way of Tom
Lane.
2022-07-29 16:31:57 -04:00
Tom Lane 283129e325 Support pg_read_[binary_]file (filename, missing_ok).
There wasn't an especially nice way to read all of a file while
passing missing_ok = true.  Add an additional overloaded variant
to support that use-case.

While here, refactor the C code to avoid a rats-nest of PG_NARGS
checks, instead handling the argument collection in the outer
wrapper functions.  It's a bit longer this way, but far more
straightforward.

(Upon looking at the code coverage report for genfile.c, I was
impelled to also add a test case for pg_stat_file() -- tgl)

Kyotaro Horiguchi

Discussion: https://postgr.es/m/20220607.160520.1984541900138970018.horikyota.ntt@gmail.com
2022-07-29 15:38:49 -04:00
Tom Lane fd96d14d95 In transformRowExpr(), check for too many columns in the row.
A RowExpr with more than MaxTupleAttributeNumber columns would fail at
execution anyway, since we cannot form a tuple datum with more than that
many columns.  While heap_form_tuple() has a check for too many columns,
it emerges that there are some intermediate bits of code that don't
check and can be driven to failure with sufficiently many columns.
Checking this at parse time seems like the most appropriate place to
install a defense, since we already check SELECT list length there.

While at it, make the SELECT-list-length error use the same errcode
(TOO_MANY_COLUMNS) as heap_form_tuple does, rather than the generic
PROGRAM_LIMIT_EXCEEDED.

Per bug #17561 from Egor Chindyaskin.  The given test case crashes
in all supported branches (and probably a lot further back),
so patch all.

Discussion: https://postgr.es/m/17561-80350151b9ad2ad4@postgresql.org
2022-07-29 13:31:10 -04:00
Robert Haas 80d6907219 Fix mistake in bbe08b8869.
The earlier commit used pg_class.relfilenode where it should have
used pg_class.oid. This could lead to emitting an UPDATE statement
into the dump that would update nothing (or the wrong thing) when
executed in the new cluster, resulting in relfrozenxid and
relminmxid being improperly carried forward for pg_largeobject.

Noticed by Dilip Kumar.

Discussion: http://postgr.es/m/CAFiTN-ty1Gzs6stk2vt9BJiq0m0hzf=aPnh3a-4Z3Tk5GzoENw@mail.gmail.com
2022-07-29 11:20:07 -04:00
Alvaro Herrera 59be1c942a
Fix test instability
On FreeBSD, the new test fails due to a WAL file being removed before
the standby has had the chance to copy it.  Fix by adding a replication
slot to prevent the removal until after the standby has connected.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reported-by: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAEze2Wj5nau_qpjbwihvmXLfkAWOZ5TKdbnqOc6nKSiRJEoPyQ@mail.gmail.com
2022-07-29 12:50:47 +02:00
Amit Kapila 0234ed81e9 Move related functions next to each other in pg_publication.c.
This also improves comments atop is_publishable_class().

Author: Peter Smith
Reviewed-by: Amit Kapila, Hou Zhijie
Discussion: https://postgr.es/m/CAHut+PupQ5UW9A9ut0Yjt21J9tHhx958z5L0k8-9hTYf_NYqxA@mail.gmail.com
2022-07-29 14:27:40 +05:30
Robert Haas bbe08b8869 Use TRUNCATE to preserve relfilenode for pg_largeobject + index.
Commit 9a974cbcba arranged to preserve
the relfilenode of user tables across pg_upgrade, but failed to notice
that pg_upgrade treats pg_largeobject as a user table and thus it needs
the same treatment. Otherwise, large objects will appear to vanish
after a  pg_upgrade.

Commit d498e052b4 fixed this problem
by teaching pg_dump to UPDATE pg_class.relfilenode for pg_largeobject
and its index. However, because an UPDATE on the catalog rows doesn't
change anything on disk, this can leave stray files behind in the new
cluster. They will normally be empty, but it's a little bit untidy.

Hence, this commit arranges to do the same thing using DDL. Specifically,
it makes TRUNCATE work for the pg_largeobject catalog when in
binary-upgrade mode, and it then uses that command in binary-upgrade
dumps as a way of setting pg_class.relfilenode for pg_largeobject and
its index. That way, the old files are removed from the new cluster.

Discussion: http://postgr.es/m/CA+TgmoYYMXGUJO5GZk1-MByJGu_bB8CbOL6GJQC8=Bzt6x6vDg@mail.gmail.com
2022-07-28 16:03:42 -04:00
Tom Lane e09d7a1262 Improve speed of hash index build.
In the initial data sort, if the bucket numbers are the same then
next sort on the hash value.  Because index pages are kept in
hash value order, this gains a little speed by allowing the
eventual tuple insertions to be done sequentially, avoiding repeated
data movement within PageAddItem.  This seems to be good for overall
speedup of 5%-9%, depending on the incoming data.

Simon Riggs, reviewed by Amit Kapila

Discussion: https://postgr.es/m/CANbhV-FG-1ZNMBuwhUF7AxxJz3u5137dYL-o6hchK1V_dMw86g@mail.gmail.com
2022-07-28 14:34:32 -04:00
Robert Haas 851f4cc75c Clean up some residual confusion between OIDs and RelFileNumbers.
Commit b0a55e4329 missed a few places
where we are referring to the number used as a part of the relation
filename as an "OID". We now want to call that a "RelFileNumber".

Some of these places actually made it sound like the OID in question
is pg_class.oid rather than pg_class.relfilenode, which is especially
good to clean up.

Dilip Kumar with some editing by me.
2022-07-28 10:20:29 -04:00
Alvaro Herrera 9e4f914b5e
Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories
when replaying database-creation WAL records.  Prior to this
patch, the standby would fail to recover in such a case;
however, the directories could be legitimately missing.
Consider the following sequence of commands:

    CREATE DATABASE
    DROP DATABASE
    DROP TABLESPACE

If, after replaying the last WAL record and removing the
tablespace directory, the standby crashes and has to replay the
create database record again, crash recovery must be able to continue.

A fix for this problem was already attempted in 49d9cfc68b, but it
was reverted because of design issues.  This new version is based
on Robert Haas' proposal: any missing tablespaces are created
during recovery before reaching consistency.  Tablespaces
are created as real directories, and should be deleted
by later replay.  CheckRecoveryConsistency ensures
they have disappeared.

The problems detected by this new code are reported as PANIC,
except when allow_in_place_tablespaces is set to ON, in which
case they are WARNING.  Apart from making tests possible, this
gives users an escape hatch in case things don't go as planned.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Author: Asim R Praveen <apraveen@pivotal.io>
Author: Paul Guo <paulguo@gmail.com>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com> (older versions)
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com> (older versions)
Reviewed-by: Michaël Paquier <michael@paquier.xyz>
Diagnosed-by: Paul Guo <paulguo@gmail.com>
Discussion: https://postgr.es/m/CAEET0ZGx9AvioViLf7nbR_8tH9-=27DN5xWJ2P9-ROH16e4JUA@mail.gmail.com
2022-07-28 08:40:06 +02:00
Fujii Masao d396606ebe Fix comment in procarray.c.
Commit fea10a6434 renamed VariableCacheData.nextFullXid to nextXid.
But commit dc7420c2c9 introduced the comment mentioning nextFullXid.
This commit changes"nextFullXid" to "nextXid" in the comment.

Author: Zhang Mingli
Discussion: https://postgr.es/m/642BA615-4B28-4B0C-BDF6-4D33E366BCDF@gmail.com
2022-07-28 14:56:20 +09:00
Thomas Munro 4fc6b6eefc Fix get_dirent_type() for symlinks on MinGW/MSYS.
On Windows with MSVC, get_dirent_type() was recently made to return
DT_LNK for junction points by commit 9d3444dc, which fixed some
defective dirent.c code.

On Windows with Cygwin, get_dirent_type() already worked for symlinks,
as it does on POSIX systems, because Cygwin has its own fake symlinks
that behave like POSIX (on closer inspection, Cygwin's dirent has the
BSD d_type extension but it's probably always DT_UNKNOWN, so we fall
back to lstat(), which understands Cygwin symlinks with S_ISLNK()).

On Windows with MinGW/MSYS, we need extra code, because the MinGW
runtime has its own readdir() without d_type, and the lstat()-based
fallback has no knowledge of our convention for treating junctions as
symlinks.

Back-patch to 14, where get_dirent_type() landed.

Reported-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/b9ddf605-6b36-f90d-7c30-7b3e95c46276%40dunslane.net
2022-07-28 14:26:12 +12:00
Robert Haas 5f858dd3be Bump catversion for commit d8cd0c6c95.
The catalog contents haven't changed, but it's good to make clear
that initdb is required. Changing RELMAPPER_FILEMAGIC would be more
appropriate, but that doesn't actually produce a useful diagnostic,
so cheat by doing this instead.

Discussion: http://postgr.es/m/20220727171939.6ixixqcjt5riil2o@alvherre.pgsql
2022-07-27 16:18:21 -04:00
Robert Haas 3ac88fddd9 Convert macros to static inline functions (buf_internals.h)
Dilip Kumar, reviewed by Vignesh C, Ashutosh Sharma, and me.

Discussion: http://postgr.es/m/CAFiTN-tYbM7D+2UGiNc2kAFMSQTa5FTeYvmg-Vj2HvPdVw2Gvg@mail.gmail.com
2022-07-27 13:54:37 -04:00
Robert Haas a2e97cb2b6 Fix read_relmap_file() concurrency on Windows.
Commit d8cd0c6c95 introduced a file
rename that could fail on Windows, probably due to other backends
having an open file handle to the old file of the same name.
Re-arrange the locking slightly to prevent that, by making sure the
open() and close() run while we hold the lock.

Thomas Munro. I added an explanatory comment.

Discussion: https://postgr.es/m/CA%2BhUKGLZtCTgp4NTWV-wGbR2Nyag71%3DEfYTKjDKnk%2BfkhuFMHw%40mail.gmail.com
2022-07-27 11:12:15 -04:00
Michael Paquier ce3049b021 Refactor code in charge of grabbing the relations of a subscription
GetSubscriptionRelations() and GetSubscriptionNotReadyRelations() share
mostly the same code, which scans pg_subscription_rel and fetches all
the relations of a given subscription.  The only difference is that the
second routine looks for all the relations not in a ready state.  This
commit refactors the code to use a single routine, shaving a bit of
code.

Author: Vignesh C
Reviewed-By: Kyotaro Horiguchi, Amit Kapila, Michael Paquier, Peter
Smith
Discussion: https://postgr.es/m/CALDaNm0eW-9g4G_EzHebnFT5zZoasWCS_EzZQ5BgnLZny9S=pg@mail.gmail.com
2022-07-27 19:50:06 +09:00
Alexander Korotkov d0b193c0fa Split tuplesortvariants.c from tuplesort.c
This commit puts the implementation of Tuple sort variants into the separate
file tuplesortvariants.c.  That gives better separation of the code and
serves well as the demonstration that Tuple sort variant can be defined outside
of tuplesort.c.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:28:26 +03:00
Alexander Korotkov ec92fe9835 Split TuplesortPublic from Tuplesortstate
The new TuplesortPublic data structure contains the definition of
sort-variant-specific interface methods and the part of Tuple sort operation
state required by their implementations.  This will let define Tuple sort
variants without knowledge of Tuplesortstate, that is without knowledge
of generic sort implementation guts.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:28:10 +03:00
Alexander Korotkov 097366c45f Move memory management away from writetup() and tuplesort_put*()
This commit puts some generic work away from sort-variant-specific function.
In particular, tuplesort_put*() now doesn't need to decrease available memory
and switch to sort context before calling puttuple_common().  writetup()
doesn't need to free SortTuple.tuple and increase available memory.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:27:58 +03:00