Commit Graph

54304 Commits

Author SHA1 Message Date
Andres Freund 2827f108d1 ci: remove minor version from freebsd image name
This way .cirrus.yml does not need to be changed just because freebsd releases
a new minor version.

Discussion: https://postgr.es/m/20220728095704.ryywoaz4dqqrwstc@alap3.anarazel.de
Backpatch: 15-, just like the CI support
2022-07-31 18:51:47 -07:00
Michael Paquier 7ff358b76a Append -X to direct invocation of psql in new test for BASE_BACKUP
Per buildfarm member wrasse, that looks to open a transaction when it
loads its .psqlrc, causing the test to fail.

Oversight in ad34146.
2022-08-01 09:58:19 +09:00
Michael Paquier ad341469b4 Add more TAP tests with BASE_BACKUP and pg_backup_start/stop
This commit adds some test coverage for ee79647 (prevent BASE_BACKUP
from running in the middle of another base backup) and b24b2be
(BASE_BACKUP cancellation followed by pg_backup_start), caused by the
interactions of replication and SQL commands in a logical replication
connection in a WAL sender.

The second test uses a design close to what has been introduced in
0475a97f, where BASE_BACKUP is throttled to give enough room for a
cancellation, though this time we rely on psql with multiple -c
switches to keep a connection around for the second query.

Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/Ys/NCI4Eo9300GnQ@paquier.xyz
2022-08-01 09:16:11 +09:00
Tom Lane 3451a57f77 Remove test_oat_hooks.c's nodetag_to_string().
In the short time this function has existed, it's already proven to be
a nontrivial maintenance burden, since it has to be updated whenever a
node tag is added or removed.  Although in principle we could now
automate that, I see little justification for having such functionality
here at all.  The function is only being applied to utility statements,
for which we already have infrastructure for obtaining string names.
Moreover, that infrastructure produces already-familiar-to-users names,
unlike nodetag_to_string().

So, remove this function and use the existing infrastructure instead.
That saves over a thousand lines of largely-unreachable code.

Back-patch to v15 where this code came in.  Although it seems unlikely
that v15's nodetag list will change anymore, we might as well keep the
two branches looking and acting alike; otherwise back-patching any
test-results changes in this area will be painful.

Discussion: https://postgr.es/m/843818.1659218928@sss.pgh.pa.us
2022-07-31 16:58:25 -04:00
Andrew Dunstan 7781f4e3e7 Add --schema and --exclude-schema options to vacuumdb.
These two new options can be used to either process all tables in
specific schemas or to skip processing all tables in specific
schemas.  This change also refactors the handling of invalid
combinations of command-line options to a new helper function.

Author: Gilles Darold
Reviewed-by: Justin Pryzby, Nathan Bossart and Michael Paquier.
Discussion: https://postgr.es/m/929fbf3c-24b8-d454-811f-1d5898ab3e91%40migops.com
2022-07-31 16:46:13 -04:00
Andres Freund e1c95e50e9 ci: switch to freebsd 13.1
freebsd 13.0 is out of support, switch to 13.1. It might be a good idea to
remove the minor version number from the image name, but there's not been a
response to that so far...

Discussion: https://postgr.es/m/20220728095704.ryywoaz4dqqrwstc@alap3.anarazel.de
Backpatch: 15-, just like the CI support
2022-07-31 12:31:36 -07:00
Tom Lane 4ddfbd2a8e Fix trim_array() for zero-dimensional array argument.
The code tried to access ARR_DIMS(v)[0] and ARR_LBOUND(v)[0]
whether or not those values exist.  This made the range check
on the "n" argument unstable --- it might or might not fail, and
if it did it would report garbage for the allowed upper limit.
These bogus accesses would probably annoy Valgrind, and if you
were very unlucky even lead to SIGSEGV.

Report and fix by Martin Kalcher.  Back-patch to v14 where this
function was added.

Discussion: https://postgr.es/m/baaeb413-b8a8-4656-5757-ef347e5ec11f@aboutsource.net
2022-07-31 13:43:17 -04:00
Tom Lane bfac42ea02 Make new auto_explain test safe for log_error_verbosity = verbose.
Allow for the possible presence of a SQLSTATE code in the expected
warning message, similarly to b998196bb and 19408aae7 (although
here I see no need to allow more than one specific SQLSTATE).
Per gripe from Andrew Dunstan.

Discussion: https://postgr.es/m/c550ac53-5db5-3958-1798-50bae3d9af71@dunslane.net
2022-07-31 12:29:44 -04:00
Michael Paquier 43231423da Feed ObjectAddress to event triggers for ALTER TABLE ATTACH/DETACH
These flavors of ALTER TABLE were already shaped to report the
ObjectAddress of the partition attached or detached, but this data was
not added to what is collected for event triggers.  The tests of
test_ddl_deparse are updated to show the modification in the data
reported.

Author: Hou Zhijie
Reviewed-by: Álvaro Herrera, Amit Kapila, Hayato Kuroda, Michael Paquier
Discussion: https://postgr.es/m/OS0PR01MB571626984BD099DADF53F38394899@OS0PR01MB5716.jpnprd01.prod.outlook.com
2022-07-31 13:04:43 +09:00
Michael Paquier 07ff701dbd Expand tests of test_ddl_deparse/ for ALTER TABLE
This module is expanded to track the description of the objects changed
in the subcommands of ALTER TABLE by reworking the function
get_altertable_subcmdtypes() (now named get_altertable_subcmdinfo) used
in the event trigger of the test.  It now returns a set of rows made of
(subcommand type, object description) instead of a text array with only
the information about the subcommand type.

The tests have been lacking a lot of the subcommands added to
AlterTableType over the years.  All the missing subcommands are added,
and the code is now structured so as the addition of a new subcommand
is detected by removing the default clause used in the switch for the
subcommand types.

The coverage of the module is increased from roughly 30% to 50%.  More
could be done but this is already a nice improvement.

Author: Michael Paquier, Hou Zhijie
Reviewed-by: Álvaro Herrera, Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/OS0PR01MB571626984BD099DADF53F38394899@OS0PR01MB5716.jpnprd01.prod.outlook.com
2022-07-31 11:48:14 +09:00
Tom Lane 6a1f082aba Improve regression test coverage of GiST index building.
Add a test case that exercises the "buffering build" code path.
This covers almost all the non-error-case lines in gistbuild.c
and gistbuildbuffers.c.

Matheus Alcantara, based on earlier work by Pavel Borisov

Discussion: https://postgr.es/m/3z8Fde-IHbW57a7bEZtaf19f4YOCWu67IZoWJoGW18rKD9R16ZHHchf4d7KFI3Yg7-0N4NonFuwKEgh98HjMCZYoVx7KOioPo6Wn2nZRpf4=@pm.me
2022-07-30 16:22:24 -04:00
Tom Lane be39d88934 Add regression test coverage for contrib/pg_buffercache.
We can't check the output of this view very closely without
creating portability headaches, but we can make sure that
the number of rows is as-expected.  In any case, this is
sufficient to exercise all the C code within, which is a
lot better than the 0% coverage we had before.

DongWook Lee

Discussion: https://postgr.es/m/CAAcByaLCHGJB7qAENEcx9D09UL=w4ma+yijwF_-1MSqQZ9wK6Q@mail.gmail.com
2022-07-30 15:33:44 -04:00
Tom Lane d8e34fa7a1 Fix incorrect is-this-the-topmost-join tests in parallel planning.
Two callers of generate_useful_gather_paths were testing the wrong
thing when deciding whether to call that function: they checked for
being at the top of the current join subproblem, rather than being at
the actual top join.  This'd result in failing to construct parallel
paths for a sub-join for which they might be useful.

While set_rel_pathlist() isn't actively broken, it seems best to
make its identical-in-intention test for this be like the other two.

This has been wrong all along, but given the lack of field complaints
I'm hesitant to back-patch into stable branches; we usually prefer
to avoid non-bug-fix changes in plan choices in minor releases.
It seems not too late for v15 though.

Richard Guo, reviewed by Antonin Houska and Tom Lane

Discussion: https://postgr.es/m/CAMbWs4-mH8Zf87-w+3P2J=nJB+5OyicO28ia9q_9o=Lamf_VHg@mail.gmail.com
2022-07-30 13:05:15 -04:00
Tom Lane d10fad96c6 Adjust new pg_read_file() test cases for more portability.
It's allowed for an installation to remove postgresql.auto.conf,
so don't rely on that being present.  Instead probe whether we can
read postmaster.pid.  (If you've removed that, you broke the data
directory's multiple-postmaster interlock, not to mention pg_ctl.)
Per gripe from Michael Paquier.

Discussion: https://postgr.es/m/YuSZTsoBMObyY+vT@paquier.xyz
2022-07-30 11:17:07 -04:00
Robert Haas 212bdc0cbc Revise test case added in 4374699639.
Instead of using command_ok() to run psql, use safe_psql(). wrasse
isn't happy, and it be because of failure to pass -X to the psql
invocation, which safe_psql() will do automatically.

Since safe_psql() returns standard output instead of writing it to
a file, this requires some changes to the incantation for running
'diff'.

Test against the 'regression' database rather than 'postgres' so
we test more than just one table. That also means we need to record
the horizons later, after the test does "VACUUM FULL pg_largeobject".

Add an ORDER BY clause to the horizon query for stability.

Patch by me, reviewed by Tom Lane.

Discussion: http://postgr.es/m/CA+TgmoaGBbpzgu3=du1f9zDUbkfycO0y=_uWrLFy=KKEqXWeLQ@mail.gmail.com
2022-07-29 23:24:39 -04:00
Andrew Dunstan b998196bb5 Fix new recovery test for log_error_verbosity=verbose case
The new test is from commit 9e4f914b5e.

With this setting messages have SQL error numbers included, so that
needs to be provided for in the pattern looked for.
2022-07-29 17:54:19 -04:00
Robert Haas 4374699639 Fix brown paper bag bug in bbe08b8869.
We must issue the TRUNCATE command first and update relfrozenxid
and relminmxid afterward; otherwise, TRUNCATE overwrites the
previously-set values.

Add a test case like I should have done the first time.

Per buildfarm report from TestUpgradeXversion.pm, by way of Tom
Lane.
2022-07-29 16:31:57 -04:00
Tom Lane 283129e325 Support pg_read_[binary_]file (filename, missing_ok).
There wasn't an especially nice way to read all of a file while
passing missing_ok = true.  Add an additional overloaded variant
to support that use-case.

While here, refactor the C code to avoid a rats-nest of PG_NARGS
checks, instead handling the argument collection in the outer
wrapper functions.  It's a bit longer this way, but far more
straightforward.

(Upon looking at the code coverage report for genfile.c, I was
impelled to also add a test case for pg_stat_file() -- tgl)

Kyotaro Horiguchi

Discussion: https://postgr.es/m/20220607.160520.1984541900138970018.horikyota.ntt@gmail.com
2022-07-29 15:38:49 -04:00
Tom Lane fd96d14d95 In transformRowExpr(), check for too many columns in the row.
A RowExpr with more than MaxTupleAttributeNumber columns would fail at
execution anyway, since we cannot form a tuple datum with more than that
many columns.  While heap_form_tuple() has a check for too many columns,
it emerges that there are some intermediate bits of code that don't
check and can be driven to failure with sufficiently many columns.
Checking this at parse time seems like the most appropriate place to
install a defense, since we already check SELECT list length there.

While at it, make the SELECT-list-length error use the same errcode
(TOO_MANY_COLUMNS) as heap_form_tuple does, rather than the generic
PROGRAM_LIMIT_EXCEEDED.

Per bug #17561 from Egor Chindyaskin.  The given test case crashes
in all supported branches (and probably a lot further back),
so patch all.

Discussion: https://postgr.es/m/17561-80350151b9ad2ad4@postgresql.org
2022-07-29 13:31:10 -04:00
Robert Haas 80d6907219 Fix mistake in bbe08b8869.
The earlier commit used pg_class.relfilenode where it should have
used pg_class.oid. This could lead to emitting an UPDATE statement
into the dump that would update nothing (or the wrong thing) when
executed in the new cluster, resulting in relfrozenxid and
relminmxid being improperly carried forward for pg_largeobject.

Noticed by Dilip Kumar.

Discussion: http://postgr.es/m/CAFiTN-ty1Gzs6stk2vt9BJiq0m0hzf=aPnh3a-4Z3Tk5GzoENw@mail.gmail.com
2022-07-29 11:20:07 -04:00
Alvaro Herrera 59be1c942a
Fix test instability
On FreeBSD, the new test fails due to a WAL file being removed before
the standby has had the chance to copy it.  Fix by adding a replication
slot to prevent the removal until after the standby has connected.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reported-by: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAEze2Wj5nau_qpjbwihvmXLfkAWOZ5TKdbnqOc6nKSiRJEoPyQ@mail.gmail.com
2022-07-29 12:50:47 +02:00
Amit Kapila 0234ed81e9 Move related functions next to each other in pg_publication.c.
This also improves comments atop is_publishable_class().

Author: Peter Smith
Reviewed-by: Amit Kapila, Hou Zhijie
Discussion: https://postgr.es/m/CAHut+PupQ5UW9A9ut0Yjt21J9tHhx958z5L0k8-9hTYf_NYqxA@mail.gmail.com
2022-07-29 14:27:40 +05:30
Robert Haas bbe08b8869 Use TRUNCATE to preserve relfilenode for pg_largeobject + index.
Commit 9a974cbcba arranged to preserve
the relfilenode of user tables across pg_upgrade, but failed to notice
that pg_upgrade treats pg_largeobject as a user table and thus it needs
the same treatment. Otherwise, large objects will appear to vanish
after a  pg_upgrade.

Commit d498e052b4 fixed this problem
by teaching pg_dump to UPDATE pg_class.relfilenode for pg_largeobject
and its index. However, because an UPDATE on the catalog rows doesn't
change anything on disk, this can leave stray files behind in the new
cluster. They will normally be empty, but it's a little bit untidy.

Hence, this commit arranges to do the same thing using DDL. Specifically,
it makes TRUNCATE work for the pg_largeobject catalog when in
binary-upgrade mode, and it then uses that command in binary-upgrade
dumps as a way of setting pg_class.relfilenode for pg_largeobject and
its index. That way, the old files are removed from the new cluster.

Discussion: http://postgr.es/m/CA+TgmoYYMXGUJO5GZk1-MByJGu_bB8CbOL6GJQC8=Bzt6x6vDg@mail.gmail.com
2022-07-28 16:03:42 -04:00
Tom Lane e09d7a1262 Improve speed of hash index build.
In the initial data sort, if the bucket numbers are the same then
next sort on the hash value.  Because index pages are kept in
hash value order, this gains a little speed by allowing the
eventual tuple insertions to be done sequentially, avoiding repeated
data movement within PageAddItem.  This seems to be good for overall
speedup of 5%-9%, depending on the incoming data.

Simon Riggs, reviewed by Amit Kapila

Discussion: https://postgr.es/m/CANbhV-FG-1ZNMBuwhUF7AxxJz3u5137dYL-o6hchK1V_dMw86g@mail.gmail.com
2022-07-28 14:34:32 -04:00
Alvaro Herrera 70a437aa45
doc: avoid saying "inheritance" ...
... when it applies to partitioned relations.  This is almost the
opposite of 0c06534bd6, which removed references to "partition" in
favour of "child".

Author: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Ashutosh Bapat <ashutosh.bapat@enterprisedb.com>
Reviewed-by: Amit Langote <amitlangote09@gmail.com>
Discussion: https://postgr.es/m/20220525013248.GO19626@telsasoft.com
2022-07-28 18:21:56 +02:00
Robert Haas 851f4cc75c Clean up some residual confusion between OIDs and RelFileNumbers.
Commit b0a55e4329 missed a few places
where we are referring to the number used as a part of the relation
filename as an "OID". We now want to call that a "RelFileNumber".

Some of these places actually made it sound like the OID in question
is pg_class.oid rather than pg_class.relfilenode, which is especially
good to clean up.

Dilip Kumar with some editing by me.
2022-07-28 10:20:29 -04:00
Alvaro Herrera 9e4f914b5e
Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories
when replaying database-creation WAL records.  Prior to this
patch, the standby would fail to recover in such a case;
however, the directories could be legitimately missing.
Consider the following sequence of commands:

    CREATE DATABASE
    DROP DATABASE
    DROP TABLESPACE

If, after replaying the last WAL record and removing the
tablespace directory, the standby crashes and has to replay the
create database record again, crash recovery must be able to continue.

A fix for this problem was already attempted in 49d9cfc68b, but it
was reverted because of design issues.  This new version is based
on Robert Haas' proposal: any missing tablespaces are created
during recovery before reaching consistency.  Tablespaces
are created as real directories, and should be deleted
by later replay.  CheckRecoveryConsistency ensures
they have disappeared.

The problems detected by this new code are reported as PANIC,
except when allow_in_place_tablespaces is set to ON, in which
case they are WARNING.  Apart from making tests possible, this
gives users an escape hatch in case things don't go as planned.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Author: Asim R Praveen <apraveen@pivotal.io>
Author: Paul Guo <paulguo@gmail.com>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com> (older versions)
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com> (older versions)
Reviewed-by: Michaël Paquier <michael@paquier.xyz>
Diagnosed-by: Paul Guo <paulguo@gmail.com>
Discussion: https://postgr.es/m/CAEET0ZGx9AvioViLf7nbR_8tH9-=27DN5xWJ2P9-ROH16e4JUA@mail.gmail.com
2022-07-28 08:40:06 +02:00
Fujii Masao d396606ebe Fix comment in procarray.c.
Commit fea10a6434 renamed VariableCacheData.nextFullXid to nextXid.
But commit dc7420c2c9 introduced the comment mentioning nextFullXid.
This commit changes"nextFullXid" to "nextXid" in the comment.

Author: Zhang Mingli
Discussion: https://postgr.es/m/642BA615-4B28-4B0C-BDF6-4D33E366BCDF@gmail.com
2022-07-28 14:56:20 +09:00
Thomas Munro 4fc6b6eefc Fix get_dirent_type() for symlinks on MinGW/MSYS.
On Windows with MSVC, get_dirent_type() was recently made to return
DT_LNK for junction points by commit 9d3444dc, which fixed some
defective dirent.c code.

On Windows with Cygwin, get_dirent_type() already worked for symlinks,
as it does on POSIX systems, because Cygwin has its own fake symlinks
that behave like POSIX (on closer inspection, Cygwin's dirent has the
BSD d_type extension but it's probably always DT_UNKNOWN, so we fall
back to lstat(), which understands Cygwin symlinks with S_ISLNK()).

On Windows with MinGW/MSYS, we need extra code, because the MinGW
runtime has its own readdir() without d_type, and the lstat()-based
fallback has no knowledge of our convention for treating junctions as
symlinks.

Back-patch to 14, where get_dirent_type() landed.

Reported-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/b9ddf605-6b36-f90d-7c30-7b3e95c46276%40dunslane.net
2022-07-28 14:26:12 +12:00
Tom Lane 5253519b27 Fix new auto_explain test case for Windows.
In commit 7c34555f8, I overlooked the need to configure SSPI
on Windows to allow login as the non-superuser role.
Fix that by adding auth_extra/--create-role incantation
(which, oddly enough, doesn't actually create the role).
Per buildfarm.

While here, upgrade the mechanism for temporarily setting
$ENV{PGUSER}, as per recommendation from ilmari.

Discussion: https://postgr.es/m/87edy7j1zz.fsf@wibble.ilmari.org
2022-07-27 18:58:35 -04:00
Robert Haas 5f858dd3be Bump catversion for commit d8cd0c6c95.
The catalog contents haven't changed, but it's good to make clear
that initdb is required. Changing RELMAPPER_FILEMAGIC would be more
appropriate, but that doesn't actually produce a useful diagnostic,
so cheat by doing this instead.

Discussion: http://postgr.es/m/20220727171939.6ixixqcjt5riil2o@alvherre.pgsql
2022-07-27 16:18:21 -04:00
Robert Haas 3ac88fddd9 Convert macros to static inline functions (buf_internals.h)
Dilip Kumar, reviewed by Vignesh C, Ashutosh Sharma, and me.

Discussion: http://postgr.es/m/CAFiTN-tYbM7D+2UGiNc2kAFMSQTa5FTeYvmg-Vj2HvPdVw2Gvg@mail.gmail.com
2022-07-27 13:54:37 -04:00
Tom Lane 03361a368e Add missing PGDLLEXPORT markings in contrib/pg_prewarm.
After commit 089480c07, it's necessary for background worker entry
points to be marked PGDLLEXPORT, else they aren't findable by
LookupBackgroundWorkerFunction().  Since pg_prewarm lacks any
regression tests, it's not surprising its worker entry points were
overlooked.  (A quick search turned up no other such oversights.)

I added some documentation pointing out the need for this, too.

Robins Tharakan and Tom Lane

CAEP4nAzndnQv3-1QKb=D-hLoK3Rko12HHMFHHtdj2GQAUXO3gw@mail.gmail.com
2022-07-27 12:00:10 -04:00
Robert Haas a2e97cb2b6 Fix read_relmap_file() concurrency on Windows.
Commit d8cd0c6c95 introduced a file
rename that could fail on Windows, probably due to other backends
having an open file handle to the old file of the same name.
Re-arrange the locking slightly to prevent that, by making sure the
open() and close() run while we hold the lock.

Thomas Munro. I added an explanatory comment.

Discussion: https://postgr.es/m/CA%2BhUKGLZtCTgp4NTWV-wGbR2Nyag71%3DEfYTKjDKnk%2BfkhuFMHw%40mail.gmail.com
2022-07-27 11:12:15 -04:00
Michael Paquier ce3049b021 Refactor code in charge of grabbing the relations of a subscription
GetSubscriptionRelations() and GetSubscriptionNotReadyRelations() share
mostly the same code, which scans pg_subscription_rel and fetches all
the relations of a given subscription.  The only difference is that the
second routine looks for all the relations not in a ready state.  This
commit refactors the code to use a single routine, shaving a bit of
code.

Author: Vignesh C
Reviewed-By: Kyotaro Horiguchi, Amit Kapila, Michael Paquier, Peter
Smith
Discussion: https://postgr.es/m/CALDaNm0eW-9g4G_EzHebnFT5zZoasWCS_EzZQ5BgnLZny9S=pg@mail.gmail.com
2022-07-27 19:50:06 +09:00
Alexander Korotkov d0b193c0fa Split tuplesortvariants.c from tuplesort.c
This commit puts the implementation of Tuple sort variants into the separate
file tuplesortvariants.c.  That gives better separation of the code and
serves well as the demonstration that Tuple sort variant can be defined outside
of tuplesort.c.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:28:26 +03:00
Alexander Korotkov ec92fe9835 Split TuplesortPublic from Tuplesortstate
The new TuplesortPublic data structure contains the definition of
sort-variant-specific interface methods and the part of Tuple sort operation
state required by their implementations.  This will let define Tuple sort
variants without knowledge of Tuplesortstate, that is without knowledge
of generic sort implementation guts.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:28:10 +03:00
Alexander Korotkov 097366c45f Move memory management away from writetup() and tuplesort_put*()
This commit puts some generic work away from sort-variant-specific function.
In particular, tuplesort_put*() now doesn't need to decrease available memory
and switch to sort context before calling puttuple_common().  writetup()
doesn't need to free SortTuple.tuple and increase available memory.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:27:58 +03:00
Alexander Korotkov 033dd02db2 Put abbreviation logic into puttuple_common()
Abbreviation code is very similar along tuplesort_put*() functions.  This
commit unifies that code and puts it into puttuple_common().  tuplesort_put*()
functions differs in the abbreviation condition, so it has been added as an
argument to the puttuple_common() function.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:27:46 +03:00
Alexander Korotkov cadfdd1edf Add new Tuplesortstate.removeabbrev function
This commit is the preparation to move abbreviation logic into
puttuple_common().  The new removeabbrev function turns datum1 representation
of SortTuple's from the abbreviated key to the first column value.  Therefore,
it encapsulates the differential part of abbreviation handling code in
tuplesort_put*() functions, making these functions similar.

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:27:29 +03:00
Alexander Korotkov d47da3162b Remove Tuplesortstate.copytup function
It's currently unclear how do we split functionality between
Tuplesortstate.copytup() function and tuplesort_put*() functions.
For instance, copytup_index() and copytup_datum() return error while
tuplesort_putindextuplevalues() and tuplesort_putdatum() do their work.
This commit removes Tuplesortstate.copytup() altogether, putting the
corresponding code into tuplesort_put*().

Discussion: https://postgr.es/m/CAPpHfdvjix0Ahx-H3Jp1M2R%2B_74P-zKnGGygx4OWr%3DbUQ8BNdw%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Pavel Borisov, Maxim Orlov, Matthias van de Meent
Reviewed-by: Andres Freund, John Naylor
2022-07-27 08:26:53 +03:00
Michael Paquier ffd1b6bb6f Add overflow protection for block-related data in WAL records
XLogRecordBlockHeader, the header holding the information for the data
related to a block, tracks the length of the data appended to the WAL
record with data_length (uint16).  This limitation in size was not
enforced by the public routine in charge of registering the data
assembled later to form the WAL record inserted, XLogRegisterBufData().
Incorrectly used, it could lead to the generation of records with some
of its data overflowed.  This commit adds some safeguards to prevent
that for the block data, complaining immediately if attempting to add to
a record block information with a size larger than UINT16_MAX, which is
the limit implied by the internal logic.

Note that this also adjusts XLogRegisterData() and XLogRegisterBufData()
so as the length of the WAL record data given by the caller is unsigned,
matching with what gets stored in XLogRecData->len.

Extracted from a larger patch by the same author.  The original patch
includes more protections when assembling a record in full that will be
looked at separately later.

Author: Matthias van de Meent
Reviewed-by: Andres Freund, Heikki Linnakangas, Michael Paquier, David
Zhang
Discussion: https://postgr.es/m/CAEze2WgGiw+LZt+vHf8tWqB_6VxeLsMeoAuod0N=ij1q17n5pw@mail.gmail.com
2022-07-27 13:35:40 +09:00
Tom Lane 70988b7b0a Improve makeArrayTypeName's algorithm for choosing array type names.
As before, we start by prepending one underscore (truncating the
base name if necessary).  But if there is a conflict, then instead of
prepending more and more underscores, append an underscore and some
digits, in much the same way that ChooseRelationName does.  While
the previous logic could be driven to fail by creating a lot of
types with long names differing only near the end, this version seems
certain enough to eventually succeed that we can remove the failure
code path that was there before.

While at it, undo 6df7a9698's decision to split this code out of
makeArrayTypeName.  That wasn't actually accomplishing anything,
because no other function was using it --- and it would have been
wrong to do so.  The convention that a prefix "_" means an array,
not something else, is too ancient to mess with.

Andrey Lepikhov and Dmitry Koval, reviewed by Masahiko Sawada and myself

Discussion: https://postgr.es/m/b84cd82c-cc67-198a-8b1c-60f44e1259ad@postgrespro.ru
2022-07-26 15:38:09 -04:00
Robert Haas 8bb3ad462f Fix brain fade in e530be2c5c.
The BoolGetDatum() call ended up in the wrong place. It should be
applied when we, err, want to convert a bool to a datum.

Thanks to Tom Lane for noticing this.

Discussion: http://postgr.es/m/2511599.1658861964@sss.pgh.pa.us
2022-07-26 15:12:09 -04:00
Robert Haas d8cd0c6c95 Remove the restriction that the relmap must be 512 bytes.
Instead of relying on the ability to atomically overwrite the
entire relmap file in one shot, write a new one and durably
rename it into place. Removing the struct padding and the
calculation showing why the map is exactly 512 bytes, and change
the maximum number of entries to a nearby round number.

Patch by me, reviewed by Andres Freund and Dilip Kumar.

Discussion: http://postgr.es/m/CA+TgmoZq5%3DLWDK7kHaUbmWXxcaTuw_QwafgG9dr-BaPym_U8WQ%40mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-ttOXLX75k_WzRo9ar=VvxFhrHi+rJxns997F+yvkm==A@mail.gmail.com
2022-07-26 14:56:25 -04:00
Robert Haas e530be2c5c Do not allow removal of superuser privileges from bootstrap user.
A bootstrap user who is not a superuser will still own many
important system objects, such as the pg_catalog schema, that
will likely allow that user to regain superuser status. Therefore,
allowing the superuser property to be removed from the superuser
creates a false perception of security where none exists.

Although removing superuser from the bootstrap user is also a bad idea
and should be considered unsupported in all released versions, no
back-patch, as this is a behavior change.

Discussion: http://postgr.es/m/CA+TgmoZirCwArJms_fgvLBFrC6b=HdxmG7iAhv+kt_=NBA7tEw@mail.gmail.com
2022-07-26 14:10:38 -04:00
Tom Lane f92944137c Force immediate commit after CREATE DATABASE etc in extended protocol.
We have a few commands that "can't run in a transaction block",
meaning that if they complete their processing but then we fail
to COMMIT, we'll be left with inconsistent on-disk state.
However, the existing defenses for this are only watertight for
simple query protocol.  In extended protocol, we didn't commit
until receiving a Sync message.  Since the client is allowed to
issue another command instead of Sync, we're in trouble if that
command fails or is an explicit ROLLBACK.  In any case, sitting
in an inconsistent state while waiting for a client message
that might not come seems pretty risky.

This case wasn't reachable via libpq before we introduced pipeline
mode, but it's always been an intended aspect of extended query
protocol, and likely there are other clients that could reach it
before.

To fix, set a flag in PreventInTransactionBlock that tells
exec_execute_message to force an immediate commit.  This seems
to be the approach that does least damage to existing working
cases while still preventing the undesirable outcomes.

While here, add some documentation to protocol.sgml that explicitly
says how to use pipelining.  That's latent in the existing docs if
you know what to look for, but it's better to spell it out; and it
provides a place to document this new behavior.

Per bug #17434 from Yugo Nagata.  It's been wrong for ages,
so back-patch to all supported branches.

Discussion: https://postgr.es/m/17434-d9f7a064ce2a88a3@postgresql.org
2022-07-26 13:07:03 -04:00
Fujii Masao 3cabe45a81 doc: Add note about re-archiving of same WAL files in docs.
The server may attempt to re-archive a WAL file that was previously archived.
This commit adds the note about how an archive library should handle such
a re-archiving.

Author: Nathan Bossart
Reviewed-by: David Steele, Kyotaro Horiguchi
Discussion: https://postgr.es/m/CA+TgmoaeTe4oUT39A4nt=8LD6UJ5u0vcmGc5+Aksn-4oKRb8-w@mail.gmail.com
2022-07-26 16:02:43 +09:00
Fujii Masao 756e221db6 Reduce overhead of renaming archive status files.
Presently, archive status files are durably renamed from .ready to
.done to indicate that a file has been archived.  Persisting this
rename to disk accounts for a significant amount of the overhead
associated with archiving.  While durably renaming the file
prevents re-archiving in most cases, archive commands and libraries
must already gracefully handle attempts to re-archive the last
archived file after a crash (e.g., a crash immediately after
archive_command exits but before the server renames the status
file).

This change reduces the amount of overhead associated with
archiving by using rename() instead of durable_rename() to rename
the archive status files.  As a consequence, the server is more
likely to attempt to re-archive files after a crash, but as noted
above, archive commands and modules are already expected to handle
this.  It is also possible that the server will attempt to re-
archive files that have been removed or recycled, but the archiver
already handles this, too.

Author: Nathan Bossart
Reviewed-by: Kyotaro Horiguchi, Fujii Masao
Discussion: https://postgr.es/m/20220222011948.GA3850532@nathanxps13
2022-07-26 16:00:18 +09:00
Michael Paquier 27e0ee57f6 Fix path reference when parsing pg_ident.conf for pg_ident_file_mappings
Since a2c8499, HbaFileName (default pg_hba.conf) was getting used
instead of IdentFileName (default pg_ident.conf) as the parent file to
use as reference when parsing the contents of pg_ident.conf, with
pg_ident.conf correctly opened, when feeding this information to
pg_ident_file_mappings.  This had two consequences:
- On an I/O error when reading pg_ident.conf, the user would get an
ERROR message referring to pg_hba.conf and not pg_ident.conf.
- When reading an external file with a relative path using '@' in
pg_ident.conf, the directory used to look at the file to load would be
the base directory of pg_hba.conf rather than the one of pg_ident.conf,
leading to errors in pg_ident_file_mappings inconsistent with what gets
loaded at startup when pg_ident.conf and pg_hba.conf are located in
different directories.

This error only impacted the SQL view pg_ident_file_mappings that uses a
logic new to v15 to fill the view with the parsed information, not the
code paths loading these authentication files at startup.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20220726050402.vsr6fmz7rsgpmdz3@jrouhaud
Backpatch-through: 15
2022-07-26 15:57:31 +09:00