Commit Graph

55060 Commits

Author SHA1 Message Date
Etsuro Fujita d54e79ba28 Fix copy-and-pasteo in comment. 2022-11-02 18:15:00 +09:00
Amit Kapila 155601de7f Doc: Update information about manually creating slots.
There are some cases (e.g. when the subscription is created using the
connect = false option) where the remote replication slot was not created
automatically and the user must create it manually before the subscription
can be activated. There was not enough information in the docs for users
to do this easily.

Author: Peter Smith
Reviewd by: Shi yu, Amit Kapila
Discussion: https://postgr.es/m/CAHut+PvqdqOanheWSHDyhQiF+Z-7w=-+k4U+bwbT=b6YQ_hrXQ@mail.gmail.com
2022-11-02 11:40:37 +05:30
Amit Kapila 568546f7e4 Improve the description of XLOG_RUNNING_XACTS.
Previously, the description of XLOG_RUNNING_XACTS showed only
top-transaction XIDs and whether subtransactions overflowed. This commit
improves it to show individual subtransaction XIDs. This also improves the
description of overflowed subtransactions.

This additional information can be helpful for testing and debugging
purposes.

Author: Masahiko Sawada
Reviewd by: Fujii Masao, Kyotaro Horiguchi, Ashutosh Bapat, Bharath Rupireddy
Discussion: https://postgr.es/m/CAD21AoAqvaE+XEeXHHPdAGQPcCoGXxuoeutq_nWhUSQvTt5+tA@mail.gmail.com
2022-11-02 10:06:55 +05:30
Michael Paquier e76502871e doc: Fix some descriptions related to pg_ident_file_mappings
pg_ident_file_mappings.line_number was described as a line number in
pg_ident.conf for a "rule" number, but this should refer to a "map".
The same inconsistent term was used in the main paragraph describing the
view.

Extracted from a patch by the same author.  Issue introduced by
a2c8499 where this view has been added.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20221026031948.cbrnzgy5e7glsq2d@jrouhaud
Backpatch-through: 15
2022-11-02 11:56:06 +09:00
David Rowley 3712e0ed47 Fix outdated comment in tuplesort.h
This was outdated by 77bae396d.

Backpatch-through: 15, where 77bae396d was added
2022-11-02 15:29:31 +13:00
Michael Paquier 8e621c10c7 Remove code handling FORCE_NULL and FORCE_NOT_NULL for COPY TO
These two options are only available with COPY FROM, so the extra logic
in charge of checking the validity of the attributes given has no
purpose.

Author: Zhang Mingli
Reviewed-by: Richard Guo, Kyotaro Horiguchi
Discussion: https://postgr.es/m/F28F0B5A-766F-4D33-BF44-43B3A052D833@gmail.com
2022-11-02 10:15:19 +09:00
David Rowley 7c335b7a20 Add doubly linked count list implementation
We have various requirements when using a dlist_head to keep track of the
number of items in the list.  This, traditionally, has been done by
maintaining a counter variable in the calling code.  Here we tidy this up
by adding "dclist", which is very similar to dlist but also keeps track of
the number of items stored in the list.

Callers may use the new dclist_count() function when they need to know how
many items are stored. Obtaining the count is an O(1) operation.

For simplicity reasons, dclist and dlist both use dlist_node as their node
type and dlist_iter/dlist_mutable_iter as their iterator type. dclists
have all of the same functionality as dlists except there is no function
named dclist_delete().  To remove an item from a list dclist_delete_from()
must be used.  This requires knowing which dclist the given item is stored
in.

Additionally, here we also convert some dlists where additional code
exists to keep track of the number of items stored and to make these use
dclists instead.

Author: David Rowley
Reviewed-by: Bharath Rupireddy, Aleksander Alekseev
Discussion: https://postgr.es/m/CAApHDvrtVxr+FXEX0VbViCFKDGxA3tWDgw9oFewNXCJMmwLjLg@mail.gmail.com
2022-11-02 14:06:05 +13:00
Michael Paquier 451d1164b9 Add more tests for COPY with incorrect option combinations
Based on the existing coverage report, some combinations were not
checked at all, so add some tests to do so.  Spotted while looking at
the area.

Discussion: https://postgr.es/m/Y2DNm9u7hzIxCXHn@paquier.xyz
2022-11-02 09:57:54 +09:00
Tom Lane e7c7605a76 Update time zone data files to tzdata release 2022f.
DST law changes in Chile, Fiji, Iran, Jordan, Mexico, Palestine,
and Syria.  Historical corrections for Chile, Crimea, Iran, and
Mexico.

Also, the Europe/Kiev zone has been renamed to Europe/Kyiv
(retaining the old name as a link).

The following zones have been merged into nearby, more-populous zones
whose clocks have agreed since 1970: Antarctica/Vostok, Asia/Brunei,
Asia/Kuala_Lumpur, Atlantic/Reykjavik, Europe/Amsterdam,
Europe/Copenhagen, Europe/Luxembourg, Europe/Monaco, Europe/Oslo,
Europe/Stockholm, Indian/Christmas, Indian/Cocos, Indian/Kerguelen,
Indian/Mahe, Indian/Reunion, Pacific/Chuuk, Pacific/Funafuti,
Pacific/Majuro, Pacific/Pohnpei, Pacific/Wake and Pacific/Wallis.
(This indirectly affects zones that were already links to one of
these: Arctic/Longyearbyen, Atlantic/Jan_Mayen, Iceland,
Pacific/Ponape, Pacific/Truk, and Pacific/Yap.)  America/Nipigon,
America/Rainy_River, America/Thunder_Bay, Europe/Uzhgorod, and
Europe/Zaporozhye were also merged into nearby zones after discovering
that their claimed post-1970 differences from those zones seem to have
been errors.

While the IANA crew have been working on merging zones that have no
post-1970 differences for some time, this batch of changes affects
some zones that are significantly more populous than those merged
in the past, notably parts of Europe.  The loss of pre-1970 timezone
history for those zones may be troublesome for applications
expecting consistency of timestamptz display.  As an example, the
stored value '1944-06-01 12:00 UTC' would previously display as
'1944-06-01 13:00:00+01' if the Europe/Stockholm zone is selected,
but now it will read out as '1944-06-01 14:00:00+02'.

There exists a "packrat" option that will build the timezone data
files with this old data preserved, but the problem is that it also
resurrects a bunch of other, far less well-attested data; so much so
that actually more zones' contents change from 2022a with that option
than without it.  I have chosen not to do that here, for that reason
and because it appears that no major OS distributions are using the
"packrat" option, so that doing so would cause Postgres' behavior
to diverge significantly depending on whether it was built with
--with-system-tzdata.  However, for anyone for whom these changes pose
significant problems, there is a solution: build a set of timezone
files with the "packrat" option and use those with Postgres.
2022-11-01 17:08:28 -04:00
Tom Lane f4857082bc Fix planner failure with extended statistics on partitioned tables.
Some cases would result in "cache lookup failed for statistics object",
due to trying to fetch inherited statistics when only non-inherited
ones are available or vice versa.

Richard Guo and Justin Pryzby

Discussion: https://postgr.es/m/20221030170520.GM16921@telsasoft.com
2022-11-01 14:34:44 -04:00
Tom Lane 495e73c207 pg_stat_statements: fetch stmt location/length before it disappears.
When executing a utility statement, we must fetch everything
we need out of the PlannedStmt data structure before calling
standard_ProcessUtility.  In certain cases (possibly only ROLLBACK
in extended query protocol), that data structure will get freed
during command execution.  The situation is probably often harmless
in production builds, but in debug builds we intentionally overwrite
the freed memory with garbage, leading to picking up garbage values
of statement location and length, typically causing an assertion
failure later in pg_stat_statements.  In non-debug builds, if
something did go wrong it would likely lead to storing garbage
for the query string.

Report and fix by zhaoqigui (with cosmetic adjustments by me).
It's an old problem, so back-patch to all supported versions.

Discussion: https://postgr.es/m/17663-a344fd0675f92128@postgresql.org
Discussion: https://postgr.es/m/1667307420050.56657@hundsun.com
2022-11-01 12:48:01 -04:00
Peter Eisentraut 84387fc889 doc: Add note about lack of publication privileges
This gives some additional advice on using row filters and column
lists on publications securely.

Author: Antonin Houska <ah@cybertec.at>
Reviewed-by: Euler Taveira <euler@eulerto.com>
Discussion: https://www.postgresql.org/message-id/flat/20330.1652105397@antos
2022-11-01 14:19:41 +01:00
Peter Eisentraut 2ea5de296e psql: Improve tab completion for ALTER TABLE on identity columns
- Add tab completion for ALTER SEQUENCE … START …
- Add tab completion for ALTER COLUMN … SET GENERATED …
- Add tab completion for ALTER COLUMN … SET <sequence option>
- Add tab completion for ALTER COLUMN … ADD GENERATED … AS IDENTITY

Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Reviewed-by: Matheus Alcantara <mths.dev@pm.me>
Discussion: https://www.postgresql.org/message-id/flat/87mta1jfax.fsf@wibble.ilmari.org
2022-11-01 12:07:40 +01:00
Tom Lane 0043aa6b85 Add basic regression tests for semi/antijoin recognition.
Add some simple tests that the planner recognizes all the
standard idioms for SEMI and ANTI joins.  Failure to optimize
in this way won't necessarily cause any visible change in
query results, so check the plans.  We had no similar coverage
before, at least for some variants of antijoin, as noted by
Richard Guo.

Discussion: https://postgr.es/m/CAMbWs4-mvPPCJ1W6iK6dD5HiNwoJdi6mZp=-7mE8N9Sh+cd0tQ@mail.gmail.com
2022-10-31 19:52:33 -04:00
Jeff Davis 0717f2fedb Fix ALTER COLLATION "default" REFRESH VERSION.
Issue a helpful error message rather than an internal error.

Discussion: https://postgr.es/m/51fb77507cafd43fc1a2e733c23045873d93ae60.camel%40j-davis.com
Reviewed-by: Thomas Munro
2022-10-31 15:44:52 -07:00
Jeff Davis 10932ed5e5 Enable pg_collation_actual_version() to work on the default collation.
Previously, it would simply return NULL, which was less useful.

Discussion: https://postgr.es/m/51fb77507cafd43fc1a2e733c23045873d93ae60.camel%40j-davis.com
Reviewed-by: Thomas Munro
2022-10-31 15:43:23 -07:00
Peter Eisentraut 6e10631d1e pg_dump test: Make concatenated create_sql commands more readable
When the pg_dump 002_pg_dump.pl test generates the command to load the
schema, it does

    # Add terminating semicolon
    $create_sql{$test_db} .= $tests{$test}->{create_sql} . ";";

In some cases, this creates a duplicate semicolon, but more
importantly, this doesn't add any newline.  So if you look at the
result in either the server log or in
tmp_check/log/regress_log_002_pg_dump, it looks like a complete mess.
This patch makes the output look cleaner for manual inspection: add
semicolon only if necessary, and add two newlines.

Discussion: https://www.postgresql.org/message-id/flat/d6aec95a-8729-43cc-2578-f2a5e46640e0%40enterprisedb.com
2022-10-31 14:19:08 +01:00
Michael Paquier a73952b795 Add check on initial and boot values when loading GUCs
This commit adds a function to perform a cross-check between the initial
value of the C declaration associated to a GUC and its actual boot
value in assert-enabled builds.  The purpose of this is to prevent
anybody reading these C declarations from being fooled by mismatched
values before they are loaded at program startup.

The following rules apply depending on the GUC type:
* bool - can be false, or same as boot_val.
* int - can be 0, or same as the boot_val.
* real - can be 0.0, or same as the boot_val.
* string - can be NULL, or strcmp'd equal to the boot_val.
* enum - equal to the boot_val.

This is done for the system as well custom GUCs loaded by external
modules, which may require extension developers to adapt the C
declaration of the variables used by these GUCs (testing this change
with some of my own modules has allowed me to catch some stupid typos,
FWIW).  This may finish by being a bad experiment depending on the
feedbcak received, but let's see how it goes.

Author: Peter Smith
Reviewed-by: Nathan Bossart, Tom Lane, Michael Paquier, Justin Pryzby
Discussion: https://postgr.es/m/CAHut+PtHE0XSfjjRQ6D4v7+dqzCw=d+1a64ujra4EX8aoc_Z+w@mail.gmail.com
2022-10-31 13:54:23 +09:00
Michael Paquier d9d873bac6 Clean up some inconsistencies with GUC declarations
This is similar to 7d25958, and this commit takes care of all the
remaining inconsistencies between the initial value used in the C
variable associated to a GUC and its default value stored in the GUC
tables (as of pg_settings.boot_val).

Some of the initial values of the GUCs updated rely on a compile-time
default.  These are refactored so as the GUC table and its C declaration
use the same values.  This makes everything consistent with other
places, backend_flush_after, bgwriter_flush_after, port,
checkpoint_flush_after doing so already, for example.

Extracted from a larger patch by Peter Smith.  The spots updated in the
modules are from me.

Author: Peter Smith, Michael Paquier
Reviewed-by: Nathan Bossart, Tom Lane, Justin Pryzby
Discussion: https://postgr.es/m/CAHut+PtHE0XSfjjRQ6D4v7+dqzCw=d+1a64ujra4EX8aoc_Z+w@mail.gmail.com
2022-10-31 12:44:48 +09:00
Noah Misch a9f8ca6005 Under has_wal_read_bug, skip recovery/t/032_relfilenode_reuse.pl.
Per buildfarm member kittiwake.  Back-patch to v15, where this test
first appeared.

Discussion: https://postgr.es/m/20220116210241.GC756210@rfd.leadboat.com
2022-10-29 10:42:16 -07:00
David Rowley 5543677ec9 Use Limit instead of Unique to implement DISTINCT, when possible
When all of the query's DISTINCT pathkeys have been marked as redundant
due to EquivalenceClasses existing which contain constants, we can just
implement the DISTINCT operation on a query by just limiting the number of
returned rows to 1 instead of performing a Unique on all of the matching
(duplicate) rows.

This applies in cases such as:

SELECT DISTINCT col,col2 FROM tab WHERE col = 1 AND col2 = 10;

If there are any matching rows, then they must all be {1,10}.  There's no
point in fetching all of those and running a Unique operator on them to
leave only a single row.  Here we effectively just find the first row and
then stop.  We are obviously unable to apply this optimization if either
the col = 1 or col2 = 10 were missing from the WHERE clause or if there
were any additional columns in the SELECT clause.

Such queries are probably not all that common, but detecting when we can
apply this optimization amounts to checking if the distinct_pathkeys are
NULL, which is very cheap indeed.

Nothing is done here to check if the query already has a LIMIT clause.  If
it does then the plan may end up with 2 Limits nodes.  There's no harm in
that and it's probably not worth the complexity to unify them into a
single Limit node.

Author: David Rowley
Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/CAApHDvqS0j8RUWRUSgCAXxOqnYjHUXmKwspRj4GzVfOO25ByHA@mail.gmail.com
Discussion: https://postgr.es/m/MEYPR01MB7101CD5DA0A07C9DE2B74850A4239@MEYPR01MB7101.ausprd01.prod.outlook.com
2022-10-28 23:04:38 +13:00
Peter Eisentraut b1099eca8f Remove AssertArg and AssertState
These don't offer anything over plain Assert, and their usage had
already been declared obsolescent.

Author: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://www.postgresql.org/message-id/20221009210148.GA900071@nathanxps13
2022-10-28 09:19:06 +02:00
David Rowley d37aa3d358 Allow nodeSort to perform Datum sorts for byref types
Here we add a new 'copy' parameter to tuplesort_getdatum so that we can
instruct the function not to datumCopy() byref Datums before returning.

Similar to 91e9e89dc, this can provide significant performance
improvements in nodeSort when sorting by a single byref column and the
sort's targetlist contains only that column.

This allows us to re-enable Datum sorts for byref types which was disabled
in 3a5817695 due to a reported memory leak.

Additionally, here we slightly optimize DISTINCT aggregates so that we no
longer perform any datumCopy() when we find the current value not to be
distinct from the previous value.  Previously the code would always take a
copy of the most recent Datum and pfree the previous value, even when the
values were the same.  Testing shows a small but noticeable performance
increase when aggregate transitions are skipped due to the current
transition value being the same as the prior one.

Author: David Rowley
Discussion: https://postgr.es/m/CAApHDvqS6wC5U==k9Hd26E4EQXH3QR67-T4=Q1rQ36NGvjfVSg@mail.gmail.com
Discussion: https://postgr.es/m/CAApHDvqHonfe9G1cVaKeHbDx70R_zCrM3qP2AGXpGrieSKGnhA@mail.gmail.com
2022-10-28 09:25:12 +13:00
Tom Lane a5fc46414d Avoid making commutatively-duplicate clauses in EquivalenceClasses.
When we decide we need to make a derived clause equating a.x and
b.y, we already will re-use a previously-made clause "a.x = b.y".
But we might instead have "b.y = a.x", which is perfectly usable
because equivclass.c has never promised anything about the
operand order in clauses it builds.  Saving construction of a
new RestrictInfo doesn't matter all that much in itself --- but
because we cache selectivity estimates and so on per-RestrictInfo,
there's a possibility of saving a fair amount of duplicative
effort downstream.

Hence, check for commutative matches as well as direct ones when
seeing if we have a pre-existing clause.  This changes the visible
clause order in several regression test cases, but they're all
clearly-insignificant changes.

Checking for the reverse operand order is simple enough, but
if we wanted to check for operator OID match we'd need to call
get_commutator here, which is not so cheap.  I concluded that
we don't really need the operator check anyway, so I just
removed it.  It's unlikely that an opfamily contains more than
one applicable operator for a given pair of operand datatypes;
and if it does they had better give the same answers, so there
seems little need to insist that we use exactly the one
select_equality_operator chose.

Using the current core regression suite as a test case, I see
this change reducing the number of new join clauses built by
create_join_clause from 9673 to 5142 (out of 26652 calls).
So not quite 50% savings, but pretty close to it.

Discussion: https://postgr.es/m/78062.1666735746@sss.pgh.pa.us
2022-10-27 14:42:18 -04:00
Michael Paquier 4ab8c81bd9 Move pg_pwritev_with_retry() to src/common/file_utils.c
This commit moves pg_pwritev_with_retry(), a convenience wrapper of
pg_writev() able to handle partial writes, to common/file_utils.c so
that the frontend code is able to use it.  A first use-case targetted
for this routine is pg_basebackup and pg_receivewal, for the
zero-padding of a newly-initialized WAL segment.  This is used currently
in the backend when the GUC wal_init_zero is enabled (default).

Author: Bharath Rupireddy
Reviewed-by: Nathan Bossart, Thomas Munro
Discussion: https://postgr.es/m/CALj2ACUq7nAb7=bJNbK3yYmp-SZhJcXFR_pLk8un6XgDzDF3OA@mail.gmail.com
2022-10-27 14:39:42 +09:00
Michael Paquier 1b9cd69c5b Add some tests to check the SQL functions of control file
As the recent commit 05d4cbf (reverted after as a448e49) has proved,
there is zero coverage for the four SQL functions that can scan the
control file data:
- pg_control_checkpoint()
- pg_control_init()
- pg_control_recovery()
- pg_control_system()

This commit adds a minimal coverage for these functions, checking that
their execution is able to complete.  This would have been enough to
catch the problems introduced in the commit mentioned above.  More
checks could be done for each individual fields, but it is unclear
whether this would be better than the other checks in place in the
backend code.

Per discussion with Bharath Rupireddy.

Discussion: https://postgr.es/m/Y1d2FZmQmyAhPSRG@paquier.xyz
2022-10-27 09:58:44 +09:00
Michael Paquier c591300a8f Add rule_number to pg_hba_file_rules and map_number to pg_ident_file_mappings
These numbers are strictly-monotone identifiers assigned to each rule
of pg_hba_file_rules and each map of pg_ident_file_mappings when loading
the HBA and ident configuration files, indicating the order in which
they are checked at authentication time, until a match is found.

With only one file loaded currently, this is equivalent to the line
numbers assigned to the entries loaded if one wants to know their order,
but this becomes mandatory once the inclusion of external files is
added to the HBA and ident files to be able to know in which order the
rules and/or maps are applied at authentication.  Note that NULL is used
when a HBA or ident entry cannot be parsed or validated, aka when an
error exists, contrary to the line number.

Bump catalog version.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya@jrouhaud
2022-10-26 15:22:15 +09:00
Michael Paquier 37d264478a Fix variable assignment thinko in hba.c
The intention behind 1b73d0b was to limit the use of TokenizedAuthLine,
but I have fat-fingered one location in parse_hba_line() when creating
the HbaLine, where this should use the local variable and not the value
coming from TokenizedAuthLine.  This logic is the exactly the same, but
let's be clean about all that on consistency grounds.

Reported-by: Julien Rouhaud
Discussion: https://postgr.es/m/20221026032730.k3sib5krgm7l6njk@jrouhaud
2022-10-26 12:57:40 +09:00
Michael Paquier 1b73d0b1c3 Refactor code handling the names of files loaded in hba.c
This has the advantage to limit the presence of the GUC values
hba_file and ident_file to the code paths where these files are loaded,
easing the introduction of an upcoming feature aimed at adding inclusion
logic for files and directories in HBA and ident files.

Note that this needs the addition of the source file name to HbaLine, in
addition to the line number, which is something needed by the backend in
two places of auth.c (authentication failure details and auth_id log
when log_connections is enabled).

While on it, adjust a log generated on authentication failure to report
the name of the actual HBA file on which the connection attempt matched,
where the line number and the raw line written in the HBA file were
already included.  This was previously hardcoded as pg_hba.conf, which
would be incorrect when a custom value is used at postmaster startup for
the GUC hba_file.

Extracted from a larger patch by the same author.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya@jrouhaud
2022-10-26 11:42:13 +09:00
Tom Lane 13d53aa7a8 Doc/improve confusing, inefficient tests to locate CTID variable.
The IsCTIDVar() tests in nodeTidscan.c and nodeTidrangescan.c
look buggy at first sight: they aren't checking that the varno
matches the table to be scanned.  Actually they're safe because
any Var in a scan-level qual must be for the correct table ...
but if we're depending on that, it's pretty pointless to verify
varlevelsup.  (Besides which, varlevelsup is *always* zero at
execution, since we've flattened the rangetable long since.)

Remove the useless varlevelsup check, and instead add some
commentary explaining why we don't need to check varno.

Noted while fooling with a planner change that causes the order
of "t1.ctid = t2.ctid" to change in some tidscan.sql tests;
I was briefly fooled into thinking there was a live bug here.
2022-10-25 17:35:19 -04:00
Heikki Linnakangas 0e972f50fd Update outdated comment for TransactionIdSetTreeStatus
Commit 06da3c570f changed the way subtransactions are marked as
SUBCOMMITTED, but the example it included actually documented the old
way. Update it.

Author: Japin Li
Discussion: https://www.postgresql.org/message-id/MEYP282MB16690BC96DFBE08CC857E1E3B6319%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
2022-10-25 21:43:52 +02:00
Michael Paquier 7d25958453 Clean up some GUC declarations and comments
This adjusts a few things for GUCs related to logical replication,
replication slots and WAL senders, in the shape of incorrect comments
and values inconsistent with their initial default value.

Author: Peter Smith
Reviewed-by: Nathan Bossart, Tom Lane, Justin Pryzby
Discussion: https://postgr.es/m/CAHut+PtHE0XSfjjRQ6D4v7+dqzCw=d+1a64ujra4EX8aoc_Z+w@mail.gmail.com
2022-10-25 14:06:07 +09:00
Thomas Munro e109e43921 Fix unlink() for STATUS_DELETE_PENDING on Windows.
Commit f357233c assumed that it was OK to return ENOENT directly if
lstat() failed that way.  If we got STATUS_DELETE_PENDING while trying
to unlink a file that we had already unlinked successfully once before
but someone else still had open (on a kernel version that has "pending"
unlinks by default), then we would no longer reach the retry loop in
pgunlink().  That loop claims to be only for handling sharing violations
(a different phenomenon), but the errno is the same.

Restore that behavior with an explicit check, to see if it fixes the
occasional 'directory not empty' failures seen in the pg_upgrade tests
on CI.  Further improvements are possible with proposed upgrades to
modern Windows APIs that would replace this convoluted code.

Reported-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/20220920013122.GA31833%40telsasoft.com
Discussion: https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 16:26:58 +13:00
Thomas Munro 4517358ee7 Fix stat() for recursive junction points on Windows.
Commit c5cb8f3b supposed that we'd only ever have to follow one junction
point in stat(), because we don't construct longer chains of them ourselves.
When examining a parent directory supplied by the user, we should really be
able to cope with longer chains, just in case someone has their system
set up that way.  Choose an arbitrary cap of 8, to match the minimum
acceptable value of SYMLOOP_MAX in POSIX.

Previously I'd avoided reporting ELOOP thinking Windows didn't have it,
but it turns out that it does, so we can use the proper error number.

Reviewed-by: Roman Zharkov <r.zharkov@postgrespro.ru>
Discussion: https://postgr.es/m/CA%2BhUKGJ7JDGWYFt9%3D-TyJiRRy5q9TtPfqeKkneWDr1XPU1%2Biqw%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 16:19:05 +13:00
Thomas Munro f71007fbb3 Fix readlink() for non-PostgreSQL junction points on Windows.
Since commit c5cb8f3b taught stat() to follow symlinks, and since initdb
uses pg_mkdir_p(), and that examines parent directories, our humble
readlink() implementation can now be exposed to junction points not of
PostgreSQL origin.  Those might be corrupted by our naive path mangling,
which doesn't really understand NT paths in general.

Simply decline to transform paths that don't look like a drive absolute
path.  That means that readlink() returns the NT path directly when
checking a parent directory of PGDATA that happen to point to a drive
using "rooted" format.  That  works for the purposes of our stat()
emulation.

Reported-by: Roman Zharkov <r.zharkov@postgrespro.ru>
Reviewed-by: Roman Zharkov <r.zharkov@postgrespro.ru>
Discussion: https://postgr.es/m/4590c37927d7b8ee84f9855d83229018%40postgrespro.ru
Discussion: https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 16:19:05 +13:00
Thomas Munro 387803d81d Fix lstat() for broken junction points on Windows.
When using junction points to emulate symlinks on Windows, one edge case
was not handled correctly by commit c5cb8f3b: if a junction point is
broken (pointing to a non-existent path), we'd report ENOENT.  This
doesn't break any known use case, but was noticed while developing a
test suite for these functions and is fixed here for completeness.

Also add translation ERROR_CANT_RESOLVE_FILENAME -> ENOENT, as that is
one of the errors Windows can report for some kinds of broken paths.

Discussion: https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 16:19:01 +13:00
Thomas Munro 4650036f5a Fix readlink() return value on Windows.
Ancient bug noticed while working on a test suite for these functions.

Discussion: https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 15:13:52 +13:00
Thomas Munro 359d601095 Fix symlink() errno on Windows.
Ancient bug noticed while working on a test suite for these functions.

https://postgr.es/m/CA%2BhUKG%2BajSQ_8eu2AogTncOnZ5me2D-Cn66iN_-wZnRjLN%2Bicg%40mail.gmail.com
2022-10-25 15:10:49 +13:00
Michael Paquier f186c7c885 doc: Fix type of cursor_position in jsonlog table
This entry was listed as a "string", but it is a "number.  The other
fields are correctly described, on a second look.

Reported-by: Nuko Yokohama
Author: Tatsuo Ishii
Discussion: https://postgr.es/m/CAF3Gu1awoVoDP5d0_eN=cR=QkGVwH+OtFvwJkkc5cB_ZMWjyeA@mail.gmail.com
Backpatch-through: 15
2022-10-25 09:29:21 +09:00
Alvaro Herrera 3b2db22fe2
Update some comments that should've covered MERGE
Oversight in 7103ebb7aa.  Backpatch to 15.

Author: Richard Guo <guofenglinux@gmail.com>
Discussion: https://postgr.es/m/CAMbWs48gnDjZXq3-b56dVpQCNUJ5hD9kdtWN4QFwKCEapspNsA@mail.gmail.com
2022-10-24 12:52:43 +02:00
Alvaro Herrera 8328a15f8f
Fix recently added incorrect assertion
Commit df3737a651 added an incorrect assertion about the preconditions
for invoking the backup cleanup callback: it misfires at session end in
case a backup completes successfully.  Fix it, using coding from Michaël
Paquier.  Also add some tests for the various cases.

Reported by Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Discussion: https://postgr.es/m/20221021.161038.1277961198945653224.horikyota.ntt@gmail.com
2022-10-24 12:02:33 +02:00
Michael Paquier 2e0d80c5bb Improve coverage of ruleutils.c for SQLValueFunctions
While looking at how these are handled in the parser and the executor, I
have noticed that there is no test coverage for most of these when
reverse-engineering an expression for a SQLValueFunction node in
ruleutils.c, including how these are reparsed when included in a FROM
clause.  Some hacking in this area has showed me that these could break
easily, so add some coverage to track the existing compatibility.

Extracted from a much larger patch by me.

Discussion: https://postgr.es/m/YzaG3MoryCguUOym@paquier.xyz
2022-10-24 16:53:54 +09:00
Michael Paquier 3cf2f7af7f Improve tab completion for ALTER STATISTICS <name> SET in psql
The code was completing this pattern with a list of settable characters,
and it was possible to reach this state after completing a "ALTER
STATISTICS <name>" with SET.

Author: Vignesh C
Discussion: https://postgr.es/m/CALDaNm2HHF_371o+EeSjxDDS17Cx7d-ko2h1fLU94=ob=4_ktg@mail.gmail.com
2022-10-24 15:46:42 +09:00
Michael Paquier 14a737bfdb Fix and improve TAP tests for pg_hba.conf and regexps
The new tests have been reporting a warning hidden in the logs, as of
"Odd number of elements in hash assignment" (perlcritic or similar did
not report an issue, actually).  This comes down to a typo in the test
"matching regexp for username" for a double-quoted regexp using commas,
where we passed an extra argument.  The test is intended to pass, but
this was causing the test to fail.  This also pointed out that the
newly-added role "md5,role" lacks an entry in the password file used to
provide the password, so add one.

While on it, make the tests pickier by checking the contents of the logs
generated on successful authentication.

Oversights in 8fea868.
2022-10-24 13:56:34 +09:00
Michael Paquier 8fea86830e Add support for regexps on database and user entries in pg_hba.conf
As of this commit, any database or user entry beginning with a slash (/)
is considered as a regular expression.  This is particularly useful for
users, as now there is no clean way to match pattern on multiple HBA
lines.  For example, a user name mapping with a regular expression needs
first to match with a HBA line, and we would skip the follow-up HBA
entries if the ident regexp does *not* match with what has matched in
the HBA line.

pg_hba.conf is able to handle multiple databases and roles with a
comma-separated list of these, hence individual regular expressions that
include commas need to be double-quoted.

At authentication time, user and database names are now checked in the
following order:
- Arbitrary keywords (like "all", the ones beginning by '+' for
membership check), that we know will never have a regexp.  A fancy case
is for physical WAL senders, we *have* to only match "replication" for
the database.
- Regular expression matching.
- Exact match.
The previous logic did the same, but without the regexp step.

We have discussed as well the possibility to support regexp pattern
matching for host names, but these happen to lead to tricky issues based
on what I understand, particularly with host entries that have CIDRs.

This commit relies heavily on the refactoring done in a903971 and
fc579e1, so as the amount of code required to compile and execute
regular expressions is now minimal.  When parsing pg_hba.conf, all the
computed regexps needs to explicitely free()'d, same as pg_ident.conf.

Documentation and TAP tests are added to cover this feature, including
cases where the regexps use commas (for clarity in the docs, coverage
for the parsing logic in the tests).

Note that this introduces a breakage with older versions, where a
database or user name beginning with a slash are treated as something to
check for an equal match.  Per discussion, we have discarded this as
being much of an issue in practice as it would require a cluster to
have database and/or role names that begin with a slash, as well as HBA
entries using these.  Hence, the consistency gained with regexps in
pg_ident.conf is more appealing in the long term.

**This compatibility change should be mentioned in the release notes.**

Author: Bertrand Drouvot
Reviewed-by: Jacob Champion, Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/fff0d7c1-8ad4-76a1-9db3-0ab6ec338bf7@amazon.com
2022-10-24 11:45:31 +09:00
Peter Eisentraut 5035c93c8a Remove pgpid_t type, use pid_t instead
It's unclear why a separate type would be needed here.  We use plain
pid_t (or int) everywhere else.

(The only relevant platform where pid_t is not int is 64-bit MinGW,
where it is long long int.  So defining pid_t as long (which is 32-bit
on Windows), as was done here, doesn't even accommodate that one.)

Reverts 66fa6eba5a.

Discussion: https://www.postgresql.org/message-id/289c2e45-c7d9-5ce4-7eff-a9e2a33e1580@enterprisedb.com
2022-10-22 10:45:19 +02:00
Peter Eisentraut 2598b76bf2 psql: Fix exit status when query is canceled
Because of a small thinko in 7844c9918a,
psql -c would exit successfully when a query is canceled.  Fix this so
that it exits with a nonzero status, just like for all other errors.
2022-10-22 09:42:52 +02:00
Michael Paquier 6cc66197ff Improve memory handling across SQL-callable backup functions
Since pg_backup_start() and pg_backup_stop() exist, the tablespace map
data and the backup state data (backup_label string until 7d70809) have
been allocated in the TopMemoryContext.  This approach would cause
memory leaks in the session calling these functions if failures happen
before pg_backup_stop() ends, leaking more memory on repeated failures.
Both things need little memory so that would not be really noticeable
for most users, except perhaps connection poolers with long-lived
connections able to trigger backup failures with these functions.

This commit improves the logic in this area by not allocating anymore
the backup-related data that needs to travel across the SQL-callable
backup functions in TopMemoryContext, by using instead a dedicated
memory context child of TopMemoryContext.  The memory context is created
in pg_backup_start() and deleted when finishing pg_backup_stop().  In
the event of an in-flight failure, this memory context gets reset in the
follow-up pg_backup_start() call, so as we are sure that only one run
worth of data is leaked at any time.  Some cleanup was already done for
the backup data on a follow-up call of pg_backup_start(), but using a
memory context makes the whole simpler.

BASE_BACKUP commands are executed in isolation, relying on the memory
context created for replication commands, hence these do not need such
an extra logic.

Author: Bharath Rupireddy
Reviewed-by: Robert Haas, Alvaro Herrera, Cary Huang, Michael Paquier
Discussion: https://postgr.es/m/CALj2ACXqvfKF2B0beQ=aJMdWnpNohmBPsRg=EDQj_6y1t2O8mQ@mail.gmail.com
2022-10-22 11:54:02 +09:00
Robert Haas 1f0c4fa255 pg_basebackup: Fix cross-platform tablespace relocation.
Specifically, when pg_basebackup is invoked with -Tx=y, don't error
out if x could plausibly be an absolute path either on Windows or on
non-Windows systems. We don't know whether the remote system is
running the same OS as the local system, so it's not appropriate to
assume that our local rule about absolute pathnames is the same as
the rule on the remote system.

Patch by me, reviewed by Tom Lane, Andrew Dunstan, and
Davinder Singh.

Discussion: http://postgr.es/m/CA+TgmoY+jC3YiskomvYKDPK3FbrmsDU7_8+wMHt02HOdJeRb0g@mail.gmail.com
2022-10-21 08:21:55 -04:00
Amit Kapila ce20f8b9f4 Add CHECK_FOR_INTERRUPTS while restoring changes during decoding.
Previously in commit 42681dffaf, we added CFI during decoding changes but
missed another similar case that can happen while restoring changes
spilled to disk back into memory in a loop.

Reported-by: Robert Haas
Author: Amit Kapila
Backpatch-through: 10
Discussion: https://postgr.es/m/CA+TgmoaLObg0QbstbC8ykDwOdD1bDkr4AbPpB=0DPgA2JW0mFg@mail.gmail.com
2022-10-21 12:57:18 +05:30