Commit Graph

16241 Commits

Author SHA1 Message Date
Tom Lane b9424d014e Support writing "CREATE/ALTER TABLE ... SET STORAGE DEFAULT".
We already allow explicitly writing DEFAULT for SET COMPRESSION,
so it seems a bit inflexible and non-orthogonal to not have it
for STORAGE.

Aleksander Alekseev

Discussion: https://postgr.es/m/CAJ7c6TMX9ui+6y3TQFaXJYVpZyBukvqhQbVDJ8OUokeLRhtnpA@mail.gmail.com
2022-11-10 18:20:49 -05:00
Tom Lane f05a5e0003 Doc: improve tutorial section about grouped aggregates.
Commit fede15417 introduced FILTER by jamming it into the existing
example introducing HAVING, which seems pedagogically poor to me;
and it added no information about what the keyword actually does.
Not to mention that the claimed output didn't match the sample
data being used in this running example.

Revert that and instead make an independent example using FILTER.
To help drive home the point that it's a per-aggregate filter,
we need to use two aggregates not just one; for consistency
expand all the examples in this segment to do that.

Also adjust the example using WHERE ... LIKE so that it'd produce
nonempty output with this sample data, and show that output.

Back-patch, as the previous patch was.  (Sadly, v10 is now out
of scope.)

Discussion: https://postgr.es/m/166794307526.652.9073408178177444190@wrigleys.postgresql.org
2022-11-08 18:25:03 -05:00
Amit Kapila 3e77b2eb34 Doc: Improve the description of confirmed_flush_lsn in pg_replication_slots.
Make it clear that the data corresponding to the transactions committed
before confirmed_flush_lsn is not available anymore.

Author: Ashutosh Sharma
Reviewd by: Ashutosh Bapat
Discussion: https://postgr.es/m/CAE9k0P=hiqRXUonnmtS-5Pu8SbO=yF6vcrVBcfEf2+93ng_f5Q@mail.gmail.com
2022-11-04 09:00:46 +05:30
Amit Kapila 155601de7f Doc: Update information about manually creating slots.
There are some cases (e.g. when the subscription is created using the
connect = false option) where the remote replication slot was not created
automatically and the user must create it manually before the subscription
can be activated. There was not enough information in the docs for users
to do this easily.

Author: Peter Smith
Reviewd by: Shi yu, Amit Kapila
Discussion: https://postgr.es/m/CAHut+PvqdqOanheWSHDyhQiF+Z-7w=-+k4U+bwbT=b6YQ_hrXQ@mail.gmail.com
2022-11-02 11:40:37 +05:30
Michael Paquier e76502871e doc: Fix some descriptions related to pg_ident_file_mappings
pg_ident_file_mappings.line_number was described as a line number in
pg_ident.conf for a "rule" number, but this should refer to a "map".
The same inconsistent term was used in the main paragraph describing the
view.

Extracted from a patch by the same author.  Issue introduced by
a2c8499 where this view has been added.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20221026031948.cbrnzgy5e7glsq2d@jrouhaud
Backpatch-through: 15
2022-11-02 11:56:06 +09:00
Peter Eisentraut 84387fc889 doc: Add note about lack of publication privileges
This gives some additional advice on using row filters and column
lists on publications securely.

Author: Antonin Houska <ah@cybertec.at>
Reviewed-by: Euler Taveira <euler@eulerto.com>
Discussion: https://www.postgresql.org/message-id/flat/20330.1652105397@antos
2022-11-01 14:19:41 +01:00
Michael Paquier c591300a8f Add rule_number to pg_hba_file_rules and map_number to pg_ident_file_mappings
These numbers are strictly-monotone identifiers assigned to each rule
of pg_hba_file_rules and each map of pg_ident_file_mappings when loading
the HBA and ident configuration files, indicating the order in which
they are checked at authentication time, until a match is found.

With only one file loaded currently, this is equivalent to the line
numbers assigned to the entries loaded if one wants to know their order,
but this becomes mandatory once the inclusion of external files is
added to the HBA and ident files to be able to know in which order the
rules and/or maps are applied at authentication.  Note that NULL is used
when a HBA or ident entry cannot be parsed or validated, aka when an
error exists, contrary to the line number.

Bump catalog version.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya@jrouhaud
2022-10-26 15:22:15 +09:00
Michael Paquier f186c7c885 doc: Fix type of cursor_position in jsonlog table
This entry was listed as a "string", but it is a "number.  The other
fields are correctly described, on a second look.

Reported-by: Nuko Yokohama
Author: Tatsuo Ishii
Discussion: https://postgr.es/m/CAF3Gu1awoVoDP5d0_eN=cR=QkGVwH+OtFvwJkkc5cB_ZMWjyeA@mail.gmail.com
Backpatch-through: 15
2022-10-25 09:29:21 +09:00
Michael Paquier 8fea86830e Add support for regexps on database and user entries in pg_hba.conf
As of this commit, any database or user entry beginning with a slash (/)
is considered as a regular expression.  This is particularly useful for
users, as now there is no clean way to match pattern on multiple HBA
lines.  For example, a user name mapping with a regular expression needs
first to match with a HBA line, and we would skip the follow-up HBA
entries if the ident regexp does *not* match with what has matched in
the HBA line.

pg_hba.conf is able to handle multiple databases and roles with a
comma-separated list of these, hence individual regular expressions that
include commas need to be double-quoted.

At authentication time, user and database names are now checked in the
following order:
- Arbitrary keywords (like "all", the ones beginning by '+' for
membership check), that we know will never have a regexp.  A fancy case
is for physical WAL senders, we *have* to only match "replication" for
the database.
- Regular expression matching.
- Exact match.
The previous logic did the same, but without the regexp step.

We have discussed as well the possibility to support regexp pattern
matching for host names, but these happen to lead to tricky issues based
on what I understand, particularly with host entries that have CIDRs.

This commit relies heavily on the refactoring done in a903971 and
fc579e1, so as the amount of code required to compile and execute
regular expressions is now minimal.  When parsing pg_hba.conf, all the
computed regexps needs to explicitely free()'d, same as pg_ident.conf.

Documentation and TAP tests are added to cover this feature, including
cases where the regexps use commas (for clarity in the docs, coverage
for the parsing logic in the tests).

Note that this introduces a breakage with older versions, where a
database or user name beginning with a slash are treated as something to
check for an equal match.  Per discussion, we have discarded this as
being much of an issue in practice as it would require a cluster to
have database and/or role names that begin with a slash, as well as HBA
entries using these.  Hence, the consistency gained with regexps in
pg_ident.conf is more appealing in the long term.

**This compatibility change should be mentioned in the release notes.**

Author: Bertrand Drouvot
Reviewed-by: Jacob Champion, Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/fff0d7c1-8ad4-76a1-9db3-0ab6ec338bf7@amazon.com
2022-10-24 11:45:31 +09:00
David Rowley 42d01f59c7 Doc: fix outdated wording about parallel seq scans
56788d215 adjusted the parallel seq scan code so that instead of handing
out a single block at a time to parallel workers, it now hands out ranges
of blocks.

Here we update the documentation which still claimed that workers received
just 1 block at a time.

Reported-by: Zhang Mingli
Discussion: https://postgr.es/m/17c99615-2c3b-4e4e-9d0b-424a66a7bccd@Spark
Backpatch-through: 14, where 56788d215 was added.
2022-10-21 09:29:08 +13:00
Peter Eisentraut c8e4030d1b Make finding openssl program a configure or meson option
Various test suites use the "openssl" program as part of their setup.
There isn't a way to override which openssl program is to be used,
other than by fiddling with the path, perhaps.  This has gotten
increasingly problematic because different versions of openssl have
different capabilities and do different things by default.

This patch checks for an openssl binary in configure and meson setup,
with appropriate ways to override it.  This is similar to how "lz4"
and "zstd" are handled, for example.  The meson build system actually
already did this, but the result was only used in some places.  This
is now applied more uniformly.

Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://www.postgresql.org/message-id/flat/dc638b75-a16a-007d-9e1c-d16ed6cf0ad2%40enterprisedb.com
2022-10-20 21:05:42 +02:00
Michael Paquier 9668c4a661 Rework shutdown callback of archiver modules
As currently designed, with a callback registered in a ERROR_CLEANUP
block, the shutdown callback would get called twice when updating
archive_library on SIGHUP, which is something that we want to avoid to
ease the life of extension writers.

Anyway, an ERROR in the archiver process is treated as a FATAL, stopping
it immediately, hence there is no need for a ERROR_CLEANUP block.
Instead of that, the shutdown callback is not called upon
before_shmem_exit(), giving to the modules the opportunity to do any
cleanup actions before the server shuts down its subsystems.

While on it, this commit adds some testing coverage for the shutdown
callback.  Neither shell_archive nor basic_archive have been using it,
and one is added to shell_archive, whose trigger is checked in a TAP
test through a shutdown sequence.

Author: Nathan Bossart, Bharath Rupireddy
Reviewed-by: Kyotaro Horiguchi, Michael Paquier
Discussion: https://postgr.es/m/20221015221328.GB1821022@nathanxps13
Backpatch-through: 15
2022-10-19 14:06:56 +09:00
Bruce Momjian 77dd153d39 doc: move the mention of aggregate JSON functions up in section
It was previously easily overlooked at the end of several tables.

Reported-by: Alex Denman

Discussion: https://postgr.es/m/166335888474.659.16897487975376230364@wrigleys.postgresql.org

Backpatch-through: 10
2022-10-17 15:21:29 -04:00
Bruce Momjian 4d070469c1 doc: warn pg_stat_reset() can cause vacuum/analyze problems
The fix is to run ANALYZE.

Discussion: https://postgr.es/m/YzRr+ys98UzVQJvK@momjian.us,
   https://postgr.es/m/flat/CAKJS1f8DTbCHf9gedU0He6ARsd58E6qOhEHM1caomqj_r9MOiQ%40mail.gmail.com,
   https://postgr.es/m/CAKJS1f80o98hcfSk8j%3DfdN09S7Sjz%2BvuzhEwbyQqvHJb_sZw0g%40mail.gmail.com

Backpatch-through: 10
2022-10-17 15:07:03 -04:00
Andres Freund c037471832 pgstat: Track time of the last scan of a relation
It can be useful to know when a relation has last been used, e.g., when
evaluating whether an index is still required. It was already possible to
infer the time of the last usage by tracking, e.g.,
pg_stat_all_indexes.idx_scan over time. But far from everybody does so.

To make it easier to detect the last time a relation has been scanned, track
that time in each relation's pgstat entry. To minimize overhead a) the
timestamp is updated only when the backend pending stats entry is flushed to
shared stats b) the last transaction's stop timestamp is used as the
timestamp.

Bumps catalog and stats format versions.

Author: Dave Page <dpage@pgadmin.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Bruce Momjian <bruce@momjian.us>
Reviewed-by: Vik Fearing <vik@postgresfriends.org>
Discussion: https://postgr.es/m/CA+OCxozrVHNFVEPkweUHMZje+t1tfY816d9MZYc6eZwOOusOaQ@mail.gmail.com
2022-10-14 11:11:34 -07:00
Peter Eisentraut 34df7b9dfd doc: Correct type of bgw_notify_pid
This has apparently been wrong since the beginning
(090d0f2050).

Discussion: https://www.postgresql.org/message-id/289c2e45-c7d9-5ce4-7eff-a9e2a33e1580@enterprisedb.com
2022-10-14 08:38:49 +02:00
Andres Freund 2589434ae0 pg_buffercache: Add pg_buffercache_summary()
Using pg_buffercache_summary() is significantly cheaper than querying
pg_buffercache and summarizing in SQL.

Author: Melih Mutlu <m.melihmutlu@gmail.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Aleksander Alekseev <aleksander@timescale.com>
Reviewed-by: Zhang Mingli <zmlpostgres@gmail.com>
Discussion: https://postgr.es/m/CAGPVpCQAXYo54Q%3D8gqBsS%3Du0uk9qhnnq4%2B710BtUhUisX1XGEg%40mail.gmail.com
2022-10-13 09:55:46 -07:00
Alvaro Herrera 7f8d9cedb3
Fix typo in CREATE PUBLICATION reference page
While at it, simplify wording a bit.

Author: Takamichi Osumi <osumi.takamichi@fujitsu.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Discussion: https://postgr.es/m/TYCPR01MB8373F93F5D094A2BE648990DED259@TYCPR01MB8373.jpnprd01.prod.outlook.com
2022-10-13 13:36:14 +02:00
Etsuro Fujita 97da48246d Allow batch insertion during COPY into a foreign table.
Commit 3d956d956 allowed the COPY, but it's done by inserting individual
rows to the foreign table, so it can be inefficient due to the overhead
caused by each round-trip to the foreign server.  To improve performance
of the COPY in such a case, this patch allows batch insertion, by
extending the multi-insert machinery in CopyFrom() to the foreign-table
case so that we insert multiple rows to the foreign table at once using
the FDW callback routine added by commit b663a4136.  This patch also
allows this for postgres_fdw.  It is enabled by the "batch_size" option
added by commit b663a4136, which is disabled by default.

When doing batch insertion, we update progress of the COPY command after
performing the FDW callback routine, to count rows not suppressed by the
FDW as well as a BEFORE ROW INSERT trigger.  For consistency, this patch
changes the timing of updating it for plain tables: previously, we
updated it immediately after adding each row to the multi-insert buffer,
but we do so only after writing the rows stored in the buffer out to the
table using table_multi_insert(), which I think would be consistent even
with non-batching mode, because in that mode we update it after writing
each row out to the table using table_tuple_insert().

Andrey Lepikhov, heavily revised by me, with review from Ian Barwick,
Andrey Lepikhov, and Zhihong Yu.

Discussion: https://postgr.es/m/bc489202-9855-7550-d64c-ad2d83c24867%40postgrespro.ru
2022-10-13 18:45:00 +09:00
Michael Paquier 63585b1ebd doc: Fix description of replication command CREATE_REPLICATION_SLOT
The output plugin name is a mandatory option when creating a logical
slot, but the grammar documented was not described as such.  While on
it, fix two comments in repl_gram.y to show that TEMPORARY is an
optional grammar choice.

Author: Ayaki Tachikake
Discussion: https://postgr.es/m/OSAPR01MB2852607B2329FFA27834105AF1229@OSAPR01MB2852.jpnprd01.prod.outlook.com
Backpatch-through: 15
2022-10-13 08:53:42 +09:00
Tom Lane 2d27e13b35 Doc: improve recommended systemd unit file.
Add
    After=network-online.target
    Wants=network-online.target
to the suggested unit file for starting a Postgres server.
This delays startup until the network interfaces have been
configured; without that, any attempt to bind to a specific
IP address will fail.

If listen_addresses is set to "localhost" or "*", it might be
possible to get away with the less restrictive "network.target",
but I don't think we need to get into such detail here.

Per suggestion from Pablo Federico.

Discussion: https://postgr.es/m/166552157407.591805.10036014441784710940@wrigleys.postgresql.org
2022-10-12 10:51:11 -04:00
Tom Lane 422f86a82e Doc: add entry for pg_get_partkeydef().
Other pg_get_XXXdef() functions are documented, so it seems reasonable
to include this as well.

Ian Barwick

Discussion: https://postgr.es/m/CAB8KJ=hb2QZXdgyrrRjPCw++DsrRcui4fKArWabQ+oij+2x=_w@mail.gmail.com
2022-10-11 14:28:43 -04:00
Bruce Momjian 23f3989d92 doc: clarify description for log_startup_progress_interval
Reported-by: Elena Indrupskaya

Discussion: https://postgr.es/m/0af43b49-1646-93d0-ccf1-bb3c635c8c6f@postgrespro.ru

Author: Elena Indrupskaya

Backpatch-through: 15
2022-10-05 15:53:40 -04:00
Andres Freund 29668e3186 meson: docs: Add xml{lint,proc} wrapper to collect dependencies
meson/ninja do not support specifying dependencies via globs (as those make it
significantly more expensive to check the current build state). Instead
targets should emit dependency information when running that then can be
cheaply re-checked during future builds.

To handle xmllint and xsltproc invocations in the docs, add and use a wrapper
that uses --load-trace to collect dependency information.

Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/c5736f70-bb6d-8d25-e35c-e3d886e4e905@enterprisedb.com
2022-10-05 09:56:05 -07:00
Tom Lane f4c7c410ee Revert "Optimize order of GROUP BY keys".
This reverts commit db0d67db24 and
several follow-on fixes.  The idea of making a cost-based choice
of the order of the sorting columns is not fundamentally unsound,
but it requires cost information and data statistics that we don't
really have.  For example, relying on procost to distinguish the
relative costs of different sort comparators is pretty pointless
so long as most such comparator functions are labeled with cost 1.0.
Moreover, estimating the number of comparisons done by Quicksort
requires more than just an estimate of the number of distinct values
in the input: you also need some idea of the sizes of the larger
groups, if you want an estimate that's good to better than a factor of
three or so.  That's data that's often unknown or not very reliable.
Worse, to arrive at estimates of the number of calls made to the
lower-order-column comparison functions, the code needs to make
estimates of the numbers of distinct values of multiple columns,
which are necessarily even less trustworthy than per-column stats.
Even if all the inputs are perfectly reliable, the cost algorithm
as-implemented cannot offer useful information about how to order
sorting columns beyond the point at which the average group size
is estimated to drop to 1.

Close inspection of the code added by db0d67db2 shows that there
are also multiple small bugs.  These could have been fixed, but
there's not much point if we don't trust the estimates to be
accurate in-principle.

Finally, the changes in cost_sort's behavior made for very large
changes (often a factor of 2 or so) in the cost estimates for all
sorting operations, not only those for multi-column GROUP BY.
That naturally changes plan choices in many situations, and there's
precious little evidence to show that the changes are for the better.
Given the above doubts about whether the new estimates are really
trustworthy, it's hard to summon much confidence that these changes
are better on the average.

Since we're hard up against the release deadline for v15, let's
revert these changes for now.  We can always try again later.

Note: in v15, I left T_PathKeyInfo in place in nodes.h even though
it's unreferenced.  Removing it would be an ABI break, and it seems
a bit late in the release cycle for that.

Discussion: https://postgr.es/m/TYAPR01MB586665EB5FB2C3807E893941F5579@TYAPR01MB5866.jpnprd01.prod.outlook.com
2022-10-03 10:56:16 -04:00
Michael Paquier 83e42a0035 doc: Fix some grammar and typos
This fixes some areas related to logical replication and custom RMGRs.

Author: Ekaterina Kiryanova
Discussion: https://postgr.es/m/fa4773f1-1396-384a-bcd7-85b5e013f399@postgrespro.ru
Backpatch-through: 15
2022-10-01 15:28:02 +09:00
Tom Lane 4e4f7b9fcc Adjust PQsslAttributeNames() to match PQsslAttribute().
Currently, PQsslAttributeNames() returns the same list of attribute
names regardless of its conn parameter.  This patch changes it to
have behavior parallel to what 80a05679d installed for PQsslAttribute:
you get OpenSSL's attributes if conn is NULL or is an SSL-encrypted
connection, or an empty list if conn is a non-encrypted connection.
The point of this is to have sensible connection-dependent behavior
in case we ever support multiple SSL libraries.  The behavior for
NULL can be defined as "the attributes for the default SSL library",
parallel to what PQsslAttribute(NULL, "library") does.

Since this is mostly just future-proofing, no back-patch.

Discussion: https://postgr.es/m/17625-fc47c78b7d71b534@postgresql.org
2022-09-30 10:26:47 -04:00
Daniel Gustafsson 51fd7cdfd0 doc: Fix PQsslAttribute docs for compression
The compression parameter to PQsslAttribute has never returned the
compression method used, it has always returned "on" or "off since
it was added in commit 91fa7b4719. Backpatch through v10.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/B9EC60EC-F665-47E8-A221-398C76E382C9@yesql.se
Backpatch-through: v10
2022-09-30 12:03:48 +02:00
Tom Lane 80a05679d5 Fix bogus behavior of PQsslAttribute(conn, "library").
Commit ebc8b7d44 intended to change the behavior of
PQsslAttribute(NULL, "library"), but accidentally also changed
what happens with a non-NULL conn pointer.  Undo that so that
only the intended behavior change happens.  Clarify some
associated documentation.

Per bug #17625 from Heath Lord.  Back-patch to v15.

Discussion: https://postgr.es/m/17625-fc47c78b7d71b534@postgresql.org
2022-09-29 17:28:09 -04:00
Tom Lane d7e39d72ca Use actual backend IDs in pg_stat_get_backend_idset() and friends.
Up to now, the ID values returned by pg_stat_get_backend_idset() and
used by pg_stat_get_backend_activity() and allied functions were just
indexes into a local array of sessions seen by the last stats refresh.
This is problematic for a few reasons.  The "ID" of a session can vary
over its existence, which is surprising.  Also, while these numbers
often match the "backend ID" used for purposes like temp schema
assignment, that isn't reliably true.  We can fairly cheaply switch
things around to make these numbers actually be the sessions' backend
IDs.  The added test case illustrates that with this definition, the
temp schema used by a given session can be obtained given its PID.

While here, delete some dead code that guarded against getting
a NULL return from pgstat_fetch_stat_local_beentry().  That can't
happen as long as the caller is careful to pass an in-range array
index, as all the callers are.  (This code may not have been dead
when written, but it surely is now.)

Nathan Bossart

Discussion: https://postgr.es/m/20220815205811.GA250990@nathanxps13
2022-09-29 12:14:39 -04:00
Michael Paquier 0823d061b0 Introduce SYSTEM_USER
SYSTEM_USER is a reserved keyword of the SQL specification that,
roughly described, is aimed at reporting some information about the
system user who has connected to the database server.  It may include
implementation-specific information about the means by the user
connected, like an authentication method.

This commit implements SYSTEM_USER as of auth_method:identity, where
"auth_method" is a keyword about the authentication method used to log
into the server (like peer, md5, scram-sha-256, gss, etc.) and
"identity" is the authentication identity as introduced by 9afffcb (peer
sets authn to the OS user name, gss to the user principal, etc.).  This
format has been suggested by Tom Lane.

Note that thanks to d951052, SYSTEM_USER is available to parallel
workers.

Bump catalog version.

Author: Bertrand Drouvot
Reviewed-by: Jacob Champion, Joe Conway, Álvaro Herrera, Michael Paquier
Discussion: https://postgr.es/m/7e692b8c-0b11-45db-1cad-3afc5b57409f@amazon.com
2022-09-29 15:05:40 +09:00
Bruce Momjian 2a66b9bfb7 doc: clarify internal behavior of RECURSIVE CTE queries
Reported-by: Tom Lane

Discussion: https://postgr.es/m/3976627.1662651004@sss.pgh.pa.us

Backpatch-through: 10
2022-09-28 13:14:38 -04:00
Bruce Momjian f77becbe5a revert "warn of SECURITY DEFINER schemas for non-sql_body funcs"
doc revert of commit 1703726488.  Change was applied to irrelevant
branches, and was not detailed enough to be helpful in relevant
branches.

Reported-by: Peter Eisentraut, Noah Misch

Discussion: https://postgr.es/m/a2dc9de4-24fc-3222-87d3-0def8057d7d8@enterprisedb.com

Backpatch-through: 10
2022-09-28 13:05:21 -04:00
Tom Lane 0937f6d172 Doc: document bpchar, clarify relationship of text and varchar.
For some reason the "bpchar" type name was defined nowhere in
our SGML docs, although several places refer to it in passing.
Give it a proper mention under Character Types.

While here, also provide an explanation of how the text and varchar
types relate.  The previous wording seemed to be doing its best
to sweep text under the rug, which doesn't seem very appropriate
given its prominence in other parts of the docs.

Minor rearrangements and word-smithing for clarity, too.

Laurenz Albe and Tom Lane, per gripe from Yanliang Lei

Discussion: https://postgr.es/m/120b3084.56b6.1833b5ffe4b.Coremail.msdnchina@163.com
2022-09-28 12:31:36 -04:00
Robert Haas a448e49bcb Revert 56-bit relfilenode change and follow-up commits.
There are still some alignment-related failures in the buildfarm,
which might or might not be able to be fixed quickly, but I've also
just realized that it increased the size of many WAL records by 4 bytes
because a block reference contains a RelFileLocator. The effect of that
hasn't been studied or discussed, so revert for now.
2022-09-28 09:55:28 -04:00
Peter Eisentraut c8b2ef05f4 Convert *GetDatum() and DatumGet*() macros to inline functions
The previous macro implementations just cast the argument to a target
type but did not check whether the input type was appropriate.  The
function implementation can do better type checking of the input type.

For the *GetDatumFast() macros, converting to an inline function
doesn't work in the !USE_FLOAT8_BYVAL case, but we can use
AssertVariableIsOfTypeMacro() to get a similar level of type checking.

Reviewed-by: Aleksander Alekseev <aleksander@timescale.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/8528fb7e-0aa2-6b54-85fb-0c0886dbd6ed%40enterprisedb.com
2022-09-27 20:50:21 +02:00
Robert Haas 05d4cbf9b6 Increase width of RelFileNumbers from 32 bits to 56 bits.
RelFileNumbers are now assigned using a separate counter, instead of
being assigned from the OID counter. This counter never wraps around:
if all 2^56 possible RelFileNumbers are used, an internal error
occurs. As the cluster is limited to 2^64 total bytes of WAL, this
limitation should not cause a problem in practice.

If the counter were 64 bits wide rather than 56 bits wide, we would
need to increase the width of the BufferTag, which might adversely
impact buffer lookup performance. Also, this lets us use bigint for
pg_class.relfilenode and other places where these values are exposed
at the SQL level without worrying about overflow.

This should remove the need to keep "tombstone" files around until
the next checkpoint when relations are removed. We do that to keep
RelFileNumbers from being recycled, but now that won't happen
anyway. However, this patch doesn't actually change anything in
this area; it just makes it possible for a future patch to do so.

Dilip Kumar, based on an idea from Andres Freund, who also reviewed
some earlier versions of the patch. Further review and some
wordsmithing by me. Also reviewed at various points by Ashutosh
Sharma, Vignesh C, Amul Sul, Álvaro Herrera, and Tom Lane.

Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
2022-09-27 13:25:21 -04:00
Tom Lane 3853664265 Introduce GUC_NO_RESET flag.
Previously, the transaction-property GUCs such as transaction_isolation
could be reset after starting a transaction, because we marked them
as GUC_NO_RESET_ALL but still allowed a targeted RESET.  That leads to
assertion failures or worse, because those properties aren't supposed
to change after we've acquired a transaction snapshot.

There are some NO_RESET_ALL variables for which RESET is okay, so
we can't just redefine the semantics of that flag.  Instead introduce
a separate GUC_NO_RESET flag.  Mark "seed", as well as the transaction
property GUCs, as GUC_NO_RESET.

We have to disallow GUC_ACTION_SAVE as well as straight RESET, because
otherwise a function having a "SET transaction_isolation" clause can
still break things: the end-of-function restore action is equivalent
to a RESET.

No back-patch, as it's conceivable that someone is doing something
this patch will forbid (like resetting one of these GUCs at transaction
start, or "CREATE FUNCTION ... SET transaction_read_only = 1") and not
running into problems with it today.  Given how long we've had this
issue and not noticed, the side effects in non-assert builds can't be
too serious.

Per bug #17385 from Andrew Bille.

Masahiko Sawada

Discussion: https://postgr.es/m/17385-9ee529fb091f0ce5@postgresql.org
2022-09-27 11:47:12 -04:00
Alvaro Herrera 249b0409b1
Fix pg_stat_statements for MERGE
We weren't jumbling the merge action list, so wildly different commands
would be considered to use the same query ID.  Add that, mention it in
the docs, and some test lines.

Backpatch to 15.

Author: Tatsu <bt22nakamorit@oss.nttdata.com>
Reviewed-by: Julien Rouhaud <rjuju123@gmail.com>
Discussion: https://postgr.es/m/d87e391694db75a038abc3b2597828e8@oss.nttdata.com
2022-09-27 10:44:42 +02:00
Tom Lane fbd2bd15b0 Doc: further adjust notes about pg_upgrade_output.d.
I'd misunderstood how it worked in 5f1048881.

Discussion: https://postgr.es/m/20220925215009.GC21938@telsasoft.com
2022-09-26 14:19:21 -04:00
Peter Eisentraut 6d6e136872 Improve terminology
Use "prepared transaction" instead of "two-phrase transaction".  This
is in line with 0e60a50e0b.
2022-09-23 21:18:17 -04:00
Tom Lane 5f1048881d Doc: minor cleanups.
Improve a couple of things I noticed while working on v15
release notes.
2022-09-23 18:20:14 -04:00
Alvaro Herrera 0032a54567
Remove PQsendQuery support in pipeline mode
The extended query protocol implementation I added in commit
acb7e4eb6b has bugs when used in pipeline mode.  Rather than spend
more time trying to fix it, remove that code and make the function rely
on simple query protocol only, meaning it can no longer be used in
pipeline mode.

Users can easily change their applications to use PQsendQueryParams
instead.  We leave PQsendQuery in place for Postgres 14, just in case
somebody is using it and has not hit the mentioned bugs; but we should
recommend that it not be used.

Backpatch to 15.

Per bug report from Gabriele Varrazzo.
Discussion: https://postgr.es/m/CA+mi_8ZGSQNmW6-mk_iSR4JZB_LJ4ww3suOF+1vGNs3MrLsv4g@mail.gmail.com
2022-09-23 18:21:22 +02:00
Amit Kapila 13a185f54b Allow publications with schema and table of the same schema.
We previously thought that allowing such cases can confuse users when they
specify DROP TABLES IN SCHEMA but that doesn't seem to be the case based
on discussion. This helps to uplift the restriction during
ALTER TABLE ... SET SCHEMA which used to ensure that we couldn't end up
with a publication having both a schema and the same schema's table.

To allow this, we need to forbid having any schema on a publication if
column lists on a table are specified (and vice versa). This is because
otherwise we still need a restriction during ALTER TABLE ... SET SCHEMA to
forbid cases where it could lead to a publication having both a schema and
the same schema's table with column list.

Based on suggestions by Peter Eisentraut.

Author: Hou Zhijie and Vignesh C
Reviewed-By: Peter Smith, Amit Kapila
Backpatch-through: 15, where it was introduced
Discussion: https://postgr.es/m/2729c9e2-9aac-8cda-f2f4-34f2bcc18f4e@enterprisedb.com
2022-09-23 08:21:26 +05:30
Tom Lane d89755dac6 Doc: adjust misleading phrasing of a few cross-references.
The pg_dump and pg_dumpall man pages referred to app-psql-patterns
as appearing "below", which I suspect was copied-and-pasted from
equivalent text in psql-ref.sgml rather than being actually thought
through.  At least to me, that phrasing means "later in this same
web page/section", which this link target is not.  Drop the
misleading and unnecessary-in-any-case adjective.
2022-09-22 22:05:09 -04:00
Alvaro Herrera 790bf615dd
Remove ALL keyword from TABLES IN SCHEMA for publication
This may be a bit too subtle, but removing that word from there makes
this clause no longer a perfect parallel of the GRANT variant "ALL
TABLES IN SCHEMA": indeed, for publications what we record is the schema
itself, not the tables therein, which means that any tables added to the
schema in the future are also published.  This is completely different
to what GRANT does, which is affect only the tables that exist when the
command is executed.

There isn't resounding support for this change, but there are a few
positive votes and no opposition.  Because the time to 15 RC1 is very
short, let's get this out now.

Backpatch to 15.

Discussion: https://postgr.es/m/2729c9e2-9aac-8cda-f2f4-34f2bcc18f4e
2022-09-22 19:02:25 +02:00
Peter Eisentraut ba50834551 Restore archive_command documentation
Commit 5ef1eefd76, which added
archive_library, purged most mentions of archive_command from the
documentation.  This is inappropriate, since archive_command is still
a feature in use and users will want to see information about it.

This restores all the removed mentions and rephrases things so that
archive_command and archive_library are presented as alternatives of
each other.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Discussion: https://www.postgresql.org/message-id/9366d634-a917-85a9-4991-b2a4859edaf9@enterprisedb.com
2022-09-22 07:35:00 -04:00
Michael Paquier 18ac08f0b4 Use min/max bounds defined by Zstd for compression level
The bounds hardcoded in compression.c since ffd5365 (minimum at 1 and
maximum at 22) do not match the reality of what zstd is able to
handle, these values being available via ZSTD_maxCLevel() and
ZSTD_minCLevel() at run-time.  The maximum of 22 is actually correct
in recent versions, but the minimum was not as the library can go down
to -131720 by design.  This commit changes the code to use the run-time
values in the code instead of some hardcoded ones.

Zstd seems to assume that these bounds could change in the future, and
Postgres will be able to adapt automatically to such changes thanks to
what's being done in this commit.

Reported-by: Justin Prysby
Discussion: https://postgr.es/m/20220922033716.GL31833@telsasoft.com
Backpatch-through: 15
2022-09-22 20:02:40 +09:00
Andres Freund e6927270cd meson: Add initial version of meson based build system
Autoconf is showing its age, fewer and fewer contributors know how to wrangle
it. Recursive make has a lot of hard to resolve dependency issues and slow
incremental rebuilds. Our home-grown MSVC build system is hard to maintain for
developers not using Windows and runs tests serially. While these and other
issues could individually be addressed with incremental improvements, together
they seem best addressed by moving to a more modern build system.

After evaluating different build system choices, we chose to use meson, to a
good degree based on the adoption by other open source projects.

We decided that it's more realistic to commit a relatively early version of
the new build system and mature it in tree.

This commit adds an initial version of a meson based build system. It supports
building postgres on at least AIX, FreeBSD, Linux, macOS, NetBSD, OpenBSD,
Solaris and Windows (however only gcc is supported on aix, solaris). For
Windows/MSVC postgres can now be built with ninja (faster, particularly for
incremental builds) and msbuild (supporting the visual studio GUI, but
building slower).

Several aspects (e.g. Windows rc file generation, PGXS compatibility, LLVM
bitcode generation, documentation adjustments) are done in subsequent commits
requiring further review. Other aspects (e.g. not installing test-only
extensions) are not yet addressed.

When building on Windows with msbuild, builds are slower when using a visual
studio version older than 2019, because those versions do not support
MultiToolTask, required by meson for intra-target parallelism.

The plan is to remove the MSVC specific build system in src/tools/msvc soon
after reaching feature parity. However, we're not planning to remove the
autoconf/make build system in the near future. Likely we're going to keep at
least the parts required for PGXS to keep working around until all supported
versions build with meson.

Some initial help for postgres developers is at
https://wiki.postgresql.org/wiki/Meson

With contributions from Thomas Munro, John Naylor, Stone Tickle and others.

Author: Andres Freund <andres@anarazel.de>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Author: Peter Eisentraut <peter@eisentraut.org>
Reviewed-By: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Discussion: https://postgr.es/m/20211012083721.hvixq4pnh2pixr3j@alap3.anarazel.de
2022-09-21 22:37:17 -07:00
Fujii Masao 2f636daac8 docs: Fix snapshot name in SET TRANSACTION docs.
Commit 6c2003f8a1 changed the snapshot names mentioned in
SET TRANSACTION docs, however, there was one place that
the commit missed updating the name.

Back-patch to all supported versions.

Author: Japin Li
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/MEYP282MB1669BD4280044501165F8B07B64F9@MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
2022-09-22 12:54:26 +09:00