Commit Graph

57590 Commits

Author SHA1 Message Date
Michael Paquier d86d20f0ba Add backend support for injection points
Injection points are a new facility that makes possible for developers
to run custom code in pre-defined code paths.  Its goal is to provide
ways to design and run advanced tests, for cases like:
- Race conditions, where processes need to do actions in a controlled
ordered manner.
- Forcing a state, like an ERROR, FATAL or even PANIC for OOM, to force
recovery, etc.
- Arbitrary sleeps.

This implements some basics, and there are plans to extend it more in
the future depending on what's required.  Hence, this commit adds a set
of routines in the backend that allows developers to attach, detach and
run injection points:
- A code path calling an injection point can be declared with the macro
INJECTION_POINT(name).
- InjectionPointAttach() and InjectionPointDetach() to respectively
attach and detach a callback to/from an injection point.  An injection
point name is registered in a shmem hash table with a library name and a
function name, which will be used to load the callback attached to an
injection point when its code path is run.

Injection point names are just strings, so as an injection point can be
declared and run by out-of-core extensions and modules, with callbacks
defined in external libraries.

This facility is hidden behind a dedicated switch for ./configure and
meson, disabled by default.

Note that backends use a local cache to store callbacks already loaded,
cleaning up their cache if a callback has found to be removed on a
best-effort basis.  This could be refined further but any tests but what
we have here was fine with the tests I've written while implementing
these backend APIs.

Author: Michael Paquier, with doc suggestions from Ashutosh Bapat.
Reviewed-by: Ashutosh Bapat, Nathan Bossart, Álvaro Herrera, Dilip
Kumar, Amul Sul, Nazir Bilal Yavuz
Discussion: https://postgr.es/m/ZTiV8tn_MIb_H2rE@paquier.xyz
2024-01-22 10:15:50 +09:00
Alexander Korotkov c03d91d9be Fix table name collision in tests in 0452b461bc 2024-01-21 23:27:29 +02:00
Alexander Korotkov 0452b461bc Explore alternative orderings of group-by pathkeys during optimization.
When evaluating a query with a multi-column GROUP BY clause, we can minimize
sort operations or avoid them if we synchronize the order of GROUP BY clauses
with the ORDER BY sort clause or sort order, which comes from the underlying
query tree. Grouping does not imply any ordering, so we can compare
the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg,
we simply compared keys in the order specified in the query. This commit
explores alternative ordering of the keys, trying to find a cheaper one.

The ordering of group keys may interact with other parts of the query, some of
which may not be known while planning the grouping. For example, there may be
an explicit ORDER BY clause or some other ordering-dependent operation higher up
in the query, and using the same ordering may allow using either incremental
sort or even eliminating the sort entirely.

The patch always keeps the ordering specified in the query, assuming the user
might have additional insights.

This introduces a new GUC enable_group_by_reordering so that the optimization
may be disabled if needed.

Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru
Author: Andrei Lepikhov, Teodor Sigaev
Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov
Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu
Reviewed-by: Tom Lane, Alexander Korotkov, Richard Guo, Alena Rybakina
2024-01-21 22:21:36 +02:00
Alexander Korotkov 7ab80ac1ca Generalize the common code of adding sort before processing of grouping
Extract the repetitive code pattern into a new function make_ordered_path().

Discussion: https://postgr.es/m/CAPpHfdtzaVa7S4onKy3YvttF2rrH5hQNHx9HtcSTLbpjx%2BMJ%2Bw%40mail.gmail.com
Author: Andrei Lepikhov
2024-01-21 22:19:47 +02:00
Tom Lane 58447e3189 Add hint about not qualifying UPDATE...SET target with relation name.
Target columns in UPDATE ... SET must not be qualified with the target
table; we disallow this because it'd create ambiguity about which name
is the column name in case of field-qualified names.  However, newbies
have been seen to expect that they could qualify a target name just
like other names.  The error message when they do is confusing:
"column "foo" of relation "foo" does not exist".  To improve matters,
issue a HINT if the invalid name is qualified and matches the
relation's alias.

James Coleman (editorialized a bit by me)

Discussion: https://postgr.es/m/CAAaqYe8S2Qa060UV-YF5GoSd5PkEhLV94x-fEi3=TOtpaXCV+w@mail.gmail.com
2024-01-20 17:54:14 -05:00
Tom Lane 075df6b208 Add planner support functions for range operators <@ and @>.
These support functions will transform expressions with constant
range values into direct comparisons on the range bound values,
which are frequently better-optimizable.  The transformation is
skipped however if it would require double evaluation of a
volatile or expensive element expression.

Along the way, add the range opfamily OID to range typcache entries,
since load_rangetype_info has to compute that anyway and it seems
silly to duplicate the work later.

Kim Johan Andersson and Jian He, reviewed by Laurenz Albe

Discussion: https://postgr.es/m/94f64d1f-b8c0-b0c5-98bc-0793a34e0851@kimmet.dk
2024-01-20 13:57:54 -05:00
Nathan Bossart abb0b4fc03 Teach autoprewarm to use the dynamic shared memory registry.
Besides showcasing the DSM registry, this prevents pg_prewarm from
stealing from the main shared memory segment's extra buffer space
when autoprewarm_start_worker() and autoprewarm_dump_now() are used
without loading the module via shared_preload_libraries.

Suggested-by: Michael Paquier
Reviewed-by: Bharath Rupireddy
Discussion: https://postgr.es/m/20231205034647.GA2705267%40nathanxps13
2024-01-19 14:43:59 -06:00
Nathan Bossart 8b2bcf3f28 Introduce the dynamic shared memory registry.
Presently, the most straightforward way for a shared library to use
shared memory is to request it at server startup via a
shmem_request_hook, which requires specifying the library in
shared_preload_libraries.  Alternatively, the library can create a
dynamic shared memory (DSM) segment, but absent a shared location
to store the segment's handle, other backends cannot use it.  This
commit introduces a registry for DSM segments so that these other
backends can look up existing segments with a library-specified
string.  This allows libraries to easily use shared memory without
needing to request it at server startup.

The registry is accessed via the new GetNamedDSMSegment() function.
This function handles allocating the segment and initializing it
via a provided callback.  If another backend already created and
initialized the segment, it simply attaches the segment.
GetNamedDSMSegment() locks the registry appropriately to ensure
that only one backend initializes the segment and that all other
backends just attach it.

The registry itself is comprised of a dshash table that stores the
DSM segment handles keyed by a library-specified string.

Reviewed-by: Michael Paquier, Andrei Lepikhov, Nikita Malakhov, Robert Haas, Bharath Rupireddy, Zhang Mingli, Amul Sul
Discussion: https://postgr.es/m/20231205034647.GA2705267%40nathanxps13
2024-01-19 14:24:36 -06:00
Nathan Bossart 964152c476 doc: Reorganize section for shared memory and LWLocks.
Presently, this section meanders through a few different features,
and the text itself is terse.  This commit attempts to improve
matters by splitting the section into smaller sections and by
expanding the text for clarity.  This is preparatory work for a
follow-up commit that will introduce a way for libraries to use
shared memory without needing to request it at startup time.

Reviewed-by: Aleksander Alekseev, Bharath Rupireddy, Abhijit Menon-Sen
Discussion: https://postgr.es/m/20240112041430.GA3557928%40nathanxps13
Discussion: https://postgr.es/m/20231205034647.GA2705267%40nathanxps13
2024-01-19 11:18:32 -06:00
Alexander Korotkov 448a3331d9 Fix name collision in c64086b79d
Reported-by: Erik Rijkers, Tom Lane
Discussion: https://postgr.es/m/E1rQqeS-002A0s-Qm%40gemulon.postgresql.org
2024-01-19 18:17:13 +02:00
Daniel Gustafsson b2a296b0cf doc: Update compiler for FreeBSD specific instructions
FreeBSD has since v13 switched to Clang from GCC, so update our
instructions to cc instead of gcc to reflect that.

Reported-by: digitaldog4@gmail.com
Discussion: https://postgr.es/m/CAJR80-xzR49jEkAC98Yo-wuSdFDV2=2H4j9Z2AukSVU=-UpynA@mail.gmail.com
2024-01-19 16:27:27 +01:00
Alexander Korotkov c64086b79d Reorder actions in ProcArrayApplyRecoveryInfo()
Since 5a1dfde833, 2PC filenames use FullTransactionId.  Thus, it needs to
convert TransactionId to FullTransactionId in StandbyTransactionIdIsPrepared()
using TransamVariables->nextXid.  However, ProcArrayApplyRecoveryInfo()
first releases locks with usage StandbyTransactionIdIsPrepared(), then advances
TransamVariables->nextXid.  This sequence of actions could cause errors.
This commit makes ProcArrayApplyRecoveryInfo() advance
TransamVariables->nextXid before releasing locks.

Reported-by: Thomas Munro, Michael Paquier
Discussion: https://postgr.es/m/CA%2BhUKGLj_ve1_pNAnxwYU9rDcv7GOhsYXJt7jMKSA%3D5-6ss-Cw%40mail.gmail.com
Discussion: https://postgr.es/m/Zadp9f4E1MYvMJqe%40paquier.xyz
2024-01-19 17:19:17 +02:00
Peter Eisentraut 6db4598fcb Add stratnum GiST support function
This is support function 12 for the GiST AM and translates
"well-known" RT*StrategyNumber values into whatever strategy number is
used by the opclass (since no particular numbers are actually
required).  We will use this to support temporal PRIMARY
KEY/UNIQUE/FOREIGN KEY/FOR PORTION OF functionality.

This commit adds two implementations, one for internal GiST opclasses
(just an identity function) and another for btree_gist opclasses.  It
updates btree_gist from 1.7 to 1.8, adding the support function for
all its opclasses.

Author: Paul A. Jungwirth <pj@illuminatedcomputing.com>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: jian he <jian.universality@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/CA+renyUApHgSZF9-nd-a0+OPGharLQLO=mDHcY4_qQ0+noCUVg@mail.gmail.com
2024-01-19 15:42:13 +01:00
Alexander Korotkov b725b7eec4 Rename COPY option from SAVE_ERROR_TO to ON_ERROR
The option names now are "stop" (default) and "ignore".  The future options
could be "file 'filename.log'" and "table 'tablename'".

Discussion: https://postgr.es/m/20240117.164859.2242646601795501168.horikyota.ntt%40gmail.com
Author: Jian He
Reviewed-by: Atsushi Torikoshi
2024-01-19 15:15:51 +02:00
John Naylor dd0a0cfc81 Fixed misspelled byteswap function for big endian machines
Per members lora and mamba
2024-01-19 13:26:18 +07:00
John Naylor 0aba255440 Add optimized C string hashing
Given an already-initialized hash state and a NUL-terminated string,
accumulate the hash of the string into the hash state and return the
length for the caller to (optionally) save for the finalizer. This
avoids a strlen call.

If the string pointer is aligned, we can use a word-at-a-time
algorithm for NUL lookahead. The aligned case is only used on 64-bit
platforms, since it's not worth the extra complexity for 32-bit.

Handling the tail of the string after finishing the word-wise loop
was inspired by NetBSD's strlen(), but no code was taken since that
is written in assembly language.

As demonstration, use this in the search path cache. This brings the
general case performance closer to the special case optimization done
in commit a86c61c9ee. There are other places that could benefit, but
that is left for future work.

Jeff Davis and John Naylor
Reviewed by Heikki Linnakangas, Jian He, Junwang Zhao

Discussion: https://postgr.es/m/3820f030fd008ff14134b3e9ce5cc6dd623ed479.camel%40j-davis.com
Discussion: https://postgr.es/m/b40292c99e623defe5eadedab1d438cf51a4107c.camel%40j-davis.com
2024-01-19 12:56:15 +07:00
John Naylor e97b672c88 Add inline incremental hash functions for in-memory use
It can be useful for a hash function to expose separate initialization,
accumulation, and finalization steps.  In particular, this is useful
for building inline hash functions for simplehash.  Instead of trying
to whack around hash_bytes while maintaining its current behavior on
all platforms, we base this work on fasthash (MIT licensed) which
is simple, faster than hash_bytes for inputs over 12 bytes long,
and also passes the hash function testing suite SMHasher.

The fasthash functions have been reimplemented using our added-on
incremental interface to validate that this method will still give
the same answer, provided we have the input length ahead of time.

This functionality lives in a new header hashfn_unstable.h. The name
implies we have the freedom to change things across versions that
would be unacceptable for our other hash functions that are used for
e.g. hash indexes and hash partitioning. As such, these should only
be used for in-memory data structures like hash tables. There is also
no guarantee of being independent of endianness or pointer size.

As demonstration, use fasthash for pgstat_hash_hash_key.  Previously
this called the 32-bit murmur finalizer on the three elements,
then joined them with hash_combine(). The new function is simpler,
faster and takes up less binary space. While the collision and bias
behavior were almost certainly fine with the previous coding, now we
have objective confidence of that.

There are other places that could benefit from this, but that is left
for future work.

Reviewed by Jeff Davis, Heikki Linnakangas, Jian He, Junwang Zhao
Credit to Andres Freund for the idea

Discussion: https://postgr.es/m/20231122223432.lywt4yz2bn7tlp27%40awork3.anarazel.de
2024-01-19 12:44:09 +07:00
Michael Paquier 04c0897d3b psql: Add ignore_slash_options in bind's inactive branch
All commands accepting arguments, handling them with OT_NORMAL, OT_SQLID
or OT_SQLIDHACK, should call ignore_slash_options() in inactive branch
to scan and discard extra arguments.  All the backslash commands that
handle arguments do so, except \bind.

This commit adds the missing ignore_slash_options to \bind's inactive
branch.  This inconsistency is a logic bug, however the behavior happens
to be unchanged as any extra arguments are discarded later in
HandleSlashCmds(), so no backpatch is done.

While on it, this adds \bind to the list of backslash commands where
inactive \if branches are checked in the tests for psql.

Reported-by: Jelte Fennema-Nio
Author: Anthonin Bonnefoy
Discussion: https://postgr.es/m/CAGECzQR1+udGKz+FbHiCQ7CWDiF1fCGi2xYuvQUODdMAfJbaLA@mail.gmail.com
2024-01-19 14:18:20 +09:00
Michael Paquier 3ada0d2cae Fix incorrect placeholder in walreceiver.c
Author: Yongtao Huang
Discussion: https://postgr.es/m/CAOe1Go3H7CgrSceO+HBhnoptk-mJhii-YT8D19CikKintjwumQ@mail.gmail.com
2024-01-19 13:20:49 +09:00
Nathan Bossart d891dcc065 Improve some documentation about the bootstrap superuser.
This commit adds some notes about the inability to remove superuser
privileges from the bootstrap superuser.  This has been blocked
since commit e530be2c5c, but it wasn't intended be a supported
feature before that, either.

In passing, change "bootstrap user" to "bootstrap superuser" in a
couple places.

Author: Yurii Rashkovskii
Reviewed-by: Vignesh C, David G. Johnston
Discussion: https://postgr.es/m/CA%2BRLCQzSx_eTC2Fch0EzeNHD3zFUcPvBYOoB%2BpPScFLch1DEQw%40mail.gmail.com
2024-01-18 21:39:51 -06:00
Jeff Davis dd3ca8cbb0 Fix buildfarm error from commit 5c31669058.
Skip test when not using unix domain sockets.

Discussion: https://postgr.es/m/CALDaNm29-8OozsBWo9H6DN_Tb_3yA1QjRJput-KhaN8ncDJtJA@mail.gmail.com
Backpatch-through: 16
2024-01-18 14:59:59 -08:00
David Rowley 4b31063643 Fix broken Bitmapset optimization in DiscreteKnapsack()
Some code in DiscreteKnapsack() attempted to zero all words in a
Bitmapset by performing bms_del_members() to delete all the members from
itself before replacing those members with members from another set.
When that code was written, this was a valid way to manipulate the set
in such a way to save from palloc having to be called to allocate a new
Bitmapset.  However, 00b41463c modified Bitmapsets so that an empty set is
*always* represented as NULL and this breaks the optimization as the
Bitmapset code will always pfree the memory when the set becomes empty.

Since DiscreteKnapsack() has been coded to avoid as many unneeded
allocations as possible, it seems risky to not fix this.  Here we add
bms_replace_members() to effectively perform an in-place copy of another
set, reusing the memory of the existing set, when possible.

This got broken in v16, but no backpatch for now as there've been no
complaints.

Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/CAApHDvoTCBkBU2PJghNOFUiO0q=QP4WAWHi5sJP6_4=b2WodrA@mail.gmail.com
2024-01-19 10:44:36 +13:00
Tom Lane 57b440ec11 Fix plpgsql to allow new-style SQL CREATE FUNCTION as a SQL command.
plpgsql fails on new-style CREATE FUNCTION/PROCEDURE commands within
a routine or DO block, because make_execsql_stmt believes that a
semicolon token always terminates a SQL command.  Now, that's actually
been wrong since the day it was written, because CREATE RULE has long
allowed multiple rule actions separated by semicolons.  But there are
few enough people using multi-action rules that there was never an
attempt to fix it.  New-style SQL functions, though, are popular.

psql has this same problem of "does this semicolon really terminate
the command?".  It deals with CREATE RULE by counting parenthesis
nesting depth: a semicolon within parens doesn't end a command.
Commits e717a9a18 and 029c5ac03 created a similar heuristic to count
matching BEGIN/END pairs (but only within CREATEs, so as not to be
fooled by plain BEGIN).  That's survived several releases now without
trouble reports, so let's just absorb those heuristics into plpgsql.

Per report from Samuel Dussault.  Back-patch to v14 where new-style
SQL function syntax came in.

Discussion: https://postgr.es/m/YT2PR01MB88552C3E9AD40A6C038774A781722@YT2PR01MB8855.CANPRD01.PROD.OUTLOOK.COM
2024-01-18 16:11:16 -05:00
Robert Haas e313a61137 Remove LVPagePruneState.
Commit cb970240f1 moved some code from
lazy_scan_heap() to lazy_scan_prune(), and now some things that used to
need to be passed back and forth are completely local to lazy_scan_prune().
Hence, this struct is mostly obsolete.  The only thing that still
needs to be passed back to the caller is has_lpdead_items, and that's
also passed back by lazy_scan_noprune(), so do it the same way in both
cases.

Melanie Plageman, reviewed and slightly revised by me.

Discussion: http://postgr.es/m/CAAKRu_aM=OL85AOr-80wBsCr=vLVzhnaavqkVPRkFBtD0zsuLQ@mail.gmail.com
2024-01-18 15:17:09 -05:00
Robert Haas cb970240f1 Move VM update code from lazy_scan_heap() to lazy_scan_prune().
Most of the output parameters of lazy_scan_prune() were being
used to update the VM in lazy_scan_heap(). Moving that code into
lazy_scan_prune() simplifies lazy_scan_heap() and requires less
communication between the two.

This change permits some further code simplification, but that
is left for a separate commit.

Melanie Plageman, reviewed by me.

Discussion: http://postgr.es/m/CAAKRu_aM=OL85AOr-80wBsCr=vLVzhnaavqkVPRkFBtD0zsuLQ@mail.gmail.com
2024-01-18 14:44:57 -05:00
Robert Haas 0a157a4d44 Clarify language in incremental backup glossary entry.
These changes were intended to be included in commit
7b1dbf0a8d, but I goofed.
2024-01-18 10:09:38 -05:00
Robert Haas c120550edb Optimize vacuuming of relations with no indexes.
If there are no indexes on a relation, items can be marked LP_UNUSED
instead of LP_DEAD when pruning. This significantly reduces WAL
volume, since we no longer need to emit one WAL record for pruning
and a second to change the LP_DEAD line pointers thus created to
LP_UNUSED.

Melanie Plageman, reviewed by Andres Freund, Peter Geoghegan, and me

Discussion: https://postgr.es/m/CAAKRu_bgvb_k0gKOXWzNKWHt560R0smrGe3E8zewKPs8fiMKkw%40mail.gmail.com
2024-01-18 10:03:42 -05:00
Robert Haas 7b1dbf0a8d More documentation updates for incremental backup.
Add new terms to glossary. Add a reference to walsummarizer
to monitoring.sgml.

Matthias van de Meent and Robert Haas

Discussion: http://postgr.es/m/CAEze2WjhdVCqEe_qqEok3NA6DwUdOGSBjAxzmYdAqiaaH1uRcg@mail.gmail.com
2024-01-18 09:25:28 -05:00
Alvaro Herrera 27d04ed531
Rework text in replication slots documentation
This is cosmetic, so no backpatch.

Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/20230413111838.e7yxke2dtwrxw3qy@alvherre.pgsql
Discussion: https://postgr.es/m/202401151537.ryhysqfncs6d@alvherre.pgsql
2024-01-18 11:30:53 +01:00
Peter Eisentraut ed1e0a6512 Error message capitalisation
per style guidelines

Author: Peter Smith <peter.b.smith@fujitsu.com>
Discussion: https://www.postgresql.org/message-id/flat/CAHut%2BPtzstExQ4%3DvFH%2BWzZ4g4xEx2JA%3DqxussxOdxVEwJce6bw%40mail.gmail.com
2024-01-18 09:35:12 +01:00
Peter Eisentraut 686db12de3 Fix an issue in PostgreSQL::Test::Cluster:psql()
Due to the commit c5385929 which made all Perl warnings to fatal, use
of PostgreSQL::Test::Cluster:psql() and safe_psql() with timeout
started to fail with the following error:

Use of uninitialized value $ret in bitwise and (&) at
..src/test/perl/PostgreSQL/Test/Cluster.pm line 2015.

Fix that by placing $ret conversion code in psql() in an if (defined
$ret) block.

With this change, the behavior of psql() becomes same as before, that
is, the whole function returns undef on timeout, which is usefully
different from returning 0.

Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/06f899fd-1826-05ab-42d6-adeb1fd5e200%40eisentraut.org
2024-01-18 08:46:42 +01:00
Michael Paquier 0ae3b46621 Improve handling of dropped partitioned indexes for REINDEX INDEX
A REINDEX INDEX done on a partitioned index builds a list of the indexes
to work on before processing its partitions in individual transactions.
When combined with a DROP of the partitioned index, there was a window
where it was possible to see some unexpected "could not open relation
with OID", synonym of relation lookup error.  The code was robust enough
to handle the case where the parent relation is missing, but not the
case where an index would be gone missing.

This is similar to 1d65416661.

Support for REINDEX on partitioned relations has been introduced in
a6642b3ae0, so backpatch down to 14.

Author: Fei Changhong
Discussion: https://postgr.es/m/tencent_6A52106095ACDE55333E3AD33F304C0C3909@qq.com
Backpatch-through: 14
2024-01-18 16:30:11 +09:00
Michael Paquier 8013850c85 Add try_index_open(), conditional variant of index_open()
try_index_open() is able to open an index if its relkind fits, except
that it would return NULL instead of generated an error if the relation
does not exist.  This new routine will be used by an upcoming patch to
make REINDEX on partitioned relations more robust when an index in a
partition tree is dropped.

Extracted from a larger patch by the same author.

Author: Fei Changhong
Discussion: https://postgr.es/m/tencent_6A52106095ACDE55333E3AD33F304C0C3909@qq.com
Backpatch-through: 14
2024-01-18 15:04:24 +09:00
Michael Paquier 2f35c14cfb seg: Add test "security" in meson.build
Oversight in 681d9e4621 where the test has been added.

Reported-by: Justin Pryzby
Discussion: https://postgr.es/m/ZK5AgYxG4zLErD5O@telsasoft.com
Backpatch-through: 16
2024-01-18 10:12:44 +09:00
Alexander Korotkov 4b885d01f9 Remove the flaky check in event_trigger_login regression test
The query checks that pg_database.dathasloginevt is unset on connect when
there are no event triggers.  However, unsetting this flag is implemented in
a non-blocking way, so a concurrent autovacuum connection breaks this check.
It doesn't seem we can do much with this, at least within a regression test.
So, remove it.

Reported-by: Alexander Lakhin
Discussion: https://postgr.es/m/44807d19-81a6-3884-3e0f-22dd99aac3ed%40gmail.com
2024-01-17 23:16:53 +02:00
Alexander Korotkov 58fbbc9d68 Fix spelling in notice
Reported-by: Atsushi Torikoshi
Discussion: https://postgr.es/m/762d7dd4d5aa9e5ecffec2ae6a255a28%40oss.nttdata.com
2024-01-17 22:59:09 +02:00
Heikki Linnakangas 2b53a462cf Fix incorrect comment on how BackendStatusArray is indexed
The comment was copy-pasted from the call to ProcSignalInit() in
AuxiliaryProcessMain(), which uses a similar scheme of having reserved
slots for aux processes after MaxBackends slots for backends. However,
ProcSignalInit() indexing starts from 1, whereas BackendStatusArray
starts from 0. The code is correct, but the comment was wrong.

Discussion: https://www.postgresql.org/message-id/f3ecd4cb-85ee-4e54-8278-5fabfb3a4ed0@iki.fi
Backpatch-through: v14
2024-01-17 15:44:10 +02:00
Daniel Gustafsson 7cfa154d15 Close socket in case of errors in setting non-blocking
If configuring the newly created socket non-blocking fails we
error out and return INVALID_SOCKET, but the socket that had
been created wasn't closed. Fix by issuing closesocket in the
errorpath.

Backpatch to all supported branches.

Author: Ranier Vilela <ranier.vf@gmail.com>
Discussion: https://postgr.es/m/CAEudQApmU5CrKefH85VbNYE2y8H=-qqEJbg6RAPU65+vCe+89A@mail.gmail.com
Backpatch-through: v12
2024-01-17 11:24:11 +01:00
Michael Paquier 44ad5129ce Fix description of DecodeInsert() in decode.c
This incorrectly referred to deletes.

Author: Yongtao Huang
Reviewed-by: Richard Guo
Description: https://postgr.es/m/CAOe1Go0Czgvo9eiDqeFpaABwJu=gBK6qjrYzZGZLn=tKDX8AUw@mail.gmail.com
2024-01-17 17:03:02 +09:00
Michael Paquier 061cc7eaca Remove some comments related to pqPipelineSync() and PQsendPipelineSync()
These comments explained how these functions behave internally, and the
equivalent is described in the documentation section dedicated to the
pipeline mode of libpq.  Let's remove these comments, getting rid of the
duplication with the docs.

Reported-by: Álvaro Herrera
Reviewed-by: Álvaro Herrera
Discussion: https://postgr.es/m/202401150949.wq7ynlmqxphy@alvherre.pgsql
2024-01-17 15:53:59 +09:00
Michael Paquier 2197d06224 Add support for parsing of large XML data (>= 10MB)
This commit adds XML_PARSE_HUGE to the libxml2 functions used in core
for the parsing of XML objects, raising up the original limit of 10MB
supported by libxml2.

In most code paths of upstream, XML_MAX_TEXT_LENGTH (10^7) is the
historical limit that gets upgraded to XML_MAX_HUGE_LENGTH (10^9) once
XML_PARSE_HUGE is given to the parser calls.  These are still limited by
any palloc() calls for text, up to 1GB.

This offers the possibility to handle within the backend XML objects
larger than 10MB in general, with also a higher depth limit.  This
change affects the contrib module xml2, the xml data type and SQL/XML.

Author: Dmitry Koval
Reviewed-by: Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/18274-98d16bc03520665f@postgresql.org
2024-01-17 14:03:55 +09:00
Michael Paquier 65c5864d7f xml2: Replace deprecated routines with recommended ones
Some functions are used in the tree and are currently marked as
deprecated by upstream.  This commit refreshes the code to use the
recommended functions, leading to the following changes:
- xmlSubstituteEntitiesDefault() is gone, and needs to be replaced with
XML_PARSE_NOENT for the paths doing the parsing.
- xmlParseMemory() -> xmlReadMemory().

These functions, as well as more functions setting global states, have
been officially marked as deprecated by upstream in August 2022.  Their
replacements exist since the 2001-ish area, as far as I have checked,
so that should be safe.

Author: Dmitry Koval
Discussion: https://postgr.es/m/18274-98d16bc03520665f@postgresql.org
2024-01-17 08:53:16 +09:00
Alexander Korotkov 8ad1f7db87 Fix format specifier for NOTICE in copyfrom.c
It's incorrect to use %lz for 64-bit numbers on 32-bit machine.
2024-01-17 00:59:44 +02:00
Alexander Korotkov 9e2d870119 Add new COPY option SAVE_ERROR_TO
Currently, when source data contains unexpected data regarding data type or
range, the entire COPY fails. However, in some cases, such data can be ignored
and just copying normal data is preferable.

This commit adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.

Currently, SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.

Later works are expected to add more choices, such as 'log' and 'table'.

Author: Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He
Discussion: https://postgr.es/m/87k31ftoe0.fsf_-_%40commandprompt.com
Reviewed-by: Pavel Stehule, Andres Freund, Tom Lane, Daniel Gustafsson,
Reviewed-by: Alena Rybakina, Andy Fan, Andrei Lepikhov, Masahiko Sawada
Reviewed-by: Vignesh C, Atsushi Torikoshi
2024-01-16 23:08:53 +02:00
David Rowley c7e5e994b2 Fix REALLOCATE_BITMAPSETS code
7d58f2342 added a compile-time option to have bitmapset.c reallocate the
set before returning when a set is modified.  That commit failed to do
its job in various cases and returned the input set when it shouldn't
have in these cases.  Here we fix those missing cases.

This commit also adds some documentation about what
REALLOCATE_BITMAPSETS is for.  This is important as future functions
that go inside bitmapset.c need to know if they need to do anything
special when this compile-time option is defined.

Also, between 71a3e8c43 and 7d58f2342 some Asserts seem to have become
duplicated.  Tidy these up.  Rather than having the Assert check each
aspect of what makes a set invalid, here we introduce a helper function
which returns false when a set is invalid and have the Asserts use this
instead.

Also, make a pass on improving the comments in bitmapset.c.  Various
comments mentioned the input sets being "recycled".  This could be
interpreted to mean that the output set will always point to the same
memory as the given input parameter.  Here we try to make it clear that
this must not be relied upon and that callers must ensure that all
references to a given set are updated on each modification.

In passing, improve comments for bms_union(), bms_intersect() and
bms_difference() to detail what they do.  I (David) have too often had to
remind myself by reading the code each time to find out if I need, for
example, to use bms_union() or bms_join().  I also removed some
low-value comments that were trying to convey information about "these
operations" without mentioning which operations it was talking about.
It seems better to document these things in the function header comment
instead.

Author: Richard Guo, David Rowley
Discussion: https://postgr.es/m/CAMbWs4-djy9qYux2gZrtmxA0StrYXJjvB-oqLxn-d7J88t=PQQ@mail.gmail.com
2024-01-17 09:30:21 +13:00
Robert Haas 45d395cd75 Be more consistent about whether to update the FSM while vacuuming.
Previously, when lazy_scan_noprune() was called and returned true, we would
update the FSM immediately if the relation had no indexes or if the page
contained no dead items. On the other hand, when lazy_scan_prune() was
called, we would update the FSM if either of those things was true or
if index vacuuming was disabled. Eliminate that behavioral difference by
considering vacrel->do_index_vacuuming in both cases.

Also, make lazy_scan_heap() responsible for deciding whether to update
the FSM, instead of doing it inside lazy_scan_noprune(). This is
more consistent with the lazy_scan_prune() case. lazy_scan_noprune()
still needs an output parameter for whether there are LP_DEAD items
on the page, but the real decision-making now happens in the caller.

Patch by me, reviewed by Peter Geoghegan and Melanie Plageman.

Discussion: http://postgr.es/m/CA+TgmoaOzvN1TcHd9iej=PR3fY40En1USxzOnXSR2CxCLaRM0g@mail.gmail.com
2024-01-16 14:16:57 -05:00
Peter Eisentraut 6995863157 Support identity columns in partitioned tables
Previously, identity columns were disallowed on partitioned tables.
(The reason was mainly that no one had gotten around to working
through all the details to make it work.)  This makes it work now.

Some details on the behavior:

* A newly created partition inherits identity property

  The partitions of a partitioned table are integral part of the
  partitioned table.  A partition inherits identity columns from the
  partitioned table.  An identity column of a partition shares the
  identity space with the corresponding column of the partitioned
  table.  In other words, the same identity column across all
  partitions of a partitioned table share the same identity space.
  This is effected by sharing the same underlying sequence.

  When INSERTing directly into a partition, the sequence associated
  with the topmost partitioned table is used to calculate the value of
  the corresponding identity column.

  In regular inheritance, identity columns and their properties in a
  child table are independent of those in its parent tables.  A child
  table does not inherit identity columns or their properties
  automatically from the parent.  (This is unchanged.)

* Attached partition inherits identity column

  A table being attached as a partition inherits the identity property
  from the partitioned table.  This should be fine since we expect
  that the partition table's column has the same type as the
  partitioned table's corresponding column.  If the table being
  attached is a partitioned table, the identity properties are
  propagated down its partition hierarchy.

  An identity column in the partitioned table is also marked as NOT
  NULL.  The corresponding column in the partition needs to be marked
  as NOT NULL for the attach to succeed.

* Drop identity property when detaching partition

  A partition's identity column shares the identity space
  (i.e. underlying sequence) as the corresponding column of the
  partitioned table.  If a partition is detached it can longer share
  the identity space as before.  Hence the identity columns of the
  partition being detached loose their identity property.

  When identity of a column of a regular table is dropped it retains
  the NOT NULL constraint that came with the identity property.
  Similarly the columns of the partition being detached retain the NOT
  NULL constraints that came with identity property, even though the
  identity property itself is lost.

  The sequence associated with the identity property is linked to the
  partitioned table (and not the partition being detached).  That
  sequence is not dropped as part of detach operation.

* Partitions with their own identity columns are not allowed.

* The usual ALTER operations (add identity column, add identity
  property to existing column, alter properties of an indentity
  column, drop identity property) are supported for partitioned
  tables.  Changing a column only in a partitioned table or a
  partition is not allowed; the change needs to be applied to the
  whole partition hierarchy.

Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/flat/CAExHW5uOykuTC+C6R1yDSp=o8Q83jr8xJdZxgPkxfZ1Ue5RRGg@mail.gmail.com
2024-01-16 17:24:52 +01:00
Daniel Gustafsson 7bcf57a0ca Decorate WITH with literal markup tags
One instance of "WITH clause" was not using <literal> tags around
WITH, while others were, so add markup to the last one to ensure
consistency.  Backpatch to v15 where MERGE was added.

Reported-by: jian he <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxGJKY9ZCPV2WDM6xFsXq9C8r7r3vU6AkScN+p9k6CEpMw@mail.gmail.com
Backpatch-through: v15
2024-01-16 13:51:15 +01:00
Heikki Linnakangas c6b86eaa55 Add missing PGDLLIMPORT markings
Since commit 8ec569479f, we have a policy of marking all backend
variables with PGDLLIMPORT.

Reported-by: Anton A. Melnikov
Discussion: https://www.postgresql.org/message-id/0b78546c-ffef-4cd9-9ba1-d1e6aab88cea@postgrespro.ru
2024-01-16 13:53:28 +02:00
Alvaro Herrera 5850253973
struct XmlTableRoutine: use C99 designated initializers
As in c27f8621ee et al.

Not as critical as other cases we've handled, but I figure if we're
going to add JsonbTableRoutine using TableFuncRoutine, this makes it
easier to jump around the code.
2024-01-16 12:48:30 +01:00