Commit Graph

52534 Commits

Author SHA1 Message Date
Heikki Linnakangas 3f968b9415 Fix ReadRecentBuffer for local buffers.
It incorrectly used GetBufferDescriptor instead of
GetLocalBufferDescriptor, causing it to not find the correct buffer in
most cases, and performing an out-of-bounds memory read in the corner
case that temp_buffers > shared_buffers.

It also bumped the usage-count on the buffer, even if it was
previously pinned. That won't lead to crashes or incorrect results,
but it's different from what the shared-buffer case does, and
different from the usual code in LocalBufferAlloc. Fix that too, and
make the code ordering match LocalBufferAlloc() more closely, so that
it's easier to verify that it's doing the same thing.

Currently, ReadRecentBuffer() is only used with non-temp relations, in
WAL redo, so the broken code is currently dead code. However, it could
be used by extensions.

Backpatch-through: 14
Discussion: https://www.postgresql.org/message-id/2d74b46f-27c9-fb31-7f99-327a87184cc0%40iki.fi
Reviewed-by: Thomas Munro, Zhang Mingli, Richard Guo
2022-07-25 08:53:27 +03:00
Tom Lane 31d5354cb1 Doc: improve documentation about random().
We didn't explicitly say that random() uses a randomly-chosen seed
if you haven't called setseed().  Do so.

Also, remove ref/set.sgml's no-longer-accurate (and never very
relevant) statement that the seed value is multiplied by 2^31-1.

Back-patch to v12 where set.sgml's claim stopped being true.
The claim that we use a source of random bits as seed was debatable
before 4203842a1, too, so v12 seems like a good place to stop.

Per question from Carl Sopchak.

Discussion: https://postgr.es/m/f37bb937-9d99-08f0-4de7-80c91a3cfc2e@sopchak.me
2022-07-23 19:00:30 -04:00
Thomas Munro fee0165fc1 Fix get_dirent_type() for Windows junction points.
Commit 87e6ed7c8 added code that intended to report Windows "junction
points" as DT_LNK (the same way we report symlinks on Unix).  Windows
junction points are *also* directories according to the Windows
attributes API, and we were reporting them as as DT_DIR.  Change the
order we check the attribute flags, to prioritize DT_LNK.

If at some point we start using Windows' recently added real symlinks
and need to distinguish them from junction points, we may need to
rethink this, but for now this continues the tradition of wrapper
functions that treat junction points as symlinks.

Back-patch to 14, where get_dirent_type() landed.

Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Alvaro Herrera <alvherre@alvh.no-ip.org>
Discussion: https://postgr.es/m/CA%2BhUKGLzLK4PUPx0_AwXEWXOYAejU%3D7XpxnYE55Y%2Be7hB2N3FA%40mail.gmail.com
Discussion: https://postgr.es/m/20220721111751.x7hod2xgrd76xr5c%40alvherre.pgsql
2022-07-22 16:57:57 +12:00
Fujii Masao 169d50ba34 postgres_fdw: Fix bug in checking of return value of PQsendQuery().
When postgres_fdw begins an asynchronous data fetch, it submits FETCH query
by using PQsendQuery(). If PQsendQuery() fails and returns 0, postgres_fdw
should report an error. But, previously, postgres_fdw reported an error
only when the return value is less than 0, though PQsendQuery() never return
the values other than 0 and 1. Therefore postgres_fdw could not handle
the failure to send FETCH query in an asynchronous data fetch.

This commit fixes postgres_fdw so that it reports an error
when PQsendQuery() returns 0.

Back-patch to v14 where asynchronous execution was supported in postgres_fdw.

Author: Fujii Masao
Reviewed-by: Japin Li, Tom Lane
Discussion: https://postgr.es/m/b187a7cf-d4e3-5a32-4d01-8383677797f3@oss.nttdata.com
2022-07-22 12:01:38 +09:00
Bruce Momjian e613466e46 doc: use wording "restore" instead of "reload" of dumps
Reported-by: axel.kluener@gmail.com

Discussion: https://postgr.es/m/164736074430.660.3645615289283943146@wrigleys.postgresql.org

Backpatch-through: 11
2022-07-21 14:55:23 -04:00
Bruce Momjian 21640e9865 doc: clarify that auth. names are lower case and case-sensitive
This is true even for acronyms that are usually upper case, like LDAP.

Reported-by: Alvaro Herrera

Discussion: https://postgr.es/m/202205141521.2nodjabmsour@alvherre.pgsql

Backpatch-through: 10
2022-07-21 13:58:39 -04:00
Tom Lane da9a28fd55 Fix ruleutils issues with dropped cols in functions-returning-composite.
Due to lack of concern for the case in the dependency code, it's
possible to drop a column of a composite type even though stored
queries have references to the dropped column via functions-in-FROM
that return the composite type.  There are "soft" references,
namely FROM-clause aliases for such columns, and "hard" references,
that is actual Vars referring to them.  The right fix for hard
references is to add dependencies preventing the drop; something
we've known for many years and not done (and this commit still doesn't
address it).  A "soft" reference shouldn't prevent a drop though.
We've been around on this before (cf. 9b35ddce9, 2c4debbd0), but
nobody had noticed that the current behavior can result in dump/reload
failures, because ruleutils.c can print more column aliases than the
underlying composite type now has.  So we need to rejigger the
column-alias-handling code to treat such columns as dropped and not
print aliases for them.

Rather than writing new code for this, I used expandRTE() which already
knows how to figure out which function result columns are dropped.
I'd initially thought maybe we could use expandRTE() in all cases, but
that fails for EXPLAIN's purposes, because the planner strips a lot of
RTE infrastructure that expandRTE() needs.  So this patch just uses it
for unplanned function RTEs and otherwise does things the old way.

If there is a hard reference (Var), then removing the column alias
causes us to fail to print the Var, since there's no longer a name
to print.  Failing seems less desirable than printing a made-up
name, so I made it print "?dropped?column?" instead.

Per report from Timo Stolz.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/5c91267e-3b6d-5795-189c-d15a55d61dbb@nullachtvierzehn.de
2022-07-21 13:56:02 -04:00
Fujii Masao be2e842c8a Fix assertion failure and segmentation fault in backup code.
When a non-exclusive backup is canceled, do_pg_abort_backup() is called
and resets some variables set by pg_backup_start (pg_start_backup in v14
or before). But previously it forgot to reset the session state indicating
whether a non-exclusive backup is in progress or not in this session.

This issue could cause an assertion failure when the session running
BASE_BACKUP is terminated after it executed pg_backup_start and
pg_backup_stop (pg_stop_backup in v14 or before). Also it could cause
a segmentation fault when pg_backup_stop is called after BASE_BACKUP
in the same session is canceled.

This commit fixes the issue by making do_pg_abort_backup reset
that session state.

Back-patch to all supported branches.

Author: Fujii Masao
Reviewed-by: Kyotaro Horiguchi, Masahiko Sawada, Michael Paquier, Robert Haas
Discussion: https://postgr.es/m/3374718f-9fbf-a950-6d66-d973e027f44c@oss.nttdata.com
2022-07-20 09:53:37 +09:00
Fujii Masao 2aedf25eb4 Prevent BASE_BACKUP in the middle of another backup in the same session.
Multiple non-exclusive backups are able to be run conrrently in different
sessions. But, in the same session, only one non-exclusive backup can be
run at the same moment. If pg_backup_start (pg_start_backup in v14 or before)
is called in the middle of another non-exclusive backup in the same session,
an error is thrown.

However, previously, in logical replication walsender mode, even if that
walsender session had already called pg_backup_start and started
a non-exclusive backup, it could execute BASE_BACKUP command and
start another non-exclusive backup. Which caused subsequent pg_backup_stop
to throw an error because BASE_BACKUP unexpectedly reset the session state
marked by pg_backup_start.

This commit prevents BASE_BACKUP command in the middle of another
non-exclusive backup in the same session.

Back-patch to all supported branches.

Author: Fujii Masao
Reviewed-by: Kyotaro Horiguchi, Masahiko Sawada, Michael Paquier, Robert Haas
Discussion: https://postgr.es/m/3374718f-9fbf-a950-6d66-d973e027f44c@oss.nttdata.com
2022-07-20 09:52:11 +09:00
Peter Eisentraut 8657946d37 Re-add SPICleanup for ABI compatibility in stable branch
This fixes an ABI break introduced by
604651880c.

Author: Markus Wanner <markus.wanner@enterprisedb.com>
Discussion: https://www.postgresql.org/message-id/defd749a-8410-841d-1126-21398686d63d@enterprisedb.com
2022-07-18 16:23:48 +02:00
Peter Eisentraut 9293589d96 pg_upgrade: Adjust quoting style in message to match guidelines 2022-07-18 14:55:54 +02:00
Tom Lane bb30410b9f Fix omissions in support for the "regcollation" type.
The patch that added regcollation doesn't seem to have been too
thorough about supporting it everywhere that other reg* types
are supported.  Fix that.  (The find_expr_references omission
is moderately serious, since it could result in missing expression
dependencies.  The others are less exciting.)

Noted while fixing bug #17483.  Back-patch to v13 where
regcollation was added.

Discussion: https://postgr.es/m/1423433.1652722406@sss.pgh.pa.us
2022-07-17 17:43:28 -04:00
Tom Lane 810bcbd383 postgres_fdw: set search_path to 'pg_catalog' while deparsing constants.
The motivation for this is to ensure successful transmission of the
values of constants of regconfig and other reg* types.  The remote
will be reading them with search_path = 'pg_catalog', so schema
qualification is necessary when referencing objects in other schemas.

Per bug #17483 from Emmanuel Quincerot.  Back-patch to all supported
versions.  (There's some other stuff to do here, but it's less
back-patchable.)

Discussion: https://postgr.es/m/1423433.1652722406@sss.pgh.pa.us
2022-07-17 17:27:50 -04:00
Thomas Munro c412c60b91 Make dsm_impl_posix_resize more future-proof.
Commit 4518c798 blocks signals for a short region of code, but it
assumed that whatever called it had the signal mask set to UnBlockSig on
entry.  That may be true today (or may even not be, in extensions in the
wild), but it would be better not to make that assumption.  We should
save-and-restore the caller's signal mask.

The PG_SETMASK() portability macro couldn't be used for that, which is
why it wasn't done before.  But... considering that commit a65e0864
established back in 9.6 that supported POSIX systems have sigprocmask(),
and that this is POSIX-only code, there is no reason not to use standard
sigprocmask() directly to achieve that.

Back-patch to all supported releases, like 4518c798 and 80845b7c.

Reviewed-by: Alvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CA%2BhUKGKx6Biq7_UuV0kn9DW%2B8QWcpJC1qwhizdtD9tN-fn0H0g%40mail.gmail.com
2022-07-16 12:23:14 +12:00
John Naylor 2ebb8416cc Clarify that pg_dump takes ACCESS SHARE lock
Add link to the description of lock levels to avoid confusing "shared locks"
with SHARE locks.

Florin Irion

Reviewed-by: Álvaro Herrera, Tom Lane, and Nathan Bossart
Discussion: https://www.postgresql.org/message-id/flat/d0f30cc2-3c76-1d43-f291-7c4b2872d653@gmail.com

This is a backpatch of 4e2e8d71f, applied through version 14
2022-07-15 08:21:09 +07:00
Bruce Momjian e1d5ac3118 docs: make monitoring "phases" table titles consistent
Reported-by: Nitin Jadhav

Discussion: https://postgr.es/m/CAMm1aWbmTHwHKC2PERH0CCaFVPoxrtLeS8=wNuoge94qdSp3vA@mail.gmail.com

Author: Nitin Jadhav

Backpatch-through: 13
2022-07-14 20:01:11 -04:00
Bruce Momjian 2fc2d805e9 doc: clarify how dropping of extensions affects dependent objs.
Clarify that functions/procedures are dropped when any extension that
depends on them is dropped.

Reported-by: David G. Johnston

Discussion: https://postgr.es/m/CAKFQuwbPSHMDGkisRUmewopweC1bFvytVqB=a=X4GFg=4ZWxPA@mail.gmail.com

Backpatch-through: 13
2022-07-14 17:41:03 -04:00
Bruce Momjian 0d8db8cf85 pg_upgrade doc: mention that replication slots must be recreated
Reported-by: Nikhil Shetty

Discussion: https://postgr.es/m/CAFpL5Vxastip0Jei-K-=7cKXTg=5sahSe5g=om=x68NOX8+PUA@mail.gmail.com

Backpatch-through: 10
2022-07-14 16:34:30 -04:00
Bruce Momjian 8e97474834 doc: add documentation about ecpg Oracle-compatibility mode
Reported-by: Takeshi Ideriha

Discussion: https://postgr.es/m/TYCPR01MB7041A157067208327D8DAAF9EAA59@TYCPR01MB7041.jpnprd01.prod.outlook.com

Backpatch-through: 11
2022-07-14 16:19:45 -04:00
Bruce Momjian 8f253ba251 doc: clarify the behavior of identically-named savepoints
Original patch by David G. Johnston.

Reported-by: David G. Johnston

Discussion: https://postgr.es/m/CAKFQuwYQCxSSuSL18skCWG8QHFswOJ3hjovHsOZUE346i4OpVQ@mail.gmail.com

Backpatch-through: 10
2022-07-14 15:44:22 -04:00
Bruce Momjian 3bfe26bd4c doc: clarify that "excluded" ON CONFLICT is a single row
Original patch by David G. Johnston.

Reported-by: David G. Johnston

Discussion: https://postgr.es/m/CAKFQuwa4J0+WuO7kW1PLbjoEvzPN+Q_j+P2bXxNnCLaszY7ZdQ@mail.gmail.com

Backpatch-through: 10
2022-07-14 15:33:28 -04:00
Bruce Momjian 4996786a90 doc: mention that INSERT can block because of unique indexes
Initial patch by David G. Johnston.

Reported-by: David G. Johnston

Discussion: https://postgr.es/m/CAKFQuwZpbdzceO41VE-xt1Xh8rWRRfgopTAK1wL9EhCo0Am-Sw@mail.gmail.com

Backpatch-through: 10
2022-07-14 15:17:19 -04:00
Bruce Momjian 6396ab3d14 doc: mention the pg_locks lock names in parentheses
Reported-by: Troy Frericks

Discussion: https://postgr.es/m/165653551130.665.8240515669521441325@wrigleys.postgresql.org

Backpatch-through: 10
2022-07-14 12:08:54 -04:00
Thomas Munro 8383645592 Don't clobber postmaster sigmask in dsm_impl_resize.
Commit 4518c798 intended to block signals in regular backends that
allocate DSM segments, but dsm_impl_resize() is also reached by
dsm_postmaster_startup().  It's not OK to clobber the postmaster's
signal mask, so only manipulate the signal mask when under the
postmaster.

Back-patch to all releases, like 4518c798.

Discussion: https://postgr.es/m/CA%2BhUKGKNpK%3D2OMeea_AZwpLg7Bm4%3DgYWk7eDjZ5F6YbozfOf8w%40mail.gmail.com
2022-07-15 02:02:28 +12:00
Thomas Munro 2019e6ecfa Block signals while allocating DSM memory.
On Linux, we call posix_fallocate() on shm_open()'d memory to avoid
later potential SIGBUS (see commit 899bd785).

Based on field reports of systems stuck in an EINTR retry loop there,
there, we made it possible to break out of that loop via slightly odd
coding where the CHECK_FOR_INTERRUPTS() call was somewhat removed from
the loop (see commit 422952ee).

On further reflection, that was not a great choice for at least two
reasons:

1.  If interrupts were held, the CHECK_FOR_INTERRUPTS() would do nothing
and the EINTR error would be surfaced to the user.

2.  If EINTR was reported but neither QueryCancelPending nor
ProcDiePending was set, then we'd dutifully retry, but with a bit more
understanding of how posix_fallocate() works, it's now clear that you
can get into a loop that never terminates.  posix_fallocate() is not a
function that can do some of the job and tell you about progress if it's
interrupted, it has to undo what it's done so far and report EINTR, and
if signals keep arriving faster than it can complete (cf recovery
conflict signals), you're stuck.

Therefore, for now, we'll simply block most signals to guarantee
progress.  SIGQUIT is not blocked (see InitPostmasterChild()), because
its expected handler doesn't return, and unblockable signals like
SIGCONT are not expected to arrive at a high rate.  For good measure,
we'll include the ftruncate() call in the blocked region, and add a
retry loop.

Back-patch to all supported releases.

Reported-by: Alvaro Herrera <alvherre@alvh.no-ip.org>
Reported-by: Nicola Contu <nicola.contu@gmail.com>
Reviewed-by: Alvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220701154105.jjfutmngoedgiad3%40alvherre.pgsql
2022-07-14 17:48:32 +12:00
Alvaro Herrera 9e038d6907
Plug memory leak
Commit 054325c5ee created a memory leak in PQsendQueryInternal in case
an error occurs while sending the message.  Repair.

Backpatch to 14, like that commit.  Reported by Coverity.
2022-07-13 12:10:03 +02:00
Tom Lane af72b08894 Invent qsort_interruptible().
Justin Pryzby reported that some scenarios could cause gathering
of extended statistics to spend many seconds in an un-cancelable
qsort() operation.  To fix, invent qsort_interruptible(), which is
just like qsort_arg() except that it will also do CHECK_FOR_INTERRUPTS
every so often.  This bloats the backend by a couple of kB, which
seems like a good investment.  (We considered just enabling
CHECK_FOR_INTERRUPTS in the existing qsort and qsort_arg functions,
but there are some callers for which that'd demonstrably be unsafe.
Opt-in seems like a better way.)

For now, just apply qsort_interruptible() in statistics collection.
There's probably more places where it could be useful, but we can
always change other call sites as we find problems.

Back-patch to v14.  Before that we didn't have extended stats on
expressions, so that the problem was less severe.  Also, this patch
depends on the sort_template infrastructure introduced in v14.

Tom Lane and Justin Pryzby

Discussion: https://postgr.es/m/20220509000108.GQ28830@telsasoft.com
2022-07-12 16:30:36 -04:00
Thomas Munro 5e7608e81e Fix lock assertions in dshash.c.
dshash.c previously maintained flags to be able to assert that you
didn't hold any partition lock.  These flags could get out of sync with
reality in error scenarios.

Get rid of all that, and make assertions about the locks themselves
instead.  Since LWLockHeldByMe() loops internally, we don't want to put
that inside another loop over all partition locks.  Introduce a new
debugging-only interface LWLockAnyHeldByMe() to avoid that.

This problem was noted by Tom and Andres while reviewing changes to
support the new shared memory stats system, and later showed up in
reality while working on commit 389869af.

Back-patch to 11, where dshash.c arrived.

Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Reported-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220311012712.botrpsikaufzteyt@alap3.anarazel.de
Discussion: https://postgr.es/m/CA%2BhUKGJ31Wce6HJ7xnVTKWjFUWQZPBngxfJVx4q0E98pDr3kAw%40mail.gmail.com
2022-07-11 15:47:12 +12:00
Michael Paquier ec5f1fe2fc doc: Fix inconsistent quotes in some jsonb fields
Single quotes are not allowed in json internals, double quotes are.

Reported-by: Eric Mutta
Discussion: https://postgr.es/m/165715362165.665.3875113264927503997@wrigleys.postgresql.org
Backpatch-through: 14
2022-07-11 10:56:48 +09:00
Thomas Munro ab7fef0acf Fix \watch's interaction with libedit on ^C.
When you hit ^C, the terminal driver in Unix-like systems echoes "^C" as
well as sending an interrupt signal (depending on stty settings).  At
least libedit (but maybe also libreadline) is then confused about the
current cursor location, and corrupts the display if you try to scroll
back.  Fix, by moving to a new line before the next prompt is displayed.

Back-patch to all supported released.

Author: Pavel Stehule <pavel.stehule@gmail.com>
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/3278793.1626198638%40sss.pgh.pa.us
2022-07-10 16:53:18 +12:00
Bruce Momjian f1c779e2b9 doc: add examples for array_length() and jsonb_array_length()
The examples show the output of array_length() and jsonb_array_length()
for empty arrays.

Discussion: https://postgr.es/m/CAKFQuwaoBmRuWdMLzLHDCFDJDX3wvfQ7egAF0bpik_BFgG1KWg@mail.gmail.com

Author: David G. Johnston

Backpatch-through: 13
2022-07-08 20:23:35 -04:00
Bruce Momjian 9d71664c88 doc: add pg_prewarm example
Discussion: https://postgr.es/m/20220618085541.ezxdaljlpo6x7msc@home-desktop

Author: Dong Wook Lee

Backpatch-through: 11
2022-07-08 18:36:27 -04:00
Dean Rasheed 8d8464445a Fix alias matching in transformLockingClause().
When locking a specific named relation for a FOR [KEY] UPDATE/SHARE
clause, transformLockingClause() finds the relation to lock by
scanning the rangetable for an RTE with a matching eref->aliasname.
However, it failed to account for the visibility rules of a join RTE.

If a join RTE doesn't have a user-supplied alias, it will have a
generated eref->aliasname of "unnamed_join" that is not visible as a
relation name in the parse namespace. Such an RTE needs to be skipped,
otherwise it might be found in preference to a regular base relation
with a user-supplied alias of "unnamed_join", preventing it from being
locked.

In addition, if a join RTE doesn't have a user-supplied alias, but
does have a join_using_alias, then the RTE needs to be matched using
that alias rather than the generated eref->aliasname, otherwise a
misleading "relation not found" error will be reported rather than a
"join cannot be locked" error.

Backpatch all the way, except for the second part which only goes back
to 14, where JOIN USING aliases were added.

Dean Rasheed, reviewed by Tom Lane.

Discussion: https://postgr.es/m/CAEZATCUY_KOBnqxbTSPf=7fz9HWPnZ5Xgb9SwYzZ8rFXe7nb=w@mail.gmail.com
2022-07-07 13:08:03 +01:00
Tom Lane 9783413cbf Tighten pg_upgrade's new check for non-upgradable anyarray usages.
We only need to reject cases when the aggregate or operator is
itself declared with a polymorphic type.  Per buildfarm.

Discussion: https://postgr.es/m/3383880.QJadu78ljV@vejsadalnx
2022-07-05 18:23:19 -04:00
Tom Lane 175e60a5e3 Fix pg_upgrade to detect non-upgradable anyarray usages.
When we changed some built-in functions to use anycompatiblearray
instead of anyarray, we created a dump/restore hazard for user-defined
operators and aggregates relying on those functions: the user objects
have to be modified to change their signatures similarly.  This causes
pg_upgrade to fail partway through if the source installation contains
such objects.  We generally try to have pg_upgrade detect such hazards
and fail before it does anything exciting, so add logic to detect
this case too.

Back-patch to v14 where the change was made.

Justin Pryzby, reviewed by Andrey Borodin

Discussion: https://postgr.es/m/3383880.QJadu78ljV@vejsadalnx
2022-07-05 13:06:31 -04:00
Alvaro Herrera 7c1f426123
libpq: Improve idle state handling in pipeline mode
We were going into IDLE state too soon when executing queries via
PQsendQuery in pipeline mode, causing several scenarios to misbehave in
different ways -- most notably, as reported by Daniele Varrazzo, that a
warning message is produced by libpq:
  message type 0x33 arrived from server while idle
But it is also possible, if queries are sent and results consumed not in
lockstep, for the expected mediating NULL result values from PQgetResult
to be lost (a problem which has not been reported, but which is more
serious).

Fix this by introducing two new concepts: one is a command queue element
PGQUERY_CLOSE to tell libpq to wait for the CloseComplete server
response to the Close message that is sent by PQsendQuery.  Because the
application is not expecting any PGresult from this, the mechanism to
consume it is a bit hackish.

The other concept, authored by Horiguchi-san, is a PGASYNC_PIPELINE_IDLE
state for libpq's state machine to differentiate "really idle" from
merely "the idle state that occurs in between reading results from the
server for elements in the pipeline".  This makes libpq not go fully
IDLE when the libpq command queue contains entries; in normal cases, we
only go IDLE once at the end of the pipeline, when the server response
to the final SYNC message is received.  (However, there are corner cases
it doesn't fix, such as terminating the query sequence by
PQsendFlushRequest instead of PQpipelineSync; this sort of scenario is
what requires PGQUERY_CLOSE bit above.)

This last bit helps make the libpq state machine clearer; in particular
we can get rid of an ugly hack in pqParseInput3 to avoid considering
IDLE as such when the command queue contains entries.

A new test mode is added to libpq_pipeline.c to tickle some related
problematic cases.

Reported-by: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Co-authored-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Discussion: https://postgr.es/m/CA+mi_8bvD0_CW3sumgwPvWdNzXY32itoG_16tDYRu_1S2gV2iw@mail.gmail.com
2022-07-05 14:21:20 +02:00
Alvaro Herrera 0b71e43c48
BRIN: improve documentation on summarization
The existing wording wasn't clear enough and some details weren't
anywhere, such as the fact that autosummarization is off by default.
Improve.

Authors: Roberto Mello, Jaime Casanova, Justin Pryzby, Álvaro Herrera
Discussion: https://postgr.es/m/CAKz==bK_NoJytRyQfX8K-erCW3Ff7--oGYpiB8+ePVS7dRVW_A@mail.gmail.com
Discussion: https://postgr.es/m/20220224193520.GY9008@telsasoft.com
2022-07-05 13:38:26 +02:00
Andrew Dunstan 7fd43684fc Remove %error-verbose directive from jsonpath parser
None of the other bison parsers contains this directive, and it gives
rise to some unfortunate and impenetrable messages, so just remove it.

Backpatch to release 12, where it was introduced.

Per gripe from Erik Rijkers

Discussion: https://postgr.es/m/ba069ce2-a98f-dc70-dc17-2ccf2a9bf7c7@xs4all.nl
2022-07-03 17:16:11 -04:00
Noah Misch 463a841d74 Fix previous commit's ecpg_clocale for ppc Darwin.
Per buildfarm member prairiedog, this platform rejects uninitialized
global variables in shared libraries.  Back-patch to v10, like the
addition of the variable.

Reviewed by Tom Lane.

Discussion: https://postgr.es/m/20220703030619.GB2378460@rfd.leadboat.com
2022-07-02 21:03:22 -07:00
Noah Misch 5b94e2bd4d ecpglib: call newlocale() once per process.
ecpglib has been calling it once per SQL query and once per EXEC SQL GET
DESCRIPTOR.  Instead, if newlocale() has not succeeded before, call it
while establishing a connection.  This mitigates three problems:
- If newlocale() failed in EXEC SQL GET DESCRIPTOR, the command silently
  proceeded without the intended locale change.
- On AIX, each newlocale()+freelocale() cycle leaked memory.
- newlocale() CPU usage may have been nontrivial.

Fail the connection attempt if newlocale() fails.  Rearrange
ecpg_do_prologue() to validate the connection before its uselocale().

The sort of program that may regress is one running in an environment
where newlocale() fails.  If that program establishes connections
without running SQL statements, it will stop working in response to this
change.  I'm betting against the importance of such an ECPG use case.
Most SQL execution (any using ECPGdo()) has long required newlocale()
success, so there's little a connection could do without newlocale().

Back-patch to v10 (all supported versions).

Reviewed by Tom Lane.  Reported by Guillaume Lelarge.

Discussion: https://postgr.es/m/20220101074055.GA54621@rfd.leadboat.com
2022-07-02 13:00:34 -07:00
Thomas Munro fb81a93a64 Harden dsm_impl.c against unexpected EEXIST.
Previously, we trusted the OS not to report EEXIST unless we'd passed in
IPC_CREAT | IPC_EXCL or O_CREAT | O_EXCL, as appropriate.  Solaris's
shm_open() can in fact do that, causing us to crash because we didn't
ereport and then we blithely assumed the mapping was successful.

Let's treat EEXIST just like any other error, unless we're actually
trying to create a new segment.  This applies to shm_open(), where this
behavior has been seen, and also to the equivalent operations for our
sysv and mmap modes just on principle.

Based on the underlying reason for the error, namely contention on a
lock file managed by Solaris librt for each distinct name, this problem
is only likely to happen on 15 and later, because the new shared memory
stats system produces shm_open() calls for the same path from
potentially large numbers of backends concurrently during
authentication.  Earlier releases only shared memory segments between a
small number of parallel workers under one Gather node.  You could
probably hit it if you tried hard enough though, and we should have been
more defensive in the first place.  Therefore, back-patch to all
supported releases.

Per build farm animal margay.  This isn't the end of the story, though,
it just changes random crashes into random "File exists" errors; more
work needed for a green build farm.

Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKqKrCV5xKWfh9rnm%3Do%3DDwZLTLtnsj_XpUi9g5%3DV%2B9oyg%40mail.gmail.com
2022-07-01 13:16:27 +12:00
Heikki Linnakangas e24615a005 Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:

    ERROR:  t_xmin is uncommitted in tuple to be updated

or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.

The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.

Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.

This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.

TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.

Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.

Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.

Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
2022-06-27 08:24:30 +03:00
Thomas Munro 99504ff826 Fix relptr's encoding of the base address.
Previously, we encoded both NULL and the first byte at the base address
as 0.  That confusion led to the assertion in commit e07d4ddc, which
failed when min_dynamic_shared_memory was used.  Give them distinct
encodings, by switching to 1-based offsets for non-NULL pointers.  Also
improve macro hygiene in passing (missing/misplaced parentheses), and
remove open-coded access to the raw offset value from freepage.c/h.

Although e07d4ddc was back-patched to 10, the only code that actually
makes use of relptr at the base address arrived in 84b1c63a, so no need
to back-patch further than 14 for now.

Reported-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Discussion: https://postgr.es/m/20220519193839.GT19626%40telsasoft.com
2022-06-27 11:45:03 +12:00
Thomas Munro e086b55381 Don't trust signalfd() on illumos.
Since commit 6a2a70a02, we've used signalfd() to receive latch wakeups
when building with WAIT_USE_EPOLL (default for Linux and illumos), and
our traditional self-pipe when falling back to WAIT_USE_POLL (default
for other Unixes with neither epoll() nor kqueue()).

Unexplained hangs and kernel panics have been reported on illumos
systems, apparently linked to this use of signalfd(), leading illumos
users and build farm members to have to define WAIT_USE_POLL explicitly
as a work-around.  A bug report exists at
https://www.illumos.org/issues/13700 but no fix is available yet.

Let's provide a way for illumos users to go back to self-pipes with
epoll(), like releases before 14, and choose that by default.  No change
for Linux users.  To help with development/debugging, macros
WAIT_USE_{EPOLL,POLL} and WAIT_USE_{SIGNALFD,SELF_PIPE} can be defined
explicitly to override the defaults.

Back-patch to 14, where we started using signalfd().

Reported-by: Japin Li <japinli@hotmail.com>
Reported-by: Olaf Bohlen <olbohlen@eenfach.de> (off-list)
Reviewed-by: Japin Li <japinli@hotmail.com>
Discussion: https://postgr.es/m/MEYP282MB1669C8D88F0997354C2313C1B6CA9%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
2022-06-26 11:00:04 +12:00
Noah Misch 4b0d21b06c Fix PostgreSQL::Test aliasing for Perl v5.10.1.
This Perl segfaults if a declaration of the to-be-aliased package
precedes the aliasing itself.  Per buildfarm members lapwing and wrasse.
Like commit 20911775de, back-patch to v10
(all supported versions).

Discussion: https://postgr.es/m/20220625171533.GA2012493@rfd.leadboat.com
2022-06-25 14:15:56 -07:00
Noah Misch ace9973867 CREATE INDEX: use the original userid for more ACL checks.
Commit a117cebd63 used the original userid
for ACL checks located directly in DefineIndex(), but it still adopted
the table owner userid for more ACL checks than intended.  That broke
dump/reload of indexes that refer to an operator class, collation, or
exclusion operator in a schema other than "public" or "pg_catalog".
Back-patch to v10 (all supported versions), like the earlier commit.

Nathan Bossart and Noah Misch

Discussion: https://postgr.es/m/f8a4105f076544c180a87ef0c4822352@stmuk.bayern.de
2022-06-25 09:07:44 -07:00
Noah Misch 20911775de For PostgreSQL::Test compatibility, alias entire package symbol tables.
Remove the need to edit back-branch-specific code sites when
back-patching the addition of a PostgreSQL::Test::Utils symbol.  Replace
per-symbol, incomplete alias lists.  Give old and new package names the
same EXPORT and EXPORT_OK semantics.  Back-patch to v10 (all supported
versions).

Reviewed by Andrew Dunstan.

Discussion: https://postgr.es/m/20220622072144.GD4167527@rfd.leadboat.com
2022-06-25 09:07:44 -07:00
Amit Kapila 3238b5c330 Fix memory leak due to LogicalRepRelMapEntry.attrmap.
When rebuilding the relation mapping on subscribers, we were not releasing
the attribute mapping's memory which was no longer required.

The attribute mapping used in logical tuple conversion was refactored in
PG13 (by commit e1551f96e6) but we forgot to update the related code that
frees the attribute map.

Author: Hou Zhijie
Reviewed-by: Amit Langote, Amit Kapila, Shi yu
Backpatch-through: 10, where it was introduced
Discussion: https://postgr.es/m/OSZPR01MB6310F46CD425A967E4AEF736FDA49@OSZPR01MB6310.jpnprd01.prod.outlook.com
2022-06-23 09:20:41 +05:30
Bruce Momjian f1e3a70758 doc: improve wording of plpgsql RAISE format text
Reported-by: pg@kirasoft.com

Discussion: https://postgr.es/m/165455351426.573551.7050474465030525109@wrigleys.postgresql.org

Backpatch-through: 10
2022-06-22 16:59:54 -04:00
Bruce Momjian 1463f22d42 doc: clarify wording about phantom reads
Reported-by: akhilhello@gmail.com

Discussion: https://postgr.es/m/165222922369.669.10475917322916060899@wrigleys.postgresql.org

Backpatch-through: 10
2022-06-22 14:33:45 -04:00