Commit Graph

46462 Commits

Author SHA1 Message Date
Bruce Momjian 9e8aec433a doc: add namespace column to pg_buffercache example query
Without the namespace, the table name could be ambiguous.

Reported-by: adunham@arbormetrix.com

Discussion: https://postgr.es/m/158155175140.23798.2189464781144503491@wrigleys.postgresql.org

Backpatch-through: 9.5
2020-03-31 17:16:33 -04:00
Bruce Momjian f116356deb doc: clarify which table creation is used for inheritance part.
Previously people might assume that the partition syntax version of
CREATE TABLE is to be used for the inheritance partition table example;
mention that the non-partitioned version should be used.

Reported-by: mib@nic.at

Discussion: https://postgr.es/m/158089540905.1098.15071165437284409576@wrigleys.postgresql.org

Backpatch-through: 10
2020-03-31 17:07:43 -04:00
Bruce Momjian 3ee692d82f doc: adjust UPDATE/DELETE's FROM/USING to match SELECT's FROM
Previously the syntax and wording were unclear.

Reported-by: Alexey Bashtanov

Discussion: https://postgr.es/m/968d4724-8e58-788f-7c45-f7b1813824cc@imap.cc

Backpatch-through: 9.5
2020-03-31 16:31:44 -04:00
Tom Lane 5b22ff764b Teach pg_ls_dir_files() to ignore ENOENT failures from stat().
Buildfarm experience shows that this function can fail with ENOENT
if some other process unlinks a file between when we read the directory
entry and when we try to stat() it.  The problem is old but we had
not noticed it until 085b6b667 added regression test coverage.

To fix, just ignore ENOENT failures.  There is one other case that
this might hide: a symlink that points to nowhere.  That seems okay
though, at least better than erroring.

Back-patch to v10 where this function was added, since the regression
test cases were too.

Discussion: https://postgr.es/m/20200308173103.GC1357@telsasoft.com
2020-03-31 12:57:55 -04:00
Tom Lane fade4d4dff Back-patch addition of stack overflow and interrupt checks for lquery.
Experimentation shows that it's not hard at all to drive the
old implementation of "ltree ~ lquery" match to stack overflow,
so throw in a check_stack_depth() call, as I just did in HEAD.

I wasn't able to make it take a long time, because all the
pathological cases I tried hit stack overflow first; but
I bet there are some others that do take a long time, so add
CHECK_FOR_INTERRUPTS() too.

Discussion: https://postgr.es/m/CAP_rww=waX2Oo6q+MbMSiZ9ktdj6eaJj0cQzNu=Ry2cCDij5fw@mail.gmail.com
2020-03-31 11:37:44 -04:00
Tom Lane 4bc3a16677 Be more careful about extracting encoding from locale strings on Windows.
GetLocaleInfoEx() can fail on strings that setlocale() was perfectly
happy with.  A common way for that to happen is if the locale string
is actually a Unix-style string, say "et_EE.UTF-8".  In that case,
what's after the dot is an encoding name, not a Windows codepage number;
blindly treating it as a codepage number led to failure, with a fairly
silly error message.  Hence, check to see if what's after the dot is
all digits, and if not, treat it as a literal encoding name rather than
a codepage number.  This will do the right thing with many Unix-style
locale strings, and produce a more sensible error message otherwise.

Somewhat independently of that, treat a zero (CP_ACP) result from
GetLocaleInfoEx() as meaning that we must use UTF-8 encoding.

Back-patch to all supported branches.

Juan José Santamaría Flecha

Discussion: https://postgr.es/m/24905.1585445371@sss.pgh.pa.us
2020-03-30 11:14:58 -04:00
Tom Lane 550041728d Doc: correct misstatement about ltree label maximum length.
The documentation says that the max length is 255 bytes, but
code inspection says it's actually 255 characters; and relevant
lengths are stored as uint16 so that that works.
2020-03-29 18:54:19 -04:00
Tom Lane 5feb3d0b3f Protect against overflow of ltree.numlevel and lquery.numlevel.
These uint16 fields could be overflowed by excessively long input,
producing strange results.  Complain for invalid input.

Likewise check for out-of-range values of the repeat counts in lquery.
(We don't try too hard on that one, notably not bothering to detect
if atoi's result has overflowed.)

Also detect length overflow in ltree_concat.

In passing, be more consistent about whether "syntax error" messages
include the type name.  Also, clarify the documentation about what
the size limit is.

This has been broken for a long time, so back-patch to all supported
branches.

Nikita Glukhov, reviewed by Benjie Gillam and Tomas Vondra

Discussion: https://postgr.es/m/CAP_rww=waX2Oo6q+MbMSiZ9ktdj6eaJj0cQzNu=Ry2cCDij5fw@mail.gmail.com
2020-03-28 17:09:51 -04:00
Andres Freund 00a0a428ef Ensure snapshot is registered within ScanPgRelation().
In 9.4 I added support to use a historical snapshot in
ScanPgRelation(), while adding logical decoding. Unfortunately a
conflict with the concurrent removal of SnapshotNow was incorrectly
resolved, leading to an unregistered snapshot being used.

It is not correct to use an unregistered (or non-active) snapshot for
anything non-trivial, because catalog invalidations can cause the
snapshot to be invalidated.

Luckily it seems unlikely to actively cause problems in practice, as
ScanPgRelation() requires that we already have a lock on the relation,
we only look for a single row, and we don't appear to rely on the
result's tid to be correct. It however is clearly wrong and potential
negative consequences would likely be hard to find. So it seems worth
backpatching the fix, even without a concrete hazard.

Discussion: https://postgr.es/m/20200229052459.wzhqnbhrriezg4v2@alap3.anarazel.de
Backpatch: 9.5-
2020-03-28 12:04:49 -07:00
Tom Lane deeda011b3 Ensure that plpgsql cleans up cleanly during parallel-worker exit.
plpgsql_xact_cb ought to treat events XACT_EVENT_PARALLEL_COMMIT and
XACT_EVENT_PARALLEL_ABORT like XACT_EVENT_COMMIT and XACT_EVENT_ABORT
respectively, since its goal is to do process-local cleanup.  This
oversight caused plpgsql's end-of-transaction cleanup to not get done
in parallel workers.  Since a parallel worker will exit just after the
transaction cleanup, the effects of this are limited.  I couldn't find
any case in the core code with user-visible effects, but perhaps there
are some in extensions.  In any case it's wrong, so let's fix it before
it bites us not after.

In passing, add some comments around the handling of expression
evaluation resources in DO blocks.  There's no live bug there, but it's
quite unobvious what's happening; at least I thought so.  This isn't
related to the other issue, except that I found both things while poking
at expression-evaluation performance.

Back-patch the plpgsql_xact_cb fix to 9.5 where those event types
were introduced, and the DO-block commentary to v11 where DO blocks
gained the ability to issue COMMIT/ROLLBACK.

Discussion: https://postgr.es/m/10353.1585247879@sss.pgh.pa.us
2020-03-26 18:06:55 -04:00
Peter Eisentraut ba4cc05ce2 Drop slot's LWLock before returning from SaveSlotToPath()
When SaveSlotToPath() is called with elevel=LOG, the early exits didn't
release the slot's io_in_progress_lock.

This could result in a walsender being stuck on the lock forever.  A
possible way to get into this situation is if the offending code paths
are triggered in a low disk space situation.

Author: Pavan Deolasee <pavan.deolasee@2ndquadrant.com>
Reported-by: Craig Ringer <craig@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/56a138c5-de61-f553-7e8f-6789296de785%402ndquadrant.com
2020-03-26 13:51:07 +01:00
Andres Freund 58995db07e Add regression tests for constraint errors in partitioned tables.
While #16293 only applied to 11 (and 10 to some degree), it seems best
to add tests to all branches with partitioning support.

Reported-By: Daniel WM
Author: Andres Freund
Bug: #16293
Discussion: https://postgr.es/m/16293-26f5777d10143a66@postgresql.org
Backpatch: 10-
2020-03-23 14:52:18 -07:00
Andres Freund d35631e37c Fix potential crash after constraint violation errors in partitioned tables.
During the reporting of constraint violations for partitioned tables,
ExecPartitionCheckEmitError(), ExecConstraints(),
ExecWithCheckOptions() set the slot descriptor of the input slot to
the root partition's tuple desc.  That's generally problematic when
the slot could be used by other routines, but can cause crashes after
the introduction of slots with "fixed" tuple descriptors in
ad7dbee368.

The problem likely escaped detection so far for two reasons: First,
currently the only known way that these routines are used with a
partitioned table that is not "owned" by partitioning code is when
"fast defaults" are used for the child partition. Second, as an error
is raised afterwards, an "external" slot that had its descriptor
changed, is very unlikely to continue being used.

Even though the issue currently is only known to cause a crash for
11 (as that has both fast defaults and "fixed" slot descriptors), it
seems worth applying the fix to 10 too. Potentially changing random
slots is hazardous.

Regression tests will be added in a separate commit, as it seems best
to add them for master and 12 too.

Reported-By: Daniel WM
Author: Andres Freund
Bug: #16293
Discussion: https://postgr.es/m/16293-26f5777d10143a66@postgresql.org
Backpatch: 11, 10 only
2020-03-23 14:52:18 -07:00
Tom Lane 25bf317ccd Doc: explain that LIKE et al can be used in ANY (sub-select) etc.
This wasn't stated anywhere, and it's perhaps not that obvious,
since we get questions about it from time to time.  Also undocumented
was that the parser actually translates these into operators.

Discussion: https://postgr.es/m/CAFj8pRBkvZ71BqGKZnBBG4=0cKG+s50Dy+DYmrizUKEpAtdc+w@mail.gmail.com
2020-03-23 12:42:15 -04:00
Tom Lane aae7e8530d Fix our getopt_long's behavior for a command line argument of just "-".
src/port/getopt_long.c failed on such an argument, always seeing it
as an unrecognized switch.  This is unhelpful; better is to treat such
an item as a non-switch argument.  That behavior is what we find in
GNU's getopt_long(); it's what src/port/getopt.c does; and it is
required by POSIX for getopt(), which getopt_long() ought to be
generally a superset of.  Moreover, it's expected by ecpg, which
intends an argument of "-" to mean "read from stdin".  So fix it.

Also add some documentation about ecpg's behavior in this area, since
that was miserably underdocumented.  I had to reverse-engineer it
from the code.

Per bug #16304 from James Gray.  Back-patch to all supported branches,
since this has been broken forever.

Discussion: https://postgr.es/m/16304-c662b00a1322db7f@postgresql.org
2020-03-23 11:58:01 -04:00
Michael Paquier 26a26d9f2b Doc: Fix type of some storage parameters in CREATE TABLE page
autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor have
been documented as "float4", but "floating type" is used in this case
for GUCs and relation options in the documentation.

Author: Atsushi Torikoshi
Discussion: https://postgr.es/m/CACZ0uYFf_p9BpbjLccx3CA=eM1Hk2Te=ULY4iptGLUhL-JxCPA@mail.gmail.com
Backpatch-through: 9.5
2020-03-23 13:38:14 +09:00
Noah Misch 2fbdebc248 Revert "Skip WAL for new relfilenodes, under wal_level=minimal."
This reverts commit cb2fd7eac2.  Per
numerous buildfarm members, it was incompatible with parallel query, and
a test case assumed LP64.  Back-patch to 9.5 (all supported versions).

Discussion: https://postgr.es/m/20200321224920.GB1763544@rfd.leadboat.com
2020-03-22 09:24:13 -07:00
Noah Misch 03b89f1949 Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this.  If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY.  See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules.  Maintainers of table access
methods should examine that section.

To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL.  A new GUC,
wal_skip_threshold, guides that choice.  If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold.  Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.

Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode.  Introduce rd_firstRelfilenodeSubid.  Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node.  Make relcache.c retain entries for certain
dropped relations until end of transaction.

Back-patch to 9.5 (all supported versions).  This introduces a new WAL
record type, XLOG_GIST_ASSIGN_LSN, without bumping XLOG_PAGE_MAGIC.  As
always, update standby systems before master systems.  This changes
sizeof(RelationData) and sizeof(IndexStmt), breaking binary
compatibility for affected extensions.  (The most recent commit to
affect the same class of extensions was
089e4d405d0f3b94c74a2c6a54357a84a681754b.)

Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas.  Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem.  Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs.  Reported by Martijn van Oosterhout.

Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-03-21 09:38:34 -07:00
Noah Misch ae86e46c3b Back-patch log_newpage_range().
Back-patch a subset of commit 9155580fd5
to v11, v10, 9.6, and 9.5.  Include the latest repairs to this function.
Use a new XLOG_FPI_MULTI value instead of reusing XLOG_FPI.  That way,
if an older server reads WAL from this function, that server will PANIC
instead of applying just one page of the record.  The next commit adds a
call to this function.

Discussion: https://postgr.es/m/20200304.162919.898938381201316571.horikyota.ntt@gmail.com
2020-03-21 09:38:33 -07:00
Noah Misch 4433c6e8c0 During heap rebuild, lock any TOAST index until end of transaction.
swap_relation_files() calls toast_get_valid_index() to find and lock
this index, just before swapping with the rebuilt TOAST index.  The
latter function releases the lock before returning.  Potential for
mischief is low; a concurrent session can issue ALTER INDEX ... SET
(fillfactor = ...), which is not alarming.  Nonetheless, changing
pg_class.relfilenode without a lock is unconventional.  Back-patch to
9.5 (all supported versions), because another fix needs this.

Discussion: https://postgr.es/m/20191226001521.GA1772687@rfd.leadboat.com
2020-03-21 09:38:30 -07:00
Noah Misch 03e99a6749 Fix cosmetic blemishes involving rd_createSubid.
Remove an obsolete comment from AtEOXact_cleanup().  Restore formatting
of a comment in struct RelationData, mangled by the pgindent run in
commit 9af4159fce.  Back-patch to 9.5 (all
supported versions), because another fix stacks on this.
2020-03-21 09:38:30 -07:00
Bruce Momjian 0c96744a87 docs: use alias in WHERE clause of full text search example
The current doc query specified an alias in the FROM clause and used in
it the target list, but not in the WHERE clause.

Reported-by: axykon@gmail.com

Discussion: https://postgr.es/m/158316348159.30450.16075357948244298217@wrigleys.postgresql.org

Backpatch-through: 9.5
2020-03-20 20:19:32 -04:00
Andrew Dunstan e8a6391fc1 Turn off deprecated bison warnings under MSVC
These are disabled by the configure code, so this is just fixing an
inconsistency in the MSVC code.

Backpatch to all live branches.
2020-03-20 14:02:15 -04:00
Bruce Momjian 6802942fa2 pg_upgrade: make get_major_server_version() err msg consistent
This patch fixes the error message in get_major_server_version() to be
"could not parse version file", and uses the full file path name, rather
than just the data directory path.

Also, commit 4109bb5de4 added the cause of the failure to the  "could
not open" error message, and improved quoting.  This patch backpatches
the "could not open" cause to PG 12, where it was first widely used, and
backpatches the quoting fix in that patch to all supported releases.

Reported-by: Tom Lane

Discussion: https://postgr.es/m/87pne2w98h.fsf@wibble.ilmari.org

Author: Dagfinn Ilmari Mannsåker

Backpatch-through: 9.5
2020-03-19 15:20:55 -04:00
Fujii Masao 375aa3fc2c Correct the descriptions of recovery-related wait events in docs.
This commit corrects the descriptions of RecoveryWalAll and RecoveryWalStream
wait events in the documentation.

Back-patch to v10 where those wait events were added.

Author: Fujii Masao
Reviewed-by: Kyotaro Horiguchi, Atsushi Torikoshi
Discussion: https://postgr.es/m/124997ee-096a-5d09-d8da-2c7a57d0816e@oss.nttdata.com
2020-03-18 23:08:24 +09:00
Amit Kapila 18c41cb6b7 Add missing errcode() in a few ereport calls.
This will allow to specifying SQLSTATE error code for the errors in the
missing places.

Reported-by: Sawada Masahiko
Author: Sawada Masahiko
Backpatch-through: 9.5
Discussion: https://postgr.es/m/CA+fd4k6N8EjNvZpM8nme+y+05mz-SM8Z_BgkixzkA34R+ej0Kw@mail.gmail.com
2020-03-18 09:37:08 +05:30
Tom Lane 974c8ea0af Doc: clarify behavior of "anyrange" pseudo-type.
I noticed that we completely failed to document the restriction
that an "anyrange" result type has to be inferred from an "anyrange"
input.  The docs also were less clear than they could be about the
relationship between "anyrange" and "anyarray".

It's been like this all along, so back-patch.
2020-03-17 15:05:17 -04:00
Tom Lane 9af2f7dc96 Use pkg-config, if available, to locate libxml2 during configure.
If pkg-config is installed and knows about libxml2, use its information
rather than asking xml2-config.  Otherwise proceed as before.  This
patch allows "configure --with-libxml" to succeed on platforms that
have pkg-config but not xml2-config, which is likely to soon become
a typical situation.

The old mechanism can be forced by setting XML2_CONFIG explicitly
(hence, build processes that were already doing so will certainly
not need adjustment).  Also, it's now possible to set XML2_CFLAGS
and XML2_LIBS explicitly to override both programs.

There is a small risk of this breaking existing build processes,
if there are multiple libxml2 installations on the machine and
pkg-config disagrees with xml2-config about which to use.  The
only case where that seems really likely is if a builder has tried
to select a non-default xml2-config by putting it early in his PATH
rather than setting XML2_CONFIG.  Plan to warn against that in the
minor release notes.

Back-patch to v10; before that we had no pkg-config infrastructure,
and it doesn't seem worth adding it for this.

Hugh McMaster and Tom Lane; Peter Eisentraut also made an earlier
attempt at this, from which I lifted most of the docs changes.

Discussion: https://postgr.es/m/CAN9BcdvfUwc9Yx5015bLH2TOiQ-M+t_NADBSPhMF7dZ=pLa_iw@mail.gmail.com
2020-03-17 12:09:27 -04:00
Tom Lane 819d552574 Avoid holding a directory FD open across assorted SRF calls.
This extends the fixes made in commit 085b6b667 to other SRFs with the
same bug, namely pg_logdir_ls(), pgrowlocks(), pg_timezone_names(),
pg_ls_dir(), and pg_tablespace_databases().

Also adjust various comments and documentation to warn against
expecting to clean up resources during a ValuePerCall SRF's final
call.

Back-patch to all supported branches, since these functions were
all born broken.

Justin Pryzby, with cosmetic tweaks by me

Discussion: https://postgr.es/m/20200308173103.GC1357@telsasoft.com
2020-03-16 21:05:54 -04:00
Alvaro Herrera a6b5ddf64c
Document pg_ls_*dir hiding of directories and special files
It's strange that a directory-listing function does not list all entries
in a directory, so let's at least document it.  This involves

pg_ls_logdir
pg_ls_waldir
pg_ls_archive_statusdir
pg_ls_tmpdir

Backpatch as far back as it applies cleanly (and as far as as each
function exists).  REL_10_STABLE uses different wording, but hopefully
people are not reading docs so old to write new apps anyway.

Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20200305161838.GJ684@telsasoft.com
2020-03-16 19:12:14 -03:00
Alvaro Herrera 55978e28cd
Plug memory leak
Introduced by b08dee24a5.  Noted by Coverity.
2020-03-16 16:27:13 -03:00
Tom Lane 540ef37cc0 Restructure polymorphic-type resolution in funcapi.c.
resolve_polymorphic_tupdesc() and resolve_polymorphic_argtypes() failed to
cover the case of having to resolve anyarray given only an anyrange input.
The bug was masked if anyelement was also used (as either input or
output), which probably helps account for our not having noticed.

While looking at this I noticed that resolve_generic_type() would produce
the wrong answer if asked to make that same resolution.  ISTM that
resolve_generic_type() is confusingly defined and overly complex, so
rather than fix it, let's just make funcapi.c do the specific lookups
it requires for itself.

With this change, resolve_generic_type() is not used anywhere, so remove
it in HEAD.  In the back branches, leave it alone (complete with bug)
just in case any external code is using it.

While we're here, make some other refactoring adjustments in funcapi.c
with an eye to upcoming future expansion of the set of polymorphic types:

* Simplify quick-exit tests by adding an overall have_polymorphic_result
flag.  This is about a wash now but will be a win when there are more
flags.

* Reduce duplication of code between resolve_polymorphic_tupdesc() and
resolve_polymorphic_argtypes().

* Don't bother to validate correct matching of anynonarray or anyenum;
the parser should have done that, and even if it didn't, just doing
"return false" here would lead to a very confusing, off-point error
message.  (Really, "return false" in these two functions should only
occur if the call_expr isn't supplied or we can't obtain data type
info from it.)

* For the same reason, throw an elog rather than "return false" if
we fail to resolve a polymorphic type.

The bug's been there since we added anyrange, so back-patch to
all supported branches.

Discussion: https://postgr.es/m/6093.1584202130@sss.pgh.pa.us
2020-03-14 14:42:22 -04:00
Tom Lane 7e32aaab58 Doc: fix mistaken reference to "PG_ARGNULL_xxx()" macro.
This should of course be just "PG_ARGISNULL()".

Also reorder a couple of paras to make the discussion of PG_ARGISNULL
less disjointed.

Back-patch to v10 where the error was introduced.

Laurenz Albe and Tom Lane, per an anonymous docs comment

Discussion: https://postgr.es/m/158399487096.5708.10696365251766477013@wrigleys.postgresql.org
2020-03-13 12:49:10 -04:00
Peter Eisentraut 15cbbf81a5 Preserve replica identity index across ALTER TABLE rewrite
If an index was explicitly set as replica identity index, this setting
was lost when a table was rewritten by ALTER TABLE.  Because this
setting is part of pg_index but actually controlled by ALTER
TABLE (not part of CREATE INDEX, say), we have to do some extra work
to restore it.

Based-on-patch-by: Quan Zongliang <quanzongliang@gmail.com>
Reviewed-by: Euler Taveira <euler.taveira@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/c70fcab2-4866-0d9f-1d01-e75e189db342@gmail.com
2020-03-13 12:24:44 +01:00
Thomas Munro a493f93342 Fix nextXid tracking bug on standbys (9.5-11 only).
RecordKnownAssignedTransactionIds() should never move
nextXid backwards.  Before this commit, that could happen
if some other code path had advanced it without advancing
latestObservedXid.

One consequence is that a well timed XLOG_CHECKPOINT_ONLINE
could cause hot standby feedback messages to get confused
and report an xmin from a future epoch, potentially allowing
vacuum to run too soon on the primary.

Repair, by making sure RecordKnownAssignedTransactionIds()
can only move nextXid forwards.

In release 12 and master, this was already done by commit
2fc7af5e, which consolidated similar code and straightened
out this bug.  Back-patch to supported releases before that.

Author: Eka Palamadai <ekanatha@amazon.com>
Discussion: https://postgr.es/m/98BB4805-D0A2-48E1-96F4-15014313EADC@amazon.com
2020-03-12 18:06:54 +13:00
Tom Lane 13f353b0e0 Fix test case instability introduced in 085b6b667.
I forgot that the WAL directory might hold other files besides WAL
segments, notably including new segments still being filled.
That means a blind test for the first file's size being 16MB can
fail.  Restrict based on file name length to make it more robust.

Per buildfarm.
2020-03-11 18:24:14 -04:00
Peter Geoghegan c788115b5e Paper over bt_metap() oldest_xact bug in backbranches.
The data types that contrib/pageinspect's bt_metap() function were
declared to return as OUT arguments were wrong in some cases.  In
particular, the oldest_xact column (a TransactionId/xid field) was
declared integer/int4 within the pageinspect extension's sql file.  This
led to errors when an oldest_xact value that exceeded 2^31-1 was
encountered.

We cannot fix the declaration on Postgres 11 or 12.  All we can do is
ameliorate the problem.  Use "%d" instead of "%u" to format the output
of the oldest_xact value.  This makes the C code match the declaration,
suppressing unhelpful error messages that might otherwise make
bt_metap() totally unusable.  A bogus negative oldest_xact value will be
displayed instead of raising an error.

This commit addresses the same issue as master branch commit 691e8b2e18,
which actually fixed the problem.  Backpatch to the 11 and 12 branches
only, since they are the only branches (other than master) that have
oldest_xact.  All of the other problematic columns already display bogus
output for out of range values.

Reported-By: Victor Yegorov
Bug: #16285
Discussion: https://postgr.es/m/20200309223557.aip5n6ewln4ixbbi@alap3.anarazel.de
Backpatch: 11 and 12 only
2020-03-11 14:15:00 -07:00
Alvaro Herrera e70187c9b4
Add pg_dump support for ALTER obj DEPENDS ON EXTENSION
pg_dump is oblivious to this kind of dependency, so they're lost on
dump/restores (and pg_upgrade).  Have pg_dump emit ALTER lines so that
they're preserved.  Add some pg_dump tests for the whole thing, also.

Reviewed-by: Tom Lane (offlist)
Reviewed-by: Ibrar Ahmed
Reviewed-by: Ahsan Hadi (who also reviewed commit 899a04f5ed)
Discussion: https://postgr.es/m/20200217225333.GA30974@alvherre.pgsql
2020-03-11 16:54:54 -03:00
Tom Lane 63b51dfec1 Avoid holding a directory FD open across pg_ls_dir_files() calls.
This coding technique is undesirable because (a) it leaks the FD for
the rest of the transaction if the SRF is not run to completion, and
(b) allocated FDs are a scarce resource, but multiple interleaved
uses of the relevant functions could eat many such FDs.

In v11 and later, a query such as "SELECT pg_ls_waldir() LIMIT 1"
yields a warning about the leaked FD, and the only reason there's
no warning in earlier branches is that fd.c didn't whine about such
leaks before commit 9cb7db3f0.  Even disregarding the warning, it
wouldn't be too hard to run a backend out of FDs with careless use
of these SQL functions.

Hence, rewrite the function so that it reads the directory within
a single call, returning the results as a tuplestore rather than
via value-per-call mode.

There are half a dozen other built-in SRFs with similar problems,
but let's fix this one to start with, just to see if the buildfarm
finds anything wrong with the code.

In passing, fix bogus error report for stat() failure: it was
whining about the directory when it should be fingering the
individual file.  Doubtless a copy-and-paste error.

Back-patch to v10 where this function was added.

Justin Pryzby, with cosmetic tweaks and test cases by me

Discussion: https://postgr.es/m/20200308173103.GC1357@telsasoft.com
2020-03-11 15:27:59 -04:00
Alvaro Herrera 7c094a11c7
Avoid duplicates in ALTER ... DEPENDS ON EXTENSION
If the command is attempted for an extension that the object already
depends on, silently do nothing.

In particular, this means that if a database containing multiple such
entries is dumped, the restore will silently do the right thing and
record just the first one.  (At least, in a world where pg_dump does
dump such entries -- which it doesn't currently, but it will.)

Backpatch to 9.6, where this kind of dependency was introduced.

Reviewed-by: Ibrar Ahmed, Tom Lane (offlist)
Discussion: https://postgr.es/m/20200217225333.GA30974@alvherre.pgsql
2020-03-11 11:04:59 -03:00
Tom Lane f5d49f2265 Fix pg_dump/pg_restore to restore event triggers later.
Previously, event triggers were restored just after regular triggers
(and FK constraints, which are basically triggers).  This is risky
since an event trigger, once installed, could interfere with subsequent
restore commands.  Worse, because event triggers don't have any
particular dependencies on any post-data objects, a parallel restore
would consider them eligible to be restored the moment the post-data
phase starts, allowing them to also interfere with restoration of a
whole bunch of objects that would have been restored before them in
a serial restore.  There's no way to completely remove the risk of a
misguided event trigger breaking the restore, since if nothing else
it could break other event triggers.  But we can certainly push them
to later in the process to minimize the hazard.

To fix, tweak the RestorePass mechanism introduced by commit 3eb9a5e7c
so that event triggers are handled as part of the post-ACL processing
pass (renaming the "REFRESH" pass to "POST_ACL" to reflect its more
general use).  This will cause them to restore after everything except
matview refreshes, which seems OK since matview refreshes really ought
to run in the post-restore state of the database.  In a parallel
restore, event triggers and matview refreshes might be intermixed,
but that seems all right as well.

Also update the code and comments in pg_dump_sort.c so that its idea
of how things are sorted agrees with what actually happens due to
the RestorePass mechanism.  This is mostly cosmetic: it'll affect the
order of objects in a dump's TOC, but not the actual restore order.
But not changing that would be quite confusing to somebody reading
the code.

Back-patch to all supported branches.

Fabrízio de Royes Mello, tweaked a bit by me

Discussion: https://postgr.es/m/CAFcNs+ow1hmFox8P--3GSdtwz-S3Binb6ZmoP6Vk+Xg=K6eZNA@mail.gmail.com
2020-03-09 14:58:11 -04:00
Fujii Masao e54183cb0b Fix bug that causes to report waiting in PS display twice, in hot standby.
Previously "waiting" could appear twice via PS in case of lock conflict
in hot standby mode. Specifically this issue happend when the delay
in WAL application determined by max_standby_archive_delay and
max_standby_streaming_delay had passed but it took more than 500 msec
to cancel all the conflicting transactions. Especially we can observe this
easily by setting those delay parameters to -1.

The cause of this issue was that WaitOnLock() and
ResolveRecoveryConflictWithVirtualXIDs() added "waiting" to
the process title in that case. This commit prevents
ResolveRecoveryConflictWithVirtualXIDs() from reporting waiting
in case of lock conflict, to fix the bug.

Back-patch to all back branches.

Author: Masahiko Sawada
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/CA+fd4k4mXWTwfQLS3RPwGr4xnfAEs1ysFfgYHvmmoUgv6Zxvmg@mail.gmail.com
2020-03-10 00:16:48 +09:00
Fujii Masao 500fe161ed Avoid assertion failure with targeted recovery in standby mode.
At the end of recovery, standby mode is turned off to re-fetch the last
valid record from archive or pg_wal. Previously, if recovery target was
reached and standby mode was turned off while the current WAL source
was stream, recovery could try to retrieve WAL file containing the last
valid record unexpectedly from stream even though not in standby mode.
This caused an assertion failure. That is, the assertion test confirms that
WAL file should not be retrieved from stream if standby mode is not true.

This commit moves back the current WAL source to archive if it's stream
even though not in standby mode, to avoid that assertion failure.

This issue doesn't cause the server to crash when built with assertion
disabled. In this case, the attempt to retrieve WAL file from stream not
in standby mode just fails. And then recovery tries to retrieve WAL file
from archive or pg_wal.

Back-patch to all supported branches.

Author: Kyotaro Horiguchi
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/20200227.124830.2197604521555566121.horikyota.ntt@gmail.com
2020-03-09 15:34:38 +09:00
Peter Eisentraut 53a039da62 Fix typo 2020-03-07 08:53:21 +01:00
Michael Paquier 75839fcf82 Fix assertion failure with ALTER TABLE ATTACH PARTITION and indexes
Using ALTER TABLE ATTACH PARTITION causes an assertion failure when
attempting to work on a partitioned index, because partitioned indexes
cannot have partition bounds.

The grammar of ALTER TABLE ATTACH PARTITION requires partition bounds,
but not ALTER INDEX, so mixing ALTER TABLE with partitioned indexes is
confusing.  Hence, on HEAD, prevent ALTER TABLE to attach a partition if
the relation involved is a partitioned index.  On back-branches, as
applications may rely on the existing behavior, just remove the
culprit assertion.

Reported-by: Alexander Lakhin
Author: Amit Langote, Michael Paquier
Discussion: https://postgr.es/m/16276-5cd1dcc8fb8be7b5@postgresql.org
Backpatch-through: 11
2020-03-03 13:56:34 +09:00
Fujii Masao 53fc42c6d1 Fix the name of the first WAL segment file, in docs.
Previously the documentation explains that WAL segment files
start at 000000010000000000000000. But the first WAL segment file
that initdb creates is 000000010000000000000001 not
000000010000000000000000. This change was caused by old
commit 8c843fff2d, but the documentation had not been updated
a long time.

Back-patch to all supported branches.

Author: Fujii Masao
Reviewed-by: David Zhang
Discussion: https://postgr.es/m/CAHGQGwHOmGe2OqGOmp8cOfNVDivq7dbV74L5nUGr+3eVd2CU2Q@mail.gmail.com
2020-03-03 12:23:45 +09:00
Tom Lane 9eb6b652a4 Correctly re-use hash tables in buildSubPlanHash().
Commit 356687bd8 omitted to remove leftover code for destroying
a hashed subplan's hash tables, with the result that the tables
were always rebuilt not reused; this leads to severe memory
leakage if a hashed subplan is re-executed enough times.
Moreover, the code for reusing the hashnulls table had a typo
that would have made it do the wrong thing if it were reached.

Looking at the code coverage report shows severe under-coverage
of the potential callers of ResetTupleHashTable, so add some test
cases that exercise them.

Andreas Karlsson and Tom Lane, per reports from Ranier Vilela
and Justin Pryzby.

Backpatch to v11, as the faulty commit was.

Discussion: https://postgr.es/m/edb62547-c453-c35b-3ed6-a069e4d6b937@proxel.se
Discussion: https://postgr.es/m/CAEudQAo=DCebm1RXtig9OH+QivpS97sMkikt0A9qHmMUs+g6ZA@mail.gmail.com
Discussion: https://postgr.es/m/20200210032547.GA1412@telsasoft.com
2020-02-29 13:48:10 -05:00
Tom Lane 7ea20a2bc6 Avoid failure if autovacuum tries to access a just-dropped temp namespace.
Such an access became possible when commit 246a6c8f7 added more
aggressive cleanup of orphaned temp relations by autovacuum.
Since autovacuum's snapshot might be slightly stale, it could
attempt to access an already-dropped temp namespace, resulting in
an assertion failure or null-pointer dereference.  (In practice,
since we don't drop temp namespaces automatically but merely
recycle them, this situation could only arise if a superuser does
a manual drop of a temp namespace.  Still, that should be allowed.)

The core of the bug, IMO, is that isTempNamespaceInUse and its callers
failed to think hard about whether to treat "temp namespace isn't there"
differently from "temp namespace isn't in use".  In hopes of forestalling
future mistakes of the same ilk, replace that function with a new one
checkTempNamespaceStatus, which makes the same tests but returns a
three-way enum rather than just a bool.  isTempNamespaceInUse is gone
entirely in HEAD; but just in case some external code is relying on it,
keep it in the back branches, as a bug-compatible wrapper around the
new function.

Per report originally from Prabhat Kumar Sahu, investigated by Mahendra
Singh and Michael Paquier; the final form of the patch is my fault.
This replaces the failed fix attempt in a052f6cbb.

Backpatch as far as v11, as 246a6c8f7 was.

Discussion: https://postgr.es/m/CAKYtNAr9Zq=1-ww4etHo-VCC-k120YxZy5OS01VkaLPaDbv2tg@mail.gmail.com
2020-02-28 20:28:34 -05:00
Tom Lane 880643b9cd Doc: correct thinko in pg_buffercache documentation.
Access to this module is granted to the pg_monitor role, not
pg_read_all_stats.  (Given the view's performance impact,
it seems wise to be restrictive, so I think this was the
correct decision --- and anyway it was clearly intentional.)

Per bug #16279 from Philip Semanchuk.

Discussion: https://postgr.es/m/16279-fcaac33c68aab0ab@postgresql.org
2020-02-28 11:29:59 -05:00
Michael Paquier 8e076f4f9a Remove TAP test for createdb --lc-ctype
OpenBSD falls back to "C" when using an incorrect input with setlocale()
and LC_CTYPE, causing this test, introduced by 008cf04, to fail.  This
removes the culprit test to avoid the portability issue.

Per report from Robert Haas, via buildfarm member curculio.

Discussion: https://postgr.es/m/CA+TgmoZ6ddh3mHD9gU8DvNYoFmuJaYYn1+4AvZNp25vTdRwCAQ@mail.gmail.com
Backpatch-through: 11
2020-02-27 21:58:50 +09:00