Commit Graph

53411 Commits

Author SHA1 Message Date
Peter Eisentraut 7844c9918a psql: Show all query results by default
Previously, psql printed only the last result if a command string
returned multiple result sets.  Now it prints all of them.  The
previous behavior can be obtained by setting the psql variable
SHOW_ALL_RESULTS to off.

This is a significantly enhanced version of
3a51306722 (that was later reverted).
There is also much more test coverage for various psql features now.

Author: Fabien COELHO <coelho@cri.ensmp.fr>
Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: "Iwata, Aya" <iwata.aya@jp.fujitsu.com> (earlier version)
Reviewed-by: Daniel Verite <daniel@manitou-mail.org> (earlier version)
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> (earlier version)
Reviewed-by: vignesh C <vignesh21@gmail.com> (earlier version)
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.1904132231510.8961@lancre
2022-04-04 20:00:33 +02:00
Tom Lane cbf4177f2c Disable synchronize_seqscans in 027_stream_regress.pl.
This script runs the core regression tests with quite a small value of
shared_buffers, making it prone to breakage due to synchronize_seqscans
kicking in where the tests don't expect that.  Disable that feature to
stabilize the tests.

Discussion: https://postgr.es/m/1258185.1648876239@sss.pgh.pa.us
2022-04-04 12:38:51 -04:00
Andrew Dunstan 4eb9798879 Avoid freeing objects during json aggregate finalization
Commit f4fb45d15c tried to free memory during aggregate finalization.
This cause issues, particularly when used as a window function, so stop
doing that.

Per complaint by Jaime Casanova and diagnosis by Andres Freund

Discussion: https://postgr.es/m/YkfeMNYRCGhySKyg@ahch-to
2022-04-04 11:03:49 -04:00
Robert Haas afb529e677 pg_basebackup: Fix code that thinks about LZ4 buffer size.
Before this patch, there was some code that tried to make sure that the
buffer was always big enough at the start, and then asserted that it
didn't need to be enlarged later. However, the code to make sure it was
big enough at the start doesn't actually work, and therefore it was
possible to fail an assertion and crash later.

Remove the code that tries to make sure the buffer is always big enough
at the start in favor of enlarging the buffer as we go along whenever
that is necessary.

The mistake probably happened because, on the server side, we do
actually need to guarantee that the buffer is big enough at the start
to avoid subsequent resizings. However, in that case, the calling
code makes promises about how much data it will provide at once, but
here, that's not the case.

Report by Justin Pryzby. Analysis by me. Patch by Dipesh Pandit.

Discussion: http://postgr.es/m/20220330143536.GG28503@telsasoft.com
2022-04-04 10:36:23 -04:00
David Rowley 40af10b571 Use Generation memory contexts to store tuples in sorts
The general usage pattern when we store tuples in tuplesort.c is that
we store a series of tuples one by one then either perform a sort or spill
them to disk.  In the common case, there is no pfreeing of already stored
tuples.  For the common case since we do not individually pfree tuples, we
have very little need for aset.c memory allocation behavior which
maintains freelists and always rounds allocation sizes up to the next
power of 2 size.

Here we conditionally use generation.c contexts for storing tuples in
tuplesort.c when the sort will never be bounded.  Unfortunately, the
memory context to store tuples is already created by the time any calls
would be made to tuplesort_set_bound(), so here we add a new sort option
that allows callers to specify if they're going to need a bounded sort or
not.  We'll use a standard aset.c allocator when this sort option is not
set.

Extension authors must ensure that the TUPLESORT_ALLOWBOUNDED flag is
used when calling tuplesort_begin_* for any sorts that make a call to
tuplesort_set_bound().

Author: David Rowley
Reviewed-by: Andy Fan
Discussion: https://postgr.es/m/CAApHDvoH4ASzsAOyHcxkuY01Qf++8JJ0paw+03dk+W25tQEcNQ@mail.gmail.com
2022-04-04 22:52:35 +12:00
David Rowley 77bae396df Adjust tuplesort API to have bitwise option flags
This replaces the bool flag for randomAccess.  An upcoming patch requires
adding another option, so instead of breaking the API for that, then
breaking it again one day if we add more options, let's just break it
once.  Any boolean options we add in the future will just make use of an
unused bit in the flags.

Any extensions making use of tuplesorts will need to update their code
to pass TUPLESORT_RANDOMACCESS instead of true for randomAccess.
TUPLESORT_NONE can be used for a set of empty options.

Author: David Rowley
Reviewed-by: Justin Pryzby
Discussion: https://postgr.es/m/CAApHDvoH4ASzsAOyHcxkuY01Qf%2B%2B8JJ0paw%2B03dk%2BW25tQEcNQ%40mail.gmail.com
2022-04-04 22:24:59 +12:00
David Rowley 1b0d9aa4f7 Improve the generation memory allocator
Here we make a series of improvements to the generation memory
allocator, namely:

1. Allow generation contexts to have a minimum, initial and maximum block
sizes. The standard allocator allows this already but when the generation
context was added, it only allowed fixed-sized blocks.  The problem with
fixed-sized blocks is that it's difficult to choose how large to make the
blocks.  If the chosen size is too small then we'd end up with a large
number of blocks and a large number of malloc calls. If the block size is
made too large, then memory is wasted.

2. Add support for "keeper" blocks.  This is a special block that is
allocated along with the context itself but is never freed.  Instead,
when the last chunk in the keeper block is freed, we simply mark the block
as empty to allow new allocations to make use of it.

3. Add facility to "recycle" newly empty blocks instead of freeing them
and having to later malloc an entire new block again.  We do this by
recording a single GenerationBlock which has become empty of any chunks.
When we run out of space in the current block, we check to see if there is
a "freeblock" and use that if it contains enough space for the allocation.

Author: David Rowley, Tomas Vondra
Reviewed-by: Andy Fan
Discussion: https://postgr.es/m/d987fd54-01f8-0f73-af6c-519f799a0ab8@enterprisedb.com
2022-04-04 20:53:13 +12:00
Thomas Munro cc58eecc5d Fix tuplesort optimization for CLUSTER-on-expression.
When dispatching sort operations to specialized variants, commit
69749243 failed to handle the case where CLUSTER-sort decides not to
initialize datum1 and isnull1.  Fix by hoisting that decision up a level
and advertising whether datum1 can be relied on, in the Tuplesortstate
object.

Per reports from UBsan and Valgrind build farm animals, while running
the cluster.sql test.

Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CAFBsxsF1TeK5Fic0M%2BTSJXzbKsY6aBqJGNj6ptURuB09ZF6k_w%40mail.gmail.com
2022-04-04 10:52:02 +12:00
Tom Lane 591e088dd5 Fix portability issues in datetime parsing.
datetime.c's parsing logic has assumed that strtod() will accept
a string that looks like ".", which it does in glibc, but not on
some less-common platforms such as AIX.  The result of this was
that datetime fields like "123." would be accepted on some platforms
but not others; which is a sufficiently odd case that it's not that
surprising we've heard no field complaints.  But commit e39f99046
extended that assumption to new places, and happened to add a test
case that exposed the platform dependency.  Remove this dependency
by special-casing situations without any digits after the decimal
point.

(Again, this is in part a pre-existing bug but I don't feel a
compulsion to back-patch.)

Also, rearrange e39f99046's changes in formatting.c to avoid a
Coverity complaint that we were copying an uninitialized field.

Discussion: https://postgr.es/m/1592893.1648969747@sss.pgh.pa.us
2022-04-03 17:04:33 -04:00
Peter Geoghegan f3c15cbe50 Generalize how VACUUM skips all-frozen pages.
Non-aggressive VACUUMs were at a gratuitous disadvantage (relative to
aggressive VACUUMs) around advancing relfrozenxid and relminmxid before
now.  The issue only came up when concurrent activity unset some heap
page's visibility map bit right as VACUUM was considering if the page
should get counted in frozenskipped_pages.  The non-aggressive case
would recheck the all-frozen bit at this point.  The aggressive case
reasoned that the page (a skippable page) must have at least been
all-frozen in the recent past, so skipping it won't make relfrozenxid
advancement unsafe (which is never okay for aggressive VACUUMs).

The recheck created a window for some other backend to confuse matters
for VACUUM.  If the page's VM bit turned out to be unset, VACUUM would
conclude that the page was _never_ all-frozen.  frozenskipped_pages was
not incremented, and yet VACUUM couldn't back out of skipping at this
late stage (it couldn't choose to scan the page instead).  This made it
unsafe to advance relfrozenxid later on.

Consistently avoid the issue by generalizing how we skip frozen pages
during aggressive VACUUMs: take the same approach when skipping any
skippable page range during aggressive and non-aggressive VACUUMs alike.
The new approach makes ranges (not individual pages) the fundamental
unit of skipping using the visibility map.  frozenskipped_pages is
replaced with a boolean flag that represents whether some skippable
range with one or more all-visible pages was actually skipped.

It is safe for VACUUM to treat a page as all-frozen provided it at least
had its all-frozen bit set after the OldestXmin cutoff was established.
VACUUM is only required to scan pages that might have XIDs < OldestXmin
(unfrozen XIDs) to be able to safely advance relfrozenxid.  Tuples
concurrently inserted on "skipped" pages can be thought of as equivalent
to tuples concurrently inserted on a block >= rel_pages.

It's possible that the issue this commit fixes hardly ever came up in
practice.  But we only had to be unlucky once to lose out on advancing
relfrozenxid -- a single affected heap page was enough to throw VACUUM
off.  That seems like something to avoid on general principle.  This is
similar to an issue fixed by commit 44fa8488, which taught vacuumlazy.c
to not give up on non-aggressive relfrozenxid advancement just because a
cleanup lock wasn't immediately available on some heap page.

Skipping an all-visible range is now explicitly structured as a choice
made by non-aggressive VACUUMs, by weighing known costs (scanning extra
skippable pages to freeze their tuples early) against known benefits
(advancing relfrozenxid early).  This works in essentially the same way
as it always has (don't skip ranges < SKIP_PAGES_THRESHOLD).  We could
do much better here in the future by considering other relevant factors.

Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Robert Haas <robertmhaas@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wzn6bGJGfOy3zSTJicKLw99PHJeSOQBOViKjSCinaxUKDQ@mail.gmail.com
Discussion: https://postgr.es/m/CA%2BTgmoZiSOY6H7aadw5ZZGm7zYmfDzL6nwmL5V7GL4HgJgLF_w%40mail.gmail.com
2022-04-03 13:35:43 -07:00
Peter Geoghegan 0b018fabaa Set relfrozenxid to oldest extant XID seen by VACUUM.
When VACUUM set relfrozenxid before now, it set it to whatever value was
used to determine which tuples to freeze -- the FreezeLimit cutoff.
This approach was very naive.  The relfrozenxid invariant only requires
that new relfrozenxid values be <= the oldest extant XID remaining in
the table (at the point that the VACUUM operation ends), which in
general might be much more recent than FreezeLimit.

VACUUM now carefully tracks the oldest remaining XID/MultiXactId as it
goes (the oldest remaining values _after_ lazy_scan_prune processing).
The final values are set as the table's new relfrozenxid and new
relminmxid in pg_class at the end of each VACUUM.  The oldest XID might
come from a tuple's xmin, xmax, or xvac fields.  It might even come from
one of the table's remaining MultiXacts.

Final relfrozenxid values must still be >= FreezeLimit in an aggressive
VACUUM (FreezeLimit still acts as a lower bound on the final value that
aggressive VACUUM can set relfrozenxid to).  Since standard VACUUMs
still make no guarantees about advancing relfrozenxid, they might as
well set relfrozenxid to a value from well before FreezeLimit when the
opportunity presents itself.  In general standard VACUUMs may now set
relfrozenxid to any value > the original relfrozenxid and <= OldestXmin.

Credit for the general idea of using the oldest extant XID to set
pg_class.relfrozenxid at the end of VACUUM goes to Andres Freund.

Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Robert Haas <robertmhaas@gmail.com>
Discussion: https://postgr.es/m/CAH2-WzkymFbz6D_vL+jmqSn_5q1wsFvFrE+37yLgL_Rkfd6Gzg@mail.gmail.com
2022-04-03 09:57:21 -07:00
Peter Geoghegan 05023a237c Doc: Add relfrozenxid Tip to XID wraparound section.
VACUUM VERBOSE and autovacuum log reports were taught to report the
details of how VACUUM advanced relfrozenxid (and relminmxid) by commit
872770fd.  Highlight this by adding a "Tip" to the documentation, next
to related discussion of age(relfrozenxid) monitoring.

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-Wzk0C1O-MKkOrj4YAfsGRru2=cA2VQpqM-9R1HNuG3nFaQ@mail.gmail.com
2022-04-02 13:34:57 -07:00
Tom Lane e39f990467 Fix overflow hazards in interval input and output conversions.
DecodeInterval (interval input) was careless about integer-overflow
hazards, allowing bogus results to be obtained for sufficiently
large input values.  Also, since it initially converted the input
to a "struct tm", it was impossible to produce the full range of
representable interval values.

Meanwhile, EncodeInterval (interval output) and a few other
functions could suffer failures if asked to process sufficiently
large interval values, because they also relied on being able to
represent an interval in "struct tm" which is not designed to
handle that.

Fix all this stuff by introducing new struct types that are more
fit for purpose.

While this is clearly a bug fix, it's also an API break for any
code that's calling these functions directly.  So back-patching
doesn't seem wise, especially in view of the lack of field
complaints.

Joe Koshakow, editorialized a bit by me

Discussion: https://postgr.es/m/CAAvxfHff0JLYHwyBrtMx_=6wr=k2Xp+D+-X3vEhHjJYMj+mQcg@mail.gmail.com
2022-04-02 16:12:29 -04:00
Peter Geoghegan f7e4d5c64f Doc: Clarify the role of aggressive VACUUMs.
Adjust the documentation's coverage of aggressive VACUUMs to make it
clearer that aggressive vacuuming isn't always strictly necessary.  It's
possible for non-aggressive VACUUMs to advance relfrozenxid/relminmxid
without fail, given the right workload conditions.

While this has always been true, it matters more with recent and pending
improvements to VACUUM.  These improvements make non-aggressive
vacuuming more likely to advance relfrozenxid/relminmxid in practice.
While this is an unrelated improvement to the docs, formally speaking,
it still doesn't seem worth backpatching.  So don't backpatch.
2022-04-02 12:29:35 -07:00
Tom Lane 1b208ebaf1 Add a couple more tests for interval input decoding.
Cover some cases that would have been broken by a proposed patch,
but we failed to notice for lack of test coverage.  I'm pushing
this separately mainly to memorialize that it *is* our historical
behavior.

Discussion: https://postgr.es/m/1344498.1648920056@sss.pgh.pa.us
2022-04-02 13:50:05 -04:00
Peter Geoghegan 14bf1e8313 vacuumlazy.c: Clean up variable declarations.
Move some of the heap_vacuum_rel() instrumentation related variables to
the scope where they're actually needed.  Also reorder some of the
variable declarations at the start of heap_vacuum_rel() so that related
variables appear together.
2022-04-02 10:33:21 -07:00
Joe Conway 9752436f04 Use has_privs_for_roles for predefined role checks: round 2
Similar to commit 6198420ad, replace is_member_of_role with
has_privs_for_role for predefined role access checks in recently
committed basebackup code. In passing fix a double-word error
in a nearby comment.

Discussion: https://postgr.es/m/flat/CAGB+Vh4Zv_TvKt2tv3QNS6tUM_F_9icmuj0zjywwcgVi4PAhFA@mail.gmail.com
2022-04-02 13:24:38 -04:00
Alvaro Herrera cfdd03f45e
Allow CLUSTER on partitioned tables
This is essentially the same as applying VACUUM FULL to a partitioned
table, which has been supported since commit 3c3bb99330 (March 2017).
While there's no great use case in applying CLUSTER to partitioned
tables, we don't have any strong reason not to allow it either.

For now, partitioned indexes cannot be marked clustered, so an index
must always be specified.

While at it, rename some variables that were RangeVars during the
development that led to 8bc717cb88 but never made it that way to the
source tree; there's no need to perpetuate names that have always been
more confusing than helpful.

Author: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/20201028003312.GU9241@telsasoft.com
Discussion: https://postgr.es/m/20200611153502.GT14879@telsasoft.com
2022-04-02 19:08:34 +02:00
Peter Geoghegan b7c485fb93 Doc: Remove MultiXact wraparound section link.
Remove circular "25.1.5.1. Multixacts And Wraparound" link that
references the section that the link itself appears in.  An explanation
of MultiXactId age appears only a few sentences before the link, so
there's no question that the link is superfluous at best.

Oversight in commit d5409295.

Author: Peter Geoghegan <pg@bowt.ie>
Backpatch: 14-
2022-04-02 09:49:08 -07:00
Alvaro Herrera 0af504733c
Remove excess semicolon in MERGE docs
Author: Euler Taveira <euler@eulerto.com>
Discussion: https://postgr.es/m/01dea9ef-85ad-4e09-a783-a1eadeae7bbe@www.fastmail.com
2022-04-02 17:18:14 +02:00
Andrew Dunstan c6dc6a0124 Use ORDER BY in catalog results in SQL/JSON tests
The buildfarm has revealed some instability in results from catalog
queries in tests from commit 1a36bc9dba. Cure this by adding ORDER BY
to such queries.
2022-04-02 10:00:10 -04:00
John Naylor 6974924347 Specialize tuplesort routines for different kinds of abbreviated keys
Previously, the specialized tuplesort routine inlined handling for
reverse-sort and NULLs-ordering but called the datum comparator via a
pointer in the SortSupport struct parameter. Testing has showed that we
can get a useful performance gain by specializing datum comparison for
the different representations of abbreviated keys -- signed and unsigned
64-bit integers and signed 32-bit integers. Almost all abbreviatable data
types will benefit -- the only exception for now is numeric, since the
datum comparison is more complex. The performance gain depends on data
type and input distribution, but often falls in the range of 10-20% faster.

Thomas Munro

Reviewed by Peter Geoghegan, review and performance testing by me

Discussion:
https://www.postgresql.org/message-id/CA%2BhUKGKKYttZZk-JMRQSVak%3DCXSJ5fiwtirFf%3Dn%3DPAbumvn1Ww%40mail.gmail.com
2022-04-02 15:22:25 +07:00
Peter Eisentraut db086de5ab Remove obsolete comment
accidentally left behind by 4cb658af70
2022-04-02 07:41:12 +02:00
Michael Paquier d2a2ce4184 Make upgradecheck a no-op in MSVC's vcregress.pl
322becb has changed upgradecheck to use the TAP tests, discarding
pg_upgrade's tests in bincheck.  However, this is proving to be a bad
idea for the Windows buildfarm clients that use MSVC when TAP tests are
disabled as this causes a hard failure at the pg_upgrade step.

This commit disables upgradecheck, moving the execution of the tests of
pg_upgrade to bincheck, as per an initial suggestion from Andres
Freund, so as the buildfarm is able to live happily with those changes.

While on it, remove the routine that was used by upgradecheck to
create databases whose names are generated with a range of ASCII
characters as it is not used since 322becb.  upgradecheck is removed
from the CI script for Windows, as bincheck takes care of that now.

Per report from buildfarm member hamerkop (MSVC 2017 without a TAP
setup).

Reviewed-by: Justin Pryzby
Discussion: https://postgr.es/m/YkbnpriYEAagZ2wH@paquier.xyz
2022-04-02 12:06:11 +09:00
Michael Paquier d43085d12e pageinspect: Use better macros to get special page area for GIN and GiST
These five code paths are the last ones that made use of
PageGetSpecialPointer() to get the special area of such pages, while
those index AMs have already macros to do this job.

Noticed while reviewing the use PageGetSpecialPointer() in the whole
tree, in relation to the recent commit d16773c.
2022-04-02 11:27:20 +09:00
Peter Eisentraut 465ab24296 libpq: Fix pkg-config without OpenSSL
Do not add OpenSSL dependencies to libpq pkg-config file if OpenSSL is
not enabled.  Oversight in beff361bc1.

Author: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/20220331163759.32665-1-fontaine.fabrice%40gmail.com
2022-04-01 17:15:24 +02:00
Tom Lane 479b69a4a5 Set minimum required version of zstd as 1.4.0.
It emerges that our new zstd code depends on features that were not
present (or at least not enabled by default) in zstd before 1.4.0.
That's already four years old, so there's little reason to try to
make our code work with something older.  Instead make configure
check that zstd is at least 1.4.0, and document this requirement.

Justin Pryzby (doc changes by me)

Discussion: https://postgr.es/m/efbd94cd1102f99fd1300e44905ee4a994ee2ef6.camel@gunduz.org
2022-04-01 11:05:52 -04:00
Peter Eisentraut c1932e5428 libpq: Allow IP address SANs in server certificates
The current implementation supports exactly one IP address in a server
certificate's Common Name, which is brittle (the strings must match
exactly).  This patch adds support for IPv4 and IPv6 addresses in a
server's Subject Alternative Names.

Per discussion on-list:

- If the client's expected host is an IP address, we allow fallback to
  the Subject Common Name if an iPAddress SAN is not present, even if
  a dNSName is present.  This matches the behavior of NSS, in
  violation of the relevant RFCs.

- We also, counter-intuitively, match IP addresses embedded in dNSName
  SANs.  From inspection this appears to have been the behavior since
  the SAN matching feature was introduced in acd08d76.

- Unlike NSS, we don't map IPv4 to IPv6 addresses, or vice-versa.

Author: Jacob Champion <pchampion@vmware.com>
Co-authored-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Co-authored-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://www.postgresql.org/message-id/flat/9f5f20974cd3a4091a788cf7f00ab663d5fcdffe.camel@vmware.com
2022-04-01 15:51:23 +02:00
Robert Haas fa25bebb82 docs: Changing column type doesn't always require an index rebuild.
James Coleman and Robert Haas, reviewed by Matthias van de Meent.

Discussion: https://postgr.es/m/CAAaqYe90Ea3RG=A7H-ONvTcx549-oQhp07BrHErwM=AyH2ximg@mail.gmail.com
2022-04-01 08:48:44 -04:00
Peter Eisentraut af9e180495 Add SSL tests for IP addresses in certificates
This tests some scenarios that already work.  A subsequent patch will
introduce more functionality.

Author: Jacob Champion <pchampion@vmware.com>
Co-authored-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Co-authored-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://www.postgresql.org/message-id/flat/9f5f20974cd3a4091a788cf7f00ab663d5fcdffe.camel@vmware.com
2022-04-01 14:08:43 +02:00
Peter Eisentraut 5519d5affd psql: Refactor ProcessResult()
Separate HandleCopyResult() from ProcessResult() in preparation for a
subsequent patch.

Author: Fabien COELHO <coelho@cri.ensmp.fr>
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.1904132231510.8961@lancre
2022-04-01 13:03:33 +02:00
Michael Paquier d16773cdc8 Add macros in hash and btree AMs to get the special area of their pages
This makes the code more consistent with SpGiST, GiST and GIN, that
already use this style, and the idea is to make easier the introduction
of more sanity checks for each of these AM-specific macros.  BRIN uses a
different set of macros to get a page's type and flags, so it has no
need for something similar.

Author: Matthias van de Meent
Discussion: https://postgr.es/m/CAEze2WjE3+tGO9Fs9+iZMU+z6mMZKo54W1Zt98WKqbEUHbHOBg@mail.gmail.com
2022-04-01 13:24:50 +09:00
Michael Paquier 73db8f4d17 Improve handling and logging of TAP tests for pg_upgrade
This commit includes a set of improvements to help with the debugging of
failures in these new TAP tests:
- Instead of a plain diff command to compare the dumps generated, use
File::Compare::compare for the same effect.  diff is still used to
provide more context in the event of an error.
- Log the contents of regression.diffs if the pg_regress command fails.
- Unify the format of the logs generated, getting inspiration from the
style used in 027_stream_regress.pl.

wrasse is the only buildfarm member that has reported a failure until
now after the introduction of 322becb, complaining that the dumps
generated do not match, and I am lacking information to understand what
is going in this environment.
2022-04-01 12:45:40 +09:00
Michael Paquier 322becb608 Switch the regression tests of pg_upgrade to use TAP tests
This simplifies a lot of code in the tests of pg_upgrade without
sacrificing its coverage:
- Removal of test.sh used for builds with make, that has accumulated
over the years tweaks for problems that are solved in a duplicated way
by the centralized TAP framework (initialization of the various
environment variables PG*, port selection).
- Removal of the code in MSVC to test pg_upgrade.  This was roughly a
duplicate of test.sh adapted for Windows, with an extra footprint of
a pg_regress command and all the assumptions behind it.

Support for upgrades with older versions is changed, not removed.
test.sh was able to set up the regression database on the old instance
by launching itself the pg_regress command and a dependency to the
source tree of thd old cluster, with tweaks on the command arguments to
adapt across the versions used.  This created a backward-compatibility
dependency with older pg_regress commands, and recent changes like
d1029bb have made that much more complicated.

Instead, this commit allows tests with older major versions by
specifying a path to a SQL dump (taken with pg_dumpall from the old
cluster's installation) that will be loaded into the old instance to
upgrade instead of running pg_regress, through an optional environment
variable called $olddump.  This requires a second variable called
$oldinstall to point to the base path of the installation of the old
cluster.  This method is more in line with the buildfarm client that
uses a set of static dumps to set up an old instance, so hopefully we
will be able to reuse what is introduced in this commit there.  The last
step of the tests that checks for differences between the two dumps
taken still needs to be improved as it can fail, requiring a manual
lookup at the dumps.  This is not different from the old way of testing
where things could fail at the last step.

Support for EXTRA_REGRESS_OPTS is kept.  vcregress.pl in the MSVC
scripts still handles the test of pg_upgrade with its upgradecheck, and
bincheck is changed to skip pg_upgrade.

Author: Michael Paquier
Reviewed-by: Andrew Dunstan, Andres Freund, Rachel Heaton, Tom Lane,
Discussion: https://postgr.es/m/YJ8xTmLQkotVLpN5@paquier.xyz
2022-04-01 10:13:50 +09:00
Tom Lane fb691bbb4c Keep plpgsql.h C++-clean.
I forgot that "typeid" is a C++ keyword.  Per buildfarm.
2022-03-31 18:29:10 -04:00
Tom Lane 53ef6c40f1 Expose a few more PL/pgSQL functions to debugger plugins.
Add exec_assign_value, exec_eval_datum, and exec_cast_value
to the set of functions a PL/pgSQL debugger plugin can
conveniently call.  This allows more convenient manipulation
of the values of PL/pgSQL function variables.

Pavel Stehule, reviewed by Aleksander Alekseev and myself

Discussion: https://postgr.es/m/CAFj8pRD+dBPU0T-KrkP7ef6QNPDEsjYCejEsBe07NDq8TybOkA@mail.gmail.com
2022-03-31 17:05:47 -04:00
Andrew Dunstan 9f91344223 Fix comments with "a expression" 2022-03-31 15:45:25 -04:00
Andrew Dunstan 49082c2cc3 RETURNING clause for JSON() and JSON_SCALAR()
This patch is extracted from a larger patch that allowed setting the
default returned value from these functions to json or jsonb. That had
problems, but this piece of it is fine. For these functions only json or
jsonb can be specified in the RETURNING clause.

Extracted from an original patch from Nikita Glukhov

Reviewers have included (in no particular order) Andres Freund, Alexander
Korotkov, Pavel Stehule, Andrew Alsup, Erik Rijkers, Zihong Yu,
Himanshu Upadhyaya, Daniel Gustafsson, Justin Pryzby.

Discussion: https://postgr.es/m/cd0bb935-0158-78a7-08b5-904886deac4b@postgrespro.ru
2022-03-31 15:45:24 -04:00
Robert Haas ad43a413c4 initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.
Dilip Kumar, reviewed by Andres Freund and by me.

Discussion: http://postgr.es/m/20220330011757.wr544o5y5my7ssoa@alap3.anarazel.de
2022-03-31 15:15:11 -04:00
Tom Lane f3dd9fe1dd Fix postgres_fdw to check shippability of sort clauses properly.
postgres_fdw would push ORDER BY clauses to the remote side without
verifying that the sort operator is safe to ship.  Moreover, it failed
to print a suitable USING clause if the sort operator isn't default
for the sort expression's type.  The net result of this is that the
remote sort might not have anywhere near the semantics we expect,
which'd be disastrous for locally-performed merge joins in particular.

We addressed similar issues in the context of ORDER BY within an
aggregate function call in commit 7012b132d, but failed to notice
that query-level ORDER BY was broken.  Thus, much of the necessary
logic already existed, but it requires refactoring to be usable
in both cases.

Back-patch to all supported branches.  In HEAD only, remove the
core code's copy of find_em_expr_for_rel, which is no longer used
and really should never have been pushed into equivclass.c in the
first place.

Ronan Dunklau, per report from David Rowley;
reviews by David Rowley, Ranier Vilela, and myself

Discussion: https://postgr.es/m/CAApHDvr4OeC2DBVY--zVP83-K=bYrTD7F8SZDhN4g+pj2f2S-A@mail.gmail.com
2022-03-31 14:29:48 -04:00
Andres Freund 28bdfa2adf Print information about type of test and subdirectory before running tests.
When testing check-world it's hard to know what the test the test failure
output belongs to. The tap test output is especially problematic, partially
due to our practice of reusing test names like 001_basic.pl.

This isn't a real issue on the buildfarm, which invokes tests separately, but
locally and for CI it's quite annoying.

To fix, the test target provisos in Makefile.global.in now output
  echo "+++ (regress|isolation|tap) [install-]check in $(subdir) +++"
before running the tests.

Discussion: https://postgr.es/m/20220330165039.3zseuiraxfjkksf5@alap3.anarazel.de
2022-03-31 11:19:24 -07:00
Andrew Dunstan d5f43a1a10
Remove use of perl parent module in Cluster.pm
Commit fb16d2c658 used the old but not quite old enough parent module,
which dates to perl version 5.10.1 as a core module. We still have a
dinosaur or two running older versions of perl, so rather than require
an upgrade in those we simply do in place what parent.pm's import()
would have done for us.

Reviewed by Tom Lane

Discussion: https://postgr.es/m/474104.1648685981@sss.pgh.pa.us
2022-03-31 14:10:47 -04:00
Robert Haas fea1cc49e4 In basebackup_to_shell tests, properly set up pg_hba.conf.
Discussion: http://postgr.es/m/485495.1648692468@sss.pgh.pa.us
2022-03-31 14:06:17 -04:00
Peter Eisentraut 8910a25fef psql: Refactor SendQuery()
This breaks out the fetch-it-all-and-print case in SendQuery() into a
separate function.  This makes the code more similar to the other
cases \gdesc and run query with FETCH_COUNT, and makes SendQuery()
itself a bit smaller.

Extracted from a larger patch with more changes in this area to
follow.

Author: Fabien COELHO <coelho@cri.ensmp.fr>
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.1904132231510.8961@lancre
2022-03-31 19:59:29 +02:00
Tom Lane 878e64d0f8 Add missing newline in one libpq error message.
Oversight in commit a59c79564.  Back-patch, as that was.
Noted by Peter Eisentraut.

Discussion: https://postgr.es/m/7f85ef6d-250b-f5ec-9867-89f0b16d019f@enterprisedb.com
2022-03-31 11:24:26 -04:00
Peter Eisentraut d3ab618290 psql: Add tests for \errverbose
This is another piece of functionality that happens while a user query
is being sent and which did not have any test coverage.
2022-03-31 16:20:27 +02:00
Andrew Dunstan ddee016b34
Fix comment typo in PotsgreSQL::Test::Cluster module
Per Dagfinn Ilmari Mannsåker
2022-03-31 08:34:39 -04:00
Daniel Gustafsson 4e31c46e1e doc: Fix typo in ANALYZE documentation
Commit 61fa6ca79b accidentally wrote constrast instead of contrast.

Backpatch-through: 10
Discussion: https://postgr.es/m/88903179-5ce2-3d4d-af43-7830372bdcb6@enterprisedb.com
2022-03-31 12:03:33 +02:00
Daniel Gustafsson 2beb4acff1 Add diagnostic output on error in pump_until
When pump_until was moved to Utils.pm in commit 6da65a3f9 the diag
calls were removed, this puts them back.

Per request from Andres Freund.

Discussion: https://postgr.es/m/20220329225819.ahk5u2tax3ez6d2t@alap3.anarazel.de
2022-03-31 11:16:16 +02:00
Amit Kapila 8f2e2bbf14 Raise a WARNING for missing publications.
When we create or alter a subscription to add publications raise a warning
for non-existent publications. We don't want to give an error here because
it is possible that users can later create the missing publications.

Author: Vignesh C
Reviewed-by: Bharath Rupireddy, Japin Li, Dilip Kumar, Euler Taveira, Ashutosh Sharma, Amit Kapila
Discussion: https://postgr.es/m/CALDaNm0f4YujGW+q-Di0CbZpnQKFFrXntikaQQKuEmGG0=Zw=Q@mail.gmail.com
2022-03-31 08:25:50 +05:30