Commit Graph

30780 Commits

Author SHA1 Message Date
Andres Freund 6e7baa3227 Introduce BYTES unit for GUCs.
This is already useful for track_activity_query_size, and will further
be used in a later commit making the WAL segment size configurable.

Author: Beena Emerson
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/CAOG9ApEu8bXVwBxkOO9J7ZpM76TASK_vFMEEiCEjwhMmSLiaqQ@mail.gmail.com
2017-09-12 12:13:12 -07:00
Peter Eisentraut 83aaac41c6 Allow custom search filters to be configured for LDAP auth
Before, only filters of the form "(<ldapsearchattribute>=<user>)"
could be used to search an LDAP server.  Introduce ldapsearchfilter
so that more general filters can be configured using patterns, like
"(|(uid=$username)(mail=$username))" and "(&(uid=$username)
(objectClass=posixAccount))".  Also allow search filters to be included
in an LDAP URL.

Author: Thomas Munro
Reviewed-By: Peter Eisentraut, Mark Cave-Ayland, Magnus Hagander
Discussion: https://postgr.es/m/CAEepm=0XTkYvMci0WRubZcf_1am8=gP=7oJErpsUfRYcKF2gwg@mail.gmail.com
2017-09-12 09:49:04 -04:00
Michael Meskes 35e1568826 Fixed ECPG to correctly handle out-of-scope cursor declarations with pointers
or array variables.
2017-09-12 04:53:36 +02:00
Tom Lane e183530550 Fix RecursiveCopy.pm to cope with disappearing files.
When copying from an active database tree, it's possible for files to be
deleted after we see them in a readdir() scan but before we can open them.
(Once we've got a file open, we don't expect any further errors from it
getting unlinked, though.)  Tweak RecursiveCopy so it can cope with this
case, so as to avoid irreproducible test failures.

Back-patch to 9.6 where this code was added.  In v10 and HEAD, also
remove unused "use RecursiveCopy" in one recovery test script.

Michael Paquier and Tom Lane

Discussion: https://postgr.es/m/24621.1504924323@sss.pgh.pa.us
2017-09-11 22:02:58 -04:00
Peter Eisentraut 6d9fa52645 pg_receivewal: Add --endpos option
This is primarily useful for making tests of this utility more
deterministic, to avoid the complexity of starting pg_receivewal as a
deamon in TAP tests.

While this is less useful than the equivalent pg_recvlogical option,
users can as well use it for example to enforce WAL streaming up to a
end-of-backup position, to save only a minimal amount of WAL.

Use this new option to stream WAL data in a deterministic way within a
new set of TAP tests.

Author: Michael Paquier <michael.paquier@gmail.com>
2017-09-11 16:48:30 -04:00
Andres Freund c1898c3e1e Constify numeric.c.
This allows the compiler/linker to move the static variables to a
read-only segment.  Not all the signature changes are necessary, but
it seems better to apply const in a consistent manner.

Reviewed-By: Tom Lane
Discussion: https://postgr.es/m/20170910232154.asgml44ji2b7lv3d@alap3.anarazel.de
2017-09-11 13:44:37 -07:00
Tom Lane b8060e41b5 Prefer argument name over "$n" for the refname of a plpgsql argument.
If a function argument has a name, use that as the "refname" of the
PLpgSQL_datum representing the argument, instead of $n as before.
This allows better error messages in some cases.

Pavel Stehule, reviewed by Jeevan Chalke

Discussion: https://postgr.es/m/CAFj8pRB9GyU2U1Sb2ssgP26DZ_yq-FYDfpvUvGQ=k4R=yOPVjg@mail.gmail.com
2017-09-11 16:24:43 -04:00
Peter Eisentraut 821fb8cdbf Message style fixes 2017-09-11 11:21:27 -04:00
Tom Lane 3c43595217 Quick-hack fix for foreign key cascade vs triggers with transition tables.
AFTER triggers using transition tables crashed if they were fired due
to a foreign key ON CASCADE update.  This is because ExecEndModifyTable
flushes the transition tables, on the assumption that any trigger that
could need them was already fired during ExecutorFinish.  Normally
that's true, because we don't allow transition-table-using triggers
to be deferred.  However, foreign key CASCADE updates force any
triggers on the referencing table to be deferred to the outer query
level, by means of the EXEC_FLAG_SKIP_TRIGGERS flag.  I don't recall
all the details of why it's like that and am pretty loath to redesign
it right now.  Instead, just teach ExecEndModifyTable to skip destroying
the TransitionCaptureState when that flag is set.  This will allow the
transition table data to survive until end of the current subtransaction.

This isn't a terribly satisfactory solution, because (1) we might be
leaking the transition tables for much longer than really necessary,
and (2) as things stand, an AFTER STATEMENT trigger will fire once per
RI updating query, ie once per row updated or deleted in the referenced
table.  I suspect that is not per SQL spec.  But redesigning this is a
research project that we're certainly not going to get done for v10.
So let's go with this hackish answer for now.

In passing, tweak AfterTriggerSaveEvent to not save the transition_capture
pointer into the event record for a deferrable trigger.  This is not
necessary to fix the current bug, but it avoids letting dangling pointers
to long-gone transition tables persist in the trigger event queue.  That's
at least a safety feature.  It might also allow merging shared trigger
states in more cases than before.

I added a regression test that demonstrates the crash on unpatched code,
and also exposes the behavior of firing the AFTER STATEMENT triggers
once per row update.

Per bug #14808 from Philippe Beaudoin.  Back-patch to v10.

Discussion: https://postgr.es/m/20170909064853.25630.12825@wrigleys.postgresql.org
2017-09-10 14:59:56 -04:00
Tom Lane 610bbdd8ac Add a test harness for the red-black tree code.
This improves the regression tests' coverage of rbtree.c from pretty
awful (because some of the functions aren't used yet) to basically 100%.

Victor Drobny, reviewed by Aleksander Alekseev and myself

Discussion: https://postgr.es/m/c9d61310e16e75f8acaf6cb1c48b7b77@postgrespro.ru
2017-09-10 13:26:46 -04:00
Tom Lane f80e782a6b Remove pre-order and post-order traversal logic for red-black trees.
This code isn't used, and there's no clear reason why anybody would ever
want to use it.  These traversal mechanisms don't yield a visitation order
that is semantically meaningful for any external purpose, nor are they
any faster or simpler than the left-to-right or right-to-left traversals.
(In fact, some rough testing suggests they are slower :-(.)  Moreover,
these mechanisms are impossible to test in any arm's-length fashion; doing
so requires knowledge of the red-black tree's internal implementation.
Hence, let's just jettison them.

Discussion: https://postgr.es/m/17735.1505003111@sss.pgh.pa.us
2017-09-10 13:19:11 -04:00
Peter Eisentraut c824c7e29f pg_upgrade: Message style fixes 2017-09-09 17:32:10 -04:00
Tom Lane fdf87ed451 Fix failure-to-copy bug in commit 6f6b99d13.
The previous coding of get_qual_for_list() was careful to copy everything
it was using from the input data structure.  The new version missed
making a copy of pass-by-ref datum values that it's inserting into Consts.
This is not optional, however, as revealed by buildfarm failures on
machines running -DRELCACHE_FORCE_RELEASE: we're copying from a relcache
entry that could go away before the required lifespan of our output
expression.  I'm pretty sure -DCLOBBER_CACHE_ALWAYS machines won't like
this either, but none of them have reported in yet.
2017-09-08 20:45:31 -04:00
Tom Lane e56dd7cf50 Fix uninitialized-variable bug.
map_partition_varattnos() failed to set its found_whole_row output
parameter if the given expression list was NIL.  This seems to be
a pre-existing bug that chanced to be exposed by commit 6f6b99d13.
It might be unreachable in v10, but I have little faith in that
proposition, so back-patch.

Per buildfarm.
2017-09-08 19:04:32 -04:00
Tom Lane f25000c832 Fix more portability issues in new pgbench TAP tests.
Not completely sure, but I think bowerbird is spitting up on attempting
to include ">" in a temporary file name.  (Why in the world are we
writing this stuff into files at all?  A hash would be a better answer.)
2017-09-08 17:37:59 -04:00
Robert Haas 6f6b99d133 Allow a partitioned table to have a default partition.
Any tuples that don't route to any other partition will route to the
default partition.

Jeevan Ladhe, Beena Emerson, Ashutosh Bapat, Rahila Syed, and Robert
Haas, with review and testing at various stages by (at least) Rushabh
Lathia, Keith Fiske, Amit Langote, Amul Sul, Rajkumar Raghuanshi, Sven
Kunze, Kyotaro Horiguchi, Thom Brown, Rafia Sabih, and Dilip Kumar.

Discussion: http://postgr.es/m/CAH2L28tbN4SYyhS7YV1YBWcitkqbhSWfQCy0G=apRcC_PEO-bg@mail.gmail.com
Discussion: http://postgr.es/m/CAOG9ApEYj34fWMcvBMBQ-YtqR9fTdXhdN82QEKG0SVZ6zeL1xg@mail.gmail.com
2017-09-08 17:28:04 -04:00
Tom Lane 2cf15ec8b1 Fix pgbench TAP tests to work with --disable-thread-safety.
Probably matters to nobody but me; but I'd like to still be able to get
through the TAP tests on gaur/pademelon, from time to time.
2017-09-08 17:25:11 -04:00
Tom Lane 3cf17c9d47 Remove mention of password_encryption = plain in postgresql.conf.sample.
Evidently missed in commit eb61136dc.

Spotted by Oleg Bartunov.

Discussion: https://postgr.es/m/CAF4Au4wz_iK5r4fnTnnd8XqioAZQs-P7-VsEAfivW34zMVpAmw@mail.gmail.com
2017-09-08 14:38:54 -04:00
Tom Lane 933851033b Fix more portability issues in new pgbench TAP tests.
Strike two on the --bad-option test.
Three strikes and it's out.

Fabien Coelho, per buildfarm
2017-09-08 14:01:51 -04:00
Tom Lane 77d63b7eaf Fix more portability issues in new pgbench TAP tests.
* Remove no-such-user test case, output isn't stable, and we really
don't need to be testing such cases here anyway.

* Fix the process exit code test logic to match PostgresNode::psql
(but I didn't bother with looking at the "core" flag).

* Give up on inf/nan tests.

Per buildfarm.
2017-09-08 13:36:13 -04:00
Peter Eisentraut ee24d2b5cf Clean up excessive code
The encoding ID was converted between string and number too many times,
probably a remnant from the shell script days.

Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-09-08 12:37:05 -04:00
Peter Eisentraut 8e67380126 Remove useless empty string initializations
This coding style probably stems from the days of shell scripts.

Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-09-08 12:37:05 -04:00
Peter Eisentraut 9361bc347c Remove useless dead code
Reviewed-by: Aleksandr Parfenov <a.parfenov@postgrespro.ru>
2017-09-08 12:37:05 -04:00
Tom Lane 869aa40a27 Fix assorted portability issues in new pgbench TAP tests.
* Our own version of getopt_long doesn't support abbreviation of
long options.

* It doesn't do automatic rearrangement of non-option arguments to the end,
either.

* Test was way too optimistic about the platform independence of
NaN and Infinity outputs.  I rather imagine we might have to lose
those tests altogether, but for the moment just allow case variation
and fully spelled out Infinity.

Per buildfarm.
2017-09-08 11:28:02 -04:00
Tom Lane ed8a7c6fcf Add much-more-extensive TAP tests for pgbench.
Fabien Coelho, reviewed by Nikolay Shaplov and myself

Discussion: https://postgr.es/m/alpine.DEB.2.20.1704171422500.4025@lancre
2017-09-08 09:32:50 -04:00
Robert Haas f0a0c17c1b Refactor get_partition_for_tuple a bit.
Pending patches for both default partitioning and hash partitioning
find the current coding pattern to be inconvenient.  Change it so that
we switch on the partitioning method first and then do whatever is
needed.

Amul Sul, reviewed by Jeevan Ladhe, with a few adjustments by me.

Discussion: http://postgr.es/m/CAAJ_b97mTb=dG2pv6+1ougxEVZFVnZJajW+0QHj46mEE7WsoOQ@mail.gmail.com
Discussion: http://postgr.es/m/CAOgcT0M37CAztEinpvjJc18EdHfm23fw0EG9-36Ya=+rEFUqaQ@mail.gmail.com
2017-09-07 21:13:42 -04:00
Tom Lane 3ca930fc39 Improve performance of get_actual_variable_range with recently-dead tuples.
In commit fccebe421, we hacked get_actual_variable_range() to scan the
index with SnapshotDirty, so that if there are many uncommitted tuples
at the end of the index range, it wouldn't laboriously scan through all
of them looking for a live value to return.  However, that didn't fix it
for the case of many recently-dead tuples at the end of the index;
SnapshotDirty recognizes those as committed dead and so we're back to
the same problem.

To improve the situation, invent a "SnapshotNonVacuumable" snapshot type
and use that instead.  The reason this helps is that, if the snapshot
rejects a given index entry, we know that the indexscan will mark that
index entry as killed.  This means the next get_actual_variable_range()
scan will proceed past that entry without visiting the heap, making the
scan a lot faster.  We may end up accepting a recently-dead tuple as
being the estimated extremal value, but that doesn't seem much worse than
the compromise we made before to accept not-yet-committed extremal values.

The cost of the scan is still proportional to the number of dead index
entries at the end of the range, so in the interval after a mass delete
but before VACUUM's cleaned up the mess, it's still possible for
get_actual_variable_range() to take a noticeable amount of time, if you've
got enough such dead entries.  But the constant factor is much much better
than before, since all we need to do with each index entry is test its
"killed" bit.

We chose to back-patch commit fccebe421 at the time, but I'm hesitant to
do so here, because this form of the problem seems to affect many fewer
people.  Also, even when it happens, it's less bad than the case fixed
by commit fccebe421 because we don't get the contention effects from
expensive TransactionIdIsInProgress tests.

Dmitriy Sarafannikov, reviewed by Andrey Borodin

Discussion: https://postgr.es/m/05C72CF7-B5F6-4DB9-8A09-5AC897653113@yandex.ru
2017-09-07 19:41:51 -04:00
Peter Eisentraut 1356f78ea9 Reduce excessive dereferencing of function pointers
It is equivalent in ANSI C to write (*funcptr) () and funcptr().  These
two styles have been applied inconsistently.  After discussion, we'll
use the more verbose style for plain function pointer variables, to make
it clear that it's a variable, and the shorter style when the function
pointer is in a struct (s.func() or s->func()), because then it's clear
that it's not a plain function name, and otherwise the excessive
punctuation makes some of those invocations hard to read.

Discussion: https://www.postgresql.org/message-id/f52c16db-14ed-757d-4b48-7ef360b1631d@2ndquadrant.com
2017-09-07 13:56:09 -04:00
Robert Haas 9d71323dac Even if some partitions are foreign, allow tuple routing.
This doesn't allow routing tuple to the foreign partitions themselves,
but it permits tuples to be routed to regular partitions despite the
presence of foreign partitions in the same inheritance hierarchy.

Etsuro Fujita, reviewed by Amit Langote and by me.

Discussion: http://postgr.es/m/bc3db4c1-1693-3b8a-559f-33ad2b50b7ad@lab.ntt.co.jp
2017-09-07 10:58:21 -04:00
Tom Lane 6eb52da394 Fix handling of savepoint commands within multi-statement Query strings.
Issuing a savepoint-related command in a Query message that contains
multiple SQL statements led to a FATAL exit with a complaint about
"unexpected state STARTED".  This is a shortcoming of commit 4f896dac1,
which attempted to prevent such misbehaviors in multi-statement strings;
its quick hack of marking the individual statements as "not top-level"
does the wrong thing in this case, and isn't a very accurate description
of the situation anyway.

To fix, let's introduce into xact.c an explicit model of what happens for
multi-statement Query strings.  This is an "implicit transaction block
in progress" state, which for many purposes works like the normal
TBLOCK_INPROGRESS state --- in particular, IsTransactionBlock returns true,
causing the desired result that PreventTransactionChain will throw error.
But in case of error abort it works like TBLOCK_STARTED, allowing the
transaction to be cancelled without need for an explicit ROLLBACK command.

Commit 4f896dac1 is reverted in toto, so that we go back to treating the
individual statements as "top level".  We could have left it as-is, but
this allows sharpening the error message for PreventTransactionChain
calls inside functions.

Except for getting a normal error instead of a FATAL exit for savepoint
commands, this patch should result in no user-visible behavioral change
(other than that one error message rewording).  There are some things
we might want to do in the line of changing the appearance or wording of
error and warning messages around this behavior, which would be much
simpler to do now that it's an explicitly modeled state.  But I haven't
done them here.

Although this fixes a long-standing bug, no backpatch.  The consequences
of the bug don't seem severe enough to justify the risk that this commit
itself creates some new issue.

Patch by me, but it owes something to previous investigation by
Takayuki Tsunakawa, who also reported the bug in the first place.
Also thanks to Michael Paquier for reviewing.

Discussion: https://postgr.es/m/0A3221C70F24FB45833433255569204D1F6BE40D@G01JPEXMBYT05
2017-09-07 09:49:55 -04:00
Tom Lane bfea92563c Further marginal hacking on generic atomic ops.
In the generic atomic ops that rely on a loop around a CAS primitive,
there's no need to force the initial read of the "old" value to be atomic.
In the typically-rare case that we get a torn value, that simply means
that the first CAS attempt will fail; but it will update "old" to the
atomically-read value, so the next attempt has a chance of succeeding.
It was already being done that way in pg_atomic_exchange_u64_impl(),
but let's duplicate the approach in the rest.

(Given the current coding of the pg_atomic_read functions, this change
is a no-op anyway on popular platforms; it only makes a difference where
pg_atomic_read_u64_impl() is implemented as a CAS.)

In passing, also remove unnecessary take-a-pointer-and-dereference-it
coding in the pg_atomic_read functions.  That seems to have been based
on a misunderstanding of what the C standard requires.  What actually
matters is that the pointer be declared as pointing to volatile, which
it is.

I don't believe this will change the assembly code at all on x86
platforms (even ignoring the likelihood that these implementations
get overridden by others); but it may help on less-mainstream CPUs.

Discussion: https://postgr.es/m/13707.1504718238@sss.pgh.pa.us
2017-09-07 08:50:01 -04:00
Simon Riggs f06588a8e6 Exclude special values in recovery_target_time
recovery_target_time accepts timestamp input, though
does not allow use of special values, e.g. “today”.
Report a useful error message for these cases.

Reported-by: Piotr Stefaniak
Author: Simon Riggs
Discussion: https://postgr.es/m/CANP8+jJdKA+BkkYLWz9zAm16Y0s2ExBv0WfpAwXdTpPfWnA9Bg@mail.gmail.com
2017-09-07 04:56:34 -07:00
Tom Lane ca4e20fde8 Merge duplicative code for \sf/\sv, \ef/\ev in psql/command.c.
Saves ~150 lines, costs little.

Fabien Coelho, reviewed by Victor Drobny

Discussion: https://postgr.es/m/alpine.DEB.2.20.1703311958001.14355@lancre
2017-09-06 17:32:40 -04:00
Simon Riggs 5b6d13eec7 Allow SET STATISTICS on expression indexes
Index columns are referenced by ordinal number rather than name, e.g.
CREATE INDEX coord_idx ON measured (x, y, (z + t));
ALTER INDEX coord_idx ALTER COLUMN 3 SET STATISTICS 1000;

Incompatibility note for release notes:
\d+ for indexes now also displays Stats Target

Authors: Alexander Korotkov, with contribution by Adrien NAYRAT
Review: Adrien NAYRAT, Simon Riggs
Wordsmith: Simon Riggs
2017-09-06 13:46:01 -07:00
Tom Lane e09db94c0a Use more of gcc's __sync_fetch_and_xxx builtin functions for atomic ops.
In addition to __sync_fetch_and_add, gcc offers __sync_fetch_and_sub,
__sync_fetch_and_and, and __sync_fetch_and_or, which correspond directly
to primitive atomic ops that we want.  Testing shows that in some cases
they generate better code than our generic implementations, so use them.

We've assumed that configure's test for __sync_val_compare_and_swap is
sufficient to allow assuming that __sync_fetch_and_add is available, so
make the same assumption for these functions.  Should that prove to be
wrong, we can add more configure tests.

Yura Sokolov, reviewed by Jesper Pedersen and myself

Discussion: https://postgr.es/m/7f65886daca545067f82bf2b463b218d@postgrespro.ru
2017-09-06 14:21:39 -04:00
Tom Lane e530be9685 Remove duplicate reads from the inner loops in generic atomic ops.
The pg_atomic_compare_exchange_xxx functions are defined to update
*expected to whatever they read from the target variable.  Therefore,
there's no need to do additional explicit reads after we've initialized
the "old" variable.  The actual benefit of this is somewhat debatable,
but it seems fairly unlikely to hurt anything, especially since we
will override the generic implementations in most performance-sensitive
cases.

Yura Sokolov, reviewed by Jesper Pedersen and myself

Discussion: https://postgr.es/m/7f65886daca545067f82bf2b463b218d@postgrespro.ru
2017-09-06 14:06:09 -04:00
Tom Lane 8689e38263 Clean up handling of dropped columns in NAMEDTUPLESTORE RTEs.
The NAMEDTUPLESTORE patch piggybacked on the infrastructure for
TABLEFUNC/VALUES/CTE RTEs, none of which can ever have dropped columns,
so the possibility was ignored most places.  Fix that, including adding a
specification to parsenodes.h about what it's supposed to look like.

In passing, clean up assorted comments that hadn't been maintained
properly by said patch.

Per bug #14799 from Philippe Beaudoin.  Back-patch to v10.

Discussion: https://postgr.es/m/20170906120005.25630.84360@wrigleys.postgresql.org
2017-09-06 10:41:05 -04:00
Tom Lane 49ca462eb1 Add \gdesc psql command.
This command acts somewhat like \g, but instead of executing the query
buffer, it merely prints a description of the columns that the query
result would have.  (Of course, this still requires parsing the query;
if parse analysis fails, you get an error anyway.)  We accomplish this
using an unnamed prepared statement, which should be invisible to psql
users.

Pavel Stehule, reviewed by Fabien Coelho

Discussion: https://postgr.es/m/CAFj8pRBhYVvO34FU=EKb=nAF5t3b++krKt1FneCmR0kuF5m-QA@mail.gmail.com
2017-09-05 18:17:47 -04:00
Tom Lane 6e427aa4e5 Use lfirst_node() and linitial_node() where appropriate in planner.c.
There's no particular reason to target this module for the first
wholesale application of these macros; but we gotta start somewhere.

Ashutosh Bapat and Jeevan Chalke

Discussion: https://postgr.es/m/CAFjFpRcNr3r=u0ni=7A4GD9NnHQVq+dkFafzqo2rS6zy=dt1eg@mail.gmail.com
2017-09-05 15:57:48 -04:00
Peter Eisentraut 153a49bb33 Remove endof macro
It has not been used in a long time, and it doesn't seem safe anyway, so
drop it.

Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Ryan Murphy <ryanfmurphy@gmail.com>
2017-09-05 14:52:55 -04:00
Peter Eisentraut 17273d059c Remove unnecessary parentheses in return statements
The parenthesized style has only been used in a few modules.  Change
that to use the style that is predominant across the whole tree.

Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Ryan Murphy <ryanfmurphy@gmail.com>
2017-09-05 14:52:55 -04:00
Peter Eisentraut ba26f5cf76 Remove our own definition of NULL
Surely everyone has that by now.

Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Ryan Murphy <ryanfmurphy@gmail.com>
2017-09-05 14:52:54 -04:00
Peter Eisentraut 90627cf98a Support retaining data dirs on successful TAP tests
This moves the data directories from using temporary directories with
randomness in the directory name to a static name, to make it easier to
debug.  The data directory will be retained if tests fail or the test
code dies/exits with failure, and is automatically removed on the next
make check.

If the environment variable PG_TEST_NOCLEAN is defined, the data
directories will be retained regardless of test or exit status.

Author: Daniel Gustafsson <daniel@yesql.se>
2017-09-05 12:24:06 -04:00
Tom Lane 5e8304fdce In psql, use PSQL_PAGER in preference to PAGER, if it's set.
This allows the user's environment to set up a psql-specific choice
of pager, in much the same way that we provide PSQL_EDITOR to allow
a psql-specific override of the more widely known EDITOR variable.

Pavel Stehule, reviewed by Thomas Munro

Discussion: https://postgr.es/m/CAFj8pRD3RRk9S1eRbnGm_T6brc3Ss5mohraNzTSJquzx+pmtKA@mail.gmail.com
2017-09-05 12:02:13 -04:00
Alvaro Herrera ebd346caf4 Correct base backup throttling
Throttling for sending a base backup in walsender is broken for the case
where there is a lot of WAL traffic, because the latch used to put the
walsender to sleep is also signalled by regular WAL traffic (and each
signal causes an additional batch of data to be sent); the net effect is
that there is no or little actual throttling.  This is undesirable, so
rewrite the sleep into a loop to achieve the desired effeect.

Author: Jeff Janes, small tweaks by me
Reviewed-by: Antonin Houska
Discussion: https://postgr.es/m/CAMkU=1xH6mde-yL-Eo1TKBGNd0PB1-TMxvrNvqcAkN-qr2E9mw@mail.gmail.com
2017-09-05 17:27:30 +02:00
Tom Lane 9ae9d8c154 Add psql variables showing server version and psql version.
We already had a psql variable VERSION that shows the verbose form of
psql's own version.  Add VERSION_NAME to show the short form (e.g.,
"11devel") and VERSION_NUM to show the numeric form (e.g., 110000).
Also add SERVER_VERSION_NAME and SERVER_VERSION_NUM to show the short and
numeric forms of the server's version.  (We'd probably add SERVER_VERSION
with the verbose string if it were readily available; but adding another
network round trip to get it seems too expensive.)

The numeric forms, in particular, are expected to be useful for scripting
purposes, now that psql can do conditional tests.

Fabien Coelho, reviewed by Pavel Stehule

Discussion: https://postgr.es/m/alpine.DEB.2.20.1704020917220.4632@lancre
2017-09-05 10:51:36 -04:00
Tom Lane 3955c8c4ed Reformat psql's --help=variables output.
The previous format with variable names and descriptions in separate
columns was extremely constraining about the length of the descriptions.
We'd dealt with that in several inconsistent ways over the years,
including letting the lines run over 80 characters, breaking descriptions
into multiple lines, or shoving the description onto a separate line.
But it's been a long time since the output could realistically fit onto
a single screen vertically, so let's just rely even more heavily on the
pager to deal with the vertical distance, and split each entry into two
(or more) lines, in the format

  variable-name
    variable description goes here

Each variable name + description remains a single translatable string,
in hopes of reducing translator confusion; we're just changing the
embedded whitespace.

I failed to resist the temptation to copy-edit one or two of the
descriptions while at it.

Discussion: https://postgr.es/m/2947.1504542679@sss.pgh.pa.us
2017-09-05 10:17:10 -04:00
Tom Lane 0b707d6ea7 Be more careful about newline-chomping in pgbench.
process_backslash_command would drop the last character of the input
command on the assumption that it was a newline.  Given a non newline
terminated input file, this could result in dropping the last character
of the command.  Fix that by doing an actual test that we're removing
a newline.

While at it, allow for Windows newlines (\r\n), and suppress multiple
newlines if any.  I do not think either of those cases really occur,
since (a) we read script files in text mode and (b) the lexer stops
when it hits a newline.  But it's cheap enough and it provides a
stronger guarantee about what the result string looks like.

This is just cosmetic, I think, since the possibly-overly-chomped
line was only used for display not for further processing.  So
it doesn't seem necessary to back-patch.

Fabien Coelho, reviewed by Nikolay Shaplov, whacked around a bit by me

Discussion: https://postgr.es/m/alpine.DEB.2.20.1704171422500.4025@lancre
2017-09-04 17:25:31 -04:00
Tom Lane c23bb6badf Fix some subtle problems in pgbench transaction stats counting.
With --latency-limit, transactions might get skipped even beyond the
transaction count limit specified by -t, throwing off the expected
number of transactions and thus the denominator for later stats.
Be sure to stop skipping transactions once -t is reached.

Also, include skipped transactions in the "cnt" fields; this requires
discounting them again in a couple of places, but most places are
better off with this definition.  In particular this is needed to
get correct overall stats for the combination of -R/-L/-t options.
Merge some more processing into processXactStats() to simplify this.

In passing, add a check that --progress-timestamp is specified only
when --progress is.

We might consider back-patching this, but given that it only matters
for a combination of options, and given the lack of field complaints,
consensus seems to be not to bother.

Fabien Coelho, reviewed by Nikolay Shaplov

Discussion: https://postgr.es/m/alpine.DEB.2.20.1704171422500.4025@lancre
2017-09-04 16:24:08 -04:00
Tom Lane 9d36a38660 Adjust pgbench to allow non-ASCII characters in variable names.
This puts it in sync with psql's notion of what is a valid variable name.
Like psql, we document that "non-Latin letters" are allowed, but actually
any non-ASCII character is accepted.

Fabien Coelho

Discussion: https://postgr.es/m/20170405.094548.1184280384967203518.t-ishii@sraoss.co.jp
2017-09-04 13:45:20 -04:00
Alvaro Herrera 863d75439e Fix translatable string
Discussion: https://postgr.es/m/20170828130545.sdajqlpr37hmmd6a@alvherre.pgsql
2017-09-04 11:08:52 +02:00
Tom Lane 4faa1dc2eb Suppress compiler warnings in dshash.c.
Some compilers complain, not unreasonably, about left-shifting an
int32 "1" and then assigning the result to an int64.  In practice
I sure hope that this data structure never gets large enough that
an overflow would actually occur; but let's cast the constant to
the right type to avoid the hazard.

In passing, fix a typo in dshash.h.

Amit Kapila, adjusted as per comment from Thomas Munro.

Discussion: https://postgr.es/m/CAA4eK1+5vfVMYtjK_NX8O3-42yM3o80qdqWnQzGquPrbq6mb+A@mail.gmail.com
2017-09-03 11:12:29 -04:00
Tom Lane e451901804 Fix macro-redefinition warning on MSVC.
In commit 9d6b160d7, I tweaked pg_config.h.win32 to use
"#define HAVE_LONG_LONG_INT_64 1" rather than defining it as empty,
for consistency with what happens in an autoconf'd build.
But Solution.pm injects another definition of that macro into
ecpg_config.h, leading to justifiable (though harmless) compiler whining.
Make that one consistent too.  Back-patch, like the previous patch.

Discussion: https://postgr.es/m/CAEepm=1dWsXROuSbRg8PbKLh0S=8Ou-V8sr05DxmJOF5chBxqQ@mail.gmail.com
2017-09-03 11:01:08 -04:00
Tom Lane 51daa7bdb3 Improve division of labor between execParallel.c and nodeGather[Merge].c.
Move the responsibility for creating/destroying TupleQueueReaders into
execParallel.c, to avoid duplicative coding in nodeGather.c and
nodeGatherMerge.c.  Also, instead of having DestroyTupleQueueReader do
shm_mq_detach, do it in the caller (which is now only ExecParallelFinish).
This means execParallel.c does both the attaching and detaching of the
tuple-queue-reader shm_mqs, which seems less weird than the previous
arrangement.

These changes also eliminate a vestigial memory leak (of the pei->tqueue
array).  It's now demonstrable that rescans of Gather or GatherMerge don't
leak memory.

Discussion: https://postgr.es/m/8670.1504192177@sss.pgh.pa.us
2017-09-01 17:39:01 -04:00
Peter Eisentraut c039ba0716 Add memory info to getrusage output
Add the maxrss field to the getrusage output (log_*_stats).  This was
previously omitted because of portability concerns, but we feel this
might not be a concern anymore.

based on patch by Justin Pryzby <pryzby@telsasoft.com>
2017-09-01 15:36:33 -04:00
Robert Haas 0cb8b7531d Tighten up some code in RelationBuildPartitionDesc.
This probably doesn't save anything meaningful in terms of
performance, but making the code simpler is a good idea anyway.

Code by Beena Emerson, extracted from a larger patch by Jeevan
Ladhe, slightly adjusted by me.

Discussion: http://postgr.es/m/CAOgcT0ONgwajdtkoq+AuYkdTPY9cLWWLjxt_k4SXue3eieAr+g@mail.gmail.com
2017-09-01 15:16:44 -04:00
Tom Lane 9d6b160d7d Make [U]INT64CONST safe for use in #if conditions.
Instead of using a cast to force the constant to be the right width,
assume we can plaster on an L, UL, LL, or ULL suffix as appropriate.
The old approach to this is very hoary, dating from before we were
willing to require compilers to have working int64 types.

This fix makes the PG_INT64_MIN, PG_INT64_MAX, and PG_UINT64_MAX
constants safe to use in preprocessor conditions, where a cast
doesn't work.  Other symbolic constants that might be defined using
[U]INT64CONST are likewise safer than before.

Also fix the SIZE_MAX macro to be similarly safe, if we are forced
to provide a definition for that.  The test added in commit 2e70d6b5e
happens to do what we want even with the hack "(size_t) -1" definition,
but we could easily get burnt on other tests in future.

Back-patch to all supported branches, like the previous commits.

Discussion: https://postgr.es/m/15883.1504278595@sss.pgh.pa.us
2017-09-01 15:14:46 -04:00
Tom Lane b79d69b087 Ensure SIZE_MAX can be used throughout our code.
Pre-C99 platforms may lack <stdint.h> and thereby SIZE_MAX.  We have
a couple of places using the hack "(size_t) -1" as a fallback, but
it wasn't universally available; which means the code added in commit
2e70d6b5e fails to compile everywhere.  Move that hack to c.h so that
we can rely on having SIZE_MAX everywhere.

Per discussion, it'd be a good idea to make the macro's value safe
for use in #if-tests, but that will take a bit more work.  This is
just a quick expedient to get the buildfarm green again.

Back-patch to all supported branches, like the previous commit.

Discussion: https://postgr.es/m/15883.1504278595@sss.pgh.pa.us
2017-09-01 13:52:53 -04:00
Robert Haas 84be67181a pg_dumpall: Add a -E flag to set the encoding, like pg_dump has.
Michael Paquier, reviewed by Fabien Coelho

Discussion: http://postgr.es/m/CAB7nPqQcYWmrm2n-dVaMUhYPKFU_DxQwPuUGuC4ZF+8B=dS5xQ@mail.gmail.com
2017-09-01 12:24:20 -04:00
Robert Haas baaf272ac9 Use group updates when setting transaction status in clog.
Commit 0e141c0fbb introduced a mechanism
to reduce contention on ProcArrayLock by having a single process clear
XIDs in the procArray on behalf of multiple processes, reducing the
need to hand the lock around.  A previous attempt to introduce a similar
mechanism for CLogControlLock in ccce90b398
crashed and burned, but the design problem which resulted in those
failures is believed to have been corrected in this version.

Amit Kapila, with some cosmetic changes by me.  See the previous commit
message for additional credits.

Discussion: http://postgr.es/m/CAA4eK1KudxzgWhuywY_X=yeSAhJMT4DwCjroV5Ay60xaeB2Eew@mail.gmail.com
2017-09-01 11:45:40 -04:00
Alvaro Herrera 89c59b742a Fix two-phase commit test for recovery mode
The original code had a race condition because it never ensured the
standby was caught up before proceeding; add a wait similar to every
other place that does this.

Author: Michaël Paquier
Discussion: https://postgr.es/m/CAB7nPqTm9p+LCm1mVJYvgpwagRK+uibT-pKq0O2-paOWxT62jw@mail.gmail.com
2017-09-01 16:51:55 +02:00
Alvaro Herrera a6979c3a68 Restore behavior for replication origin drop
Do for replication origins what the previous commit did for replication
slots: restore the original behavior of replication origin drop to raise
an error rather than blocking, because users might be depending on the
original behavior.  Maintain the blocking behavior when invoked
internally from logical replication subscription handling.

Discussion: https://postgr.es/m/20170830133922.tlpo3lgfejm4n2cs@alvherre.pgsql
2017-09-01 16:30:02 +02:00
Simon Riggs 4f27c674fd Avoid race condition in logical replication test
Wait for slot to become inactive before continuing.

Author: Petr Jelinek
2017-09-01 14:55:44 +01:00
Alvaro Herrera be7161566d Add a WAIT option to DROP_REPLICATION_SLOT
Commit 9915de6c1c changed the default behavior of
DROP_REPLICATION_SLOT so that it would wait until any session holding
the slot active would release it, instead of raising an error.  But
users are already depending on the original behavior, so revert to it by
default and add a WAIT option to invoke the new behavior.

Per complaint from Simone Gotti, in
Discussion: https://postgr.es/m/CAEvsy6Wgdf90O6pUvg2wSVXL2omH5OPC-38OD4Zzgk-FXavj3Q@mail.gmail.com
2017-09-01 13:44:14 +02:00
Robert Haas 7b69b6ceb8 Fix assorted carelessness about Datum vs. int64 vs. uint64
Bugs introduced by commit 81c5e46c49
2017-09-01 00:14:54 -04:00
Robert Haas 0d9506d125 Try to repair poorly-considered code in previous commit. 2017-08-31 23:09:00 -04:00
Robert Haas 81c5e46c49 Introduce 64-bit hash functions with a 64-bit seed.
This will be useful for hash partitioning, which needs a way to seed
the hash functions to avoid problems such as a hash index on a hash
partitioned table clumping all values into a small portion of the
bucket space; it's also useful for anything that wants a 64-bit hash
value rather than a 32-bit hash value.

Just in case somebody wants a 64-bit hash value that is compatible
with the existing 32-bit hash values, make the low 32-bits of the
64-bit hash value match the 32-bit hash value when the seed is 0.

Robert Haas and Amul Sul

Discussion: http://postgr.es/m/CA+Tgmoafx2yoJuhCQQOL5CocEi-w_uG4S2xT0EtgiJnPGcHW3g@mail.gmail.com
2017-08-31 22:21:21 -04:00
Tom Lane 2d44c58c79 Avoid memory leaks when a GatherMerge node is rescanned.
Rescanning a GatherMerge led to leaking some memory in the executor's
query-lifespan context, because most of the node's working data structures
were simply abandoned and rebuilt from scratch.  In practice, this might
never amount to much, given the cost of relaunching worker processes ---
but it's still pretty messy, so let's fix it.

We can rearrange things so that the tuple arrays are simply cleared and
reused, and we don't need to rebuild the TupleTableSlots either, just
clear them.  One small complication is that because we might get a
different number of workers on each iteration, we can't keep the old
convention that the leader's gm_slots[] entry is the last one; the leader
might clobber a TupleTableSlot that we need for a worker in a future
iteration.  Hence, adjust the logic so that the leader has slot 0 always,
while the active workers have slots 1..n.

Back-patch to v10 to keep all the existing versions of nodeGatherMerge.c
in sync --- because of the renumbering of the slots, there would otherwise
be a very large risk that any future backpatches in this module would
introduce bugs.

Discussion: https://postgr.es/m/8670.1504192177@sss.pgh.pa.us
2017-08-31 16:21:05 -04:00
Robert Haas 30833ba154 Expand partitioned tables in PartDesc order.
Previously, we expanded the inheritance hierarchy in the order in
which find_all_inheritors had locked the tables, but that turns out
to block quite a bit of useful optimization.  For example, a
partition-wise join can't count on two tables with matching bounds
to get expanded in the same order.

Where possible, this change results in expanding partitioned tables in
*bound* order.  Bound order isn't well-defined for a list-partitioned
table with a null-accepting partition or for a list-partitioned table
where the bounds for a single partition are interleaved with other
partitions.  However, when expansion in bound order is possible, it
opens up further opportunities for optimization, such as
strength-reducing MergeAppend to Append when the expansion order
matches the desired sort order.

Patch by me, with cosmetic revisions by Ashutosh Bapat.

Discussion: http://postgr.es/m/CA+TgmoZrKj7kEzcMSum3aXV4eyvvbh9WD=c6m=002WMheDyE3A@mail.gmail.com
2017-08-31 15:50:18 -04:00
Tom Lane 6708e447ef Clean up shm_mq cleanup.
The logic around shm_mq_detach was a few bricks shy of a load, because
(contrary to the comments for shm_mq_attach) all it did was update the
shared shm_mq state.  That left us leaking a bit of process-local
memory, but much worse, the on_dsm_detach callback for shm_mq_detach
was still armed.  That means that whenever we ultimately detach from
the DSM segment, we'd run shm_mq_detach again for already-detached,
possibly long-dead queues.  This accidentally fails to fail today,
because we only ever re-use a shm_mq's memory for another shm_mq, and
multiple detach attempts on the last such shm_mq are fairly harmless.
But it's gonna bite us someday, so let's clean it up.

To do that, change shm_mq_detach's API so it takes a shm_mq_handle
not the underlying shm_mq.  This makes the callers simpler in most
cases anyway.  Also fix a few places in parallel.c that were just
pfree'ing the handle structs rather than doing proper cleanup.

Back-patch to v10 because of the risk that the revenant shm_mq_detach
callbacks would cause a live bug sometime.  Since this is an API
change, it's too late to do it in 9.6.  (We could make a variant
patch that preserves API, but I'm not excited enough to do that.)

Discussion: https://postgr.es/m/8670.1504192177@sss.pgh.pa.us
2017-08-31 15:10:24 -04:00
Tom Lane 4b1dd62a25 Improve code coverage of select_parallel test.
Make sure that rescans of parallel indexscans are tested.
Per code coverage report.
2017-08-31 13:15:54 -04:00
Peter Eisentraut b5c75feca7 Remove to pre-8.2 coding convention for PG_MODULE_MAGIC
PG_MODULE_MAGIC has been around since 8.2, with 8.1 long since in EOL,
so remove the mention of #ifdef guards for compiling against pre-8.2
sources from the documentation.

Author: Daniel Gustafsson <daniel@yesql.se>
2017-08-30 22:40:24 -04:00
Tom Lane 04e9678614 Code review for nodeGatherMerge.c.
Comment the fields of GatherMergeState, and organize them a bit more
sensibly.  Comment GMReaderTupleBuffer more usefully too.  Improve
assorted other comments that were obsolete or just not very good English.

Get rid of the use of a GMReaderTupleBuffer for the leader process;
that was confusing, since only the "done" field was used, and that
in a way redundant with need_to_scan_locally.

In gather_merge_init, avoid calling load_tuple_array for
already-known-exhausted workers.  I'm not sure if there's a live bug there,
but the case is unlikely to be well tested due to timing considerations.

Remove some useless code, such as duplicating the tts_isempty test done by
TupIsNull.

Remove useless initialization of ps.qual, replacing that with an assertion
that we have no qual to check.  (If we did, the code would fail to check
it.)

Avoid applying heap_copytuple to a null tuple.  While that fails to crash,
it's confusing and it makes the code less legible not more so IMO.

Propagate a couple of these changes into nodeGather.c, as well.

Back-patch to v10, partly because of the possibility that the
gather_merge_init change is fixing a live bug, but mostly to keep
the branches in sync to ease future bug fixes.
2017-08-30 17:21:08 -04:00
Tom Lane 41b0dd987d Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes.  This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call).  That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized.  Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.

Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper.  ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.

As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.

Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 13:18:16 -04:00
Tom Lane 6c2c5bea3c Restore test case from a2b70c89ca.
Revert the reversion commits a20aac890 and 9b644745c.  In the wake of
commit 7df2c1f8d, we should get stable buildfarm results from this test;
if not, I'd like to know sooner not later.

Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 09:59:23 -04:00
Tom Lane 7df2c1f8da Force rescanning of parallel-aware scan nodes below a Gather[Merge].
The ExecReScan machinery contains various optimizations for postponing
or skipping rescans of plan subtrees; for example a HashAgg node may
conclude that it can re-use the table it built before, instead of
re-reading its input subtree.  But that is wrong if the input contains
a parallel-aware table scan node, since the portion of the table scanned
by the leader process is likely to vary from one rescan to the next.
This explains the timing-dependent buildfarm failures we saw after
commit a2b70c89c.

The established mechanism for showing that a plan node's output is
potentially variable is to mark it as depending on some runtime Param.
Hence, to fix this, invent a dummy Param (one that has a PARAM_EXEC
parameter number, but carries no actual value) associated with each Gather
or GatherMerge node, mark parallel-aware nodes below that node as dependent
on that Param, and arrange for ExecReScanGather[Merge] to flag that Param
as changed whenever the Gather[Merge] node is rescanned.

This solution breaks an undocumented assumption made by the parallel
executor logic, namely that all rescans of nodes below a Gather[Merge]
will happen synchronously during the ReScan of the top node itself.
But that's fundamentally contrary to the design of the ExecReScan code,
and so was doomed to fail someday anyway (even if you want to argue
that the bug being fixed here wasn't a failure of that assumption).
A follow-on patch will address that issue.  In the meantime, the worst
that's expected to happen is that given very bad timing luck, the leader
might have to do all the work during a rescan, because workers think
they have nothing to do, if they are able to start up before the eventual
ReScan of the leader's parallel-aware table scan node has reset the
shared scan state.

Although this problem exists in 9.6, there does not seem to be any way
for it to manifest there.  Without GatherMerge, it seems that a plan tree
that has a rescan-short-circuiting node below Gather will always also
have one above it that will short-circuit in the same cases, preventing
the Gather from being rescanned.  Hence we won't take the risk of
back-patching this change into 9.6.  But v10 needs it.

Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 09:29:55 -04:00
Tom Lane 2e70d6b5e9 Teach libpq to detect integer overflow in the row count of a PGresult.
Adding more than 1 billion rows to a PGresult would overflow its ntups and
tupArrSize fields, leading to client crashes.  It'd be desirable to use
wider fields on 64-bit machines, but because all of libpq's external APIs
use plain "int" for row counters, that's going to be hard to accomplish
without an ABI break.  Given the lack of complaints so far, and the general
pain that would be involved in using such huge PGresults, let's settle for
just preventing the overflow and reporting a useful error message if it
does happen.  Also, for a couple more lines of code we can increase the
threshold of trouble from INT_MAX/2 to INT_MAX rows.

To do that, refactor pqAddTuple() to allow returning an error message that
replaces the default assumption that it failed because of out-of-memory.

Along the way, fix PQsetvalue() so that it reports all failures via
pqInternalNotice().  It already did so in the case of bad field number,
but neglected to report anything for other error causes.

Because of the potential for crashes, this seems like a back-patchable
bug fix, despite the lack of field reports.

Michael Paquier, per a complaint from Igor Korot.

Discussion: https://postgr.es/m/CA+FnnTxyLWyjY1goewmJNxC==HQCCF4fKkoCTa9qR36oRAHDPw@mail.gmail.com
2017-08-29 15:18:01 -04:00
Robert Haas bf11e7ee2e Propagate sort instrumentation from workers back to leader.
Up until now, when parallel query was used, no details about the
sort method or space used by the workers were available; details
were shown only for any sorting done by the leader.  Fix that.

Commit 1177ab1dab forced the test case
added by commit 1f6d515a67 to run
without parallelism; now that we have this infrastructure, allow
that again, with a little tweaking to make it pass with and without
force_parallel_mode.

Robert Haas and Tom Lane

Discussion: http://postgr.es/m/CA+Tgmoa2VBZW6S8AAXfhpHczb=Rf6RqQ2br+zJvEgwJ0uoD_tQ@mail.gmail.com
2017-08-29 13:26:33 -04:00
Robert Haas 3452dc5240 Push tuple limits through Gather and Gather Merge.
If we only need, say, 10 tuples in total, then we certainly don't need
more than 10 tuples from any single process.  Pushing down the limit
lets workers exit early when possible.  For Gather Merge, there is
an additional benefit: a Sort immediately below the Gather Merge can
be done as a bounded sort if there is an applicable limit.

Robert Haas and Tom Lane

Discussion: http://postgr.es/m/CA+TgmoYa3QKKrLj5rX7UvGqhH73G1Li4B-EKxrmASaca2tFu9Q@mail.gmail.com
2017-08-29 13:16:55 -04:00
Tom Lane 95e28b7f0c Fix over-aggressive sanity check in misc_sanity.sql.
Fix thinko in commit 8be8510cf: it's okay to have dbid == 0 in normal
(non-pin) entries in pg_shdepend, because global objects such as
databases are entered that way.  The test would pass so long as it
was run in a cluster containing no databases/tablespaces owned by,
or granted to, roles other than the bootstrap superuser.  That's the
expected situation for "make check", but for "make installcheck", not
so much.

Reported by Ryan Murphy.

Discussion: https://postgr.es/m/CAHeEsBc6EQe0mxGBKDXAwJbntgfvoAd5MQC-5362SmC3Tng_6g@mail.gmail.com
2017-08-28 10:14:46 -04:00
Michael Meskes 04fbe0e451 Changed order of statements and added an additiona MSVC safeguard to make ecpg
thread test cases work on Windows.
2017-08-26 19:11:58 +02:00
Peter Eisentraut 2073c641b4 pg_test_timing: Some NLS fixes
The string "% of total" was marked by xgettext to be a c-format, but it
is actually not, so mark up the source to prevent that.

Compute the column widths of the final display dynamically based on the
translated strings, so that translations don't mess up the display
accidentally.
2017-08-26 09:25:19 -04:00
Michael Meskes a772624b1d Make setlocale in ECPG test cases thread aware on Windows.
Fix threaded test cases on Windows not to crash in setlocale() which can be
global or local to a thread on Windows.

Author: Christian Ullrich
2017-08-26 12:57:21 +02:00
Tom Lane aae62278e1 Fix locale dependency in new ecpg test case.
Force sorting in "C" locale so that the output ordering doesn't vary,
per buildfarm.

In passing, add missing .gitignore entries.

Discussion: https://postgr.es/m/0975f4bb-5dee-c33c-b719-3ce44026d397@chrullrich.net
2017-08-25 14:17:33 -04:00
Peter Eisentraut 99ce446ada pg_upgrade: Remove more dead code
related to 6ce6a61840

Reported-by: Christoph Berg <myon@debian.org>
2017-08-25 12:02:29 -04:00
Peter Eisentraut e86ac70d6e Message translatability fixes 2017-08-25 11:52:31 -04:00
Michael Meskes d22e9d5305 Implement DO CONTINUE action for ECPG WHENEVER statement.
Author: Vinayak Pokale
Reviewed-By: Masahiko Sawada
2017-08-25 15:17:29 +02:00
Tom Lane 3f4c7917b3 Code review for pushing LIMIT through subqueries.
Minor improvements for commit 1f6d515a6.  We do not need the (rather
expensive) test for SRFs in the targetlist, because since v10 any
such SRFs would appear in separate ProjectSet nodes.  Also, make the
code look more like the existing cases by turning it into a simple
recursion --- the argument that there might be some performance
benefit to contorting the code seems unfounded to me, especially since
any good compiler should turn the tail-recursion into iteration anyway.

Discussion: http://postgr.es/m/CADE5jYLuugnEEUsyW6Q_4mZFYTxHxaVCQmGAsF0yiY8ZDggi-w@mail.gmail.com
2017-08-25 09:05:17 -04:00
Andres Freund d36f7efb39 Add minimal regression test for blessed record type transfer.
Test that blessed records can be transferred through a TupleQueue and
correctly decoded by another backend.  While touching the file, make
sure that force_parallel_mode settings only cover relevant tests.

Author: Thomas Munro, editorialized by Andres Freund
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/20170823054644.efuzftxjpfi6wwqs%40alap3.anarazel.de
2017-08-24 17:42:49 -07:00
Andres Freund d7694fc148 Consolidate the function pointer types used by dshash.c.
Commit 8c0d7bafad introduced dshash with hash
and compare functions like DynaHash's, and also variants that take a user
data pointer instead of size.  Simplify the interface by merging them into
a single pair of function pointer types that take both size and a user data
pointer.

Since it is anticipated that memcmp and tag_hash behavior will be a common
requirement, provide wrapper functions dshash_memcmp and dshash_memhash that
conform to the new function types.

Author: Thomas Munro
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/20170823054644.efuzftxjpfi6wwqs%40alap3.anarazel.de
2017-08-24 17:01:36 -07:00
Andres Freund 4569715bd6 Fix unlikely shared memory leak after failure in dshash_create().
Tidy-up for commit 8c0d7bafad, based on a
complaint from Andres Freund.

Author: Thomas Munro
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/20170823054644.efuzftxjpfi6wwqs%40alap3.anarazel.de
2017-08-24 16:58:30 -07:00
Andres Freund 20fbf25533 Fix harmless thinko in dsa.c.
Commit 16be2fd100 added DSA_ALLOC_HUGE,
DSA_ALLOC_ZERO and DSA_ALLOC_NO_OOM which have the same numerical
values and meanings as the similarly named MCXT_... macros.  In one
place we accidentally used MCXT_ALLOC_NO_OOM when DSA_ALLOC_NO_OOM is
wanted, so tidy that up.

Author: Thomas Munro
Discussion: http://postgr.es/m/CAEepm=2AimHxVkkxnMfQvbZMkXy0uKbVa0-D38c5-qwrCm4CMQ@mail.gmail.com
Backpatch: 10, where dsa was introduced.
2017-08-24 15:07:40 -07:00
Stephen Frost 0cdc3e47be psql: Fix \gx when FETCH_COUNT is used
Set expanded output when requested through \gx in ExecQueryUsingCursor()
(used when FETCH_COUNT is set).

Discussion: https://www.postgresql.org/message-id/CB7A53AA-5645-4BDD-AB07-4D22CD9D8FF1%40gmx.net
Author: Tobias Bussmann
2017-08-24 16:20:50 -04:00
Peter Eisentraut 6ce6a61840 pg_upgrade: Remove dead code
Remove code meant for upgrading to a particular version of PostgreSQL
9.0.  Since pg_upgrade only supports upgrading to the current major
version, this code is no longer useful.
2017-08-24 15:29:35 -04:00
Peter Eisentraut fe7774144d Increase SCRAM salt length
The original value 12 was set based on RFC 5802 for SCRAM-SHA-1, but RFC
7677 for SCRAM-SHA-256 uses 16, so use that.  (This does not affect the
validity of already stored verifiers.)

Discussion: https://www.postgresql.org/message-id/flat/12cc9297-7e05-932f-d863-765e5626ead4%402ndquadrant.com
2017-08-24 14:04:28 -04:00
Tom Lane 1177ab1dab Make new regression test case parallel-safe, and improve its output.
The test case added by commit 1f6d515a6 fails on buildfarm members that
have force_parallel_mode turned on, because we currently don't report sort
performance details from worker processes back to the master.  To fix that,
just make the test table be temp rather than regular; that's a good idea
anyway to forestall any possible interference from auto-analyze.
(The restriction that workers can't access temp tables might go away
someday, but almost certainly not before the other thing gets fixed.)

Also, improve the test so that we retain as much as possible of the
EXPLAIN ANALYZE output.  This aids debugging failures, and might also
expose problems that the preceding version masked.

Discussion: http://postgr.es/m/CADE5jYLuugnEEUsyW6Q_4mZFYTxHxaVCQmGAsF0yiY8ZDggi-w@mail.gmail.com
2017-08-24 13:39:58 -04:00
Peter Eisentraut 27b89876c0 Fix up secondary expected files
for commit 237a0b87b1
2017-08-24 11:13:55 -04:00
Peter Eisentraut 6d242ee980 Update code comment for temporary replication slots
Reported-by: Alvaro Herrera <alvherre@2ndquadrant.com>
2017-08-23 14:59:25 -04:00
Peter Eisentraut 1e1b01cd16 Fix outdated comment
Author: Thomas Munro <thomas.munro@enterprisedb.com>
2017-08-23 14:19:35 -04:00
Peter Eisentraut 237a0b87b1 Improve plural handling in error message
This does not use the normal plural handling, because no numbers appear
in the actual message.
2017-08-23 13:56:59 -04:00