Commit Graph

4629 Commits

Author SHA1 Message Date
Stephen Frost 0f97c722bb Disable BLOB test in pg_dump TAP tests
Buildfarm member jacana appears to have an issue with running this
test.  It's not entirely clear to me why, but rather than try to
fight with it, just disable it for now.

None of the other tests try to write out from psql directly as
this test does, so it seems likely that the rest of the tests will
be fine (as they have been on numerous other systems).
2016-05-06 21:24:31 -04:00
Stephen Frost c778e27e13 Correct query in pg_dumpall:dumpRoles
We need to use a new branch due to the 9.5 addition of bypassrls
when adding in the clause to exclude pg_* roles from being dumped
by pg_dumpall.

Pointed out by Noah, patch by me.
2016-05-06 16:15:52 -04:00
Stephen Frost eccfeeb631 Remove MODULES_big from test_pg_dump
The Makefile for test_pg_dump shouldn't have a MODULES_big line
because there's no actual compiled bit for that extension.  Hopefully
this will fix the Windows buildfarm members which were complaining.

In passing, also add the 'prove_installcheck' bit to the pg_dump and
test_pg_dump Makefiles, to get the buildfarm members to actually run
those tests.
2016-05-06 15:26:57 -04:00
Tom Lane 73b9952e82 Improve pg_upgrade's report about failure to match up old and new tables.
Ordinarily, pg_upgrade shouldn't have any difficulty in matching up all
the relations it sees in the old and new databases.  If it does, however,
it just goes belly-up with a pretty unhelpful error message.  That seemed
fine as long as we expected the case never to occur in the wild, but
Alvaro reported that it had been seen in a database whose pg_largeobject
table had somehow acquired a TOAST table.  That doesn't quite seem like
a case that pg_upgrade actually needs to handle, but it would be good if
the report were more diagnosable.  Hence, extend the logic to print out
as much information as we can about the mismatch(es) before we quit.

In passing, improve the readability of get_rel_infos()'s data collection
query, which had suffered seriously from lets-not-bother-to-update-comments
syndrome, and generally was unnecessarily disrespectful to readers.

It could be argued that this is a bug fix, but given that we have so few
reports, I don't feel a need to back-patch; at least not before this has
baked awhile in HEAD.
2016-05-06 14:45:01 -04:00
Stephen Frost 6bd356c33a Add TAP tests for pg_dump
This TAP test suite will create a new cluster, populate it based on
the 'create_sql' values in the '%tests' hash, run all of the runs
defined in the '%pgdump_runs' hash, and then for each test in the
'%tests' hash, compare each run's output the the regular expression
defined for the test under the 'like' and 'unlike' functions, as
appropriate.

While this test suite covers a fair bit of ground (67% of pg_dump.c
and quite a bit of the other files in src/bin/pg_dump), there is
still quite a bit which remains to be added to provide better code
coverage.  Still, this is quite a bit better than we had, and has
found a few bugs already (note that the CREATE TRANSFORM test is
commented out, as it is currently failing).

Idea for using the TAP system from Tom, though all of the code is mine.
2016-05-06 14:06:50 -04:00
Stephen Frost e1b120a8cb Only issue LOCK TABLE commands when necessary
Reviewing the cases where we need to LOCK a given table during a dump,
it was pointed out by Tom that we really don't need to LOCK a table if
we are only looking to dump the ACL for it, or certain other
components.  After reviewing the queries run for all of the component
pieces, a list of components were determined to not require LOCK'ing
of the table.

This implements a check to avoid LOCK'ing those tables.

Initial complaint from Rushabh Lathia, discussed with Robert and Tom,
the patch is mine.
2016-05-06 14:06:50 -04:00
Stephen Frost 5d589993ca pg_dump performance and other fixes
Do not try to dump objects which do not have ACLs when only ACLs are
being requested.  This results in a significant performance improvement
as we can avoid querying for further information on these objects when
we don't need to.

When limiting the components to dump for an extension, consider what
components have been requested.  Initially, we incorrectly hard-coded
the components of the extension objects to dump, which would mean that
we wouldn't dump some components even with they were asked for and in
other cases we would dump components which weren't requested.

Correct defaultACLs to use 'dump_contains' instead of 'dump'.  The
defaultACL is considered a member of the namespace and should be
dumped based on the same set of components that the other objects in
the schema are, not based on what we're dumping for the namespace
itself (which might not include ACLs, if the namespace has just the
default or initial ACL).

Use DUMP_COMPONENT_ACL for from-initdb objects, to allow users to
change their ACLs, should they wish to.  This just extends what we
are doing for the pg_catalog namespace to objects which are not
members of namespaces.

Due to column ACLs being treated a bit differently from other ACLs
(they are actually reset to NULL when all privileges are revoked),
adjust the query which gathers column-level ACLs to consider all of
the ACL-relevant columns.
2016-05-06 14:06:50 -04:00
Stephen Frost 64d60c8bf0 Correct pg_dump WHERE clause for functions/aggregates
The query to grab the function/aggregate information is now joining
to pg_init_privs, so we can simplify (and correct) the WHERE clause
used to determine if a given function's ACL has changed from the
initial ACL on the function.

Bug found by Noah, patch by me.
2016-05-06 14:06:50 -04:00
Tom Lane 6b8b4e4d83 Fix pgbench's parsing of double values to notice trailing garbage.
Noted by Fabien Coelho, though this isn't exactly his proposed patch.
(The technique used here is borrowed from the zic sources.)
2016-05-06 11:08:48 -04:00
Tom Lane 9515299485 Improve handling of numeric-valued variables in pgbench.
The previous coding always stored variable values as strings, doing
conversion on-the-fly when a numeric value was needed or a number was to be
assigned.  This was a bit inefficient and risked loss of precision for
floating-point values.  The precision aspect had been hacked around by
printing doubles in "%.18e" format, which is ugly and has machine-dependent
results.  Instead, arrange to preserve an assigned numeric value in the
original binary numeric format, converting to string only when and if
needed.  When we do need to convert a double to string, convert in "%g"
format with DBL_DIG precision, which is the standard way to do it and
produces the least surprising results in most cases.

The implementation supports storing both a string value and a numeric
value for any one variable, with lazy conversion between them.  I also
arranged for lazy re-sorting of the variable array when new variables are
added.  That was mainly to allow a clean refactoring of putVariable()
into two levels of subroutine, but it may allow us to save a few sorts.

Discussion: <9188.1462475559@sss.pgh.pa.us>
2016-05-06 11:01:05 -04:00
Dean Rasheed 9b66aa006f Fix psql's \ev and \sv commands so that they handle view reloptions.
Commit 8eb6407aae added support for
editing and showing view definitions, but neglected to account for
view options such as security_barrier and WITH CHECK OPTION which are
not returned by pg_get_viewdef() and so need special handling.

Author: Dean Rasheed
Reviewed-by: Peter Eisentraut
Discussion: http://www.postgresql.org/message-id/CAEZATCWZjCgKRyM-agE0p8ax15j9uyQoF=qew7D2xB6cF76T8A@mail.gmail.com
2016-05-06 12:48:27 +01:00
Dean Rasheed 93a8c6fd6c Move and rename fmtReloptionsArray().
Move fmtReloptionsArray() from pg_dump.c to string_utils.c so that it
is available to other frontend code. In particular psql's \ev and \sv
commands need it to handle view reloptions. Also rename the function
to appendReloptionsArray(), which is a more accurate description of
what it does.

Author: Dean Rasheed
Reviewed-by: Peter Eisentraut
Discussion: http://www.postgresql.org/message-id/CAEZATCWZjCgKRyM-agE0p8ax15j9uyQoF=qew7D2xB6cF76T8A@mail.gmail.com
2016-05-06 12:45:36 +01:00
Tom Lane 7a622b2731 Rename pgbench min/max to least/greatest, and fix handling of double args.
These functions behave like the backend's least/greatest functions,
not like min/max, so the originally-chosen names invite confusion.
Per discussion, rename to least/greatest.

I also took it upon myself to make them return double if any input is
double.  The previous behavior of silently coercing all inputs to int
surely does not meet the principle of least astonishment.

Copy-edit some of the other new functions' documentation, too.
2016-05-05 14:51:00 -04:00
Andrew Dunstan 0fb54de9aa Support building with Visual Studio 2015
Adjust the way we detect the locale. As a result the minumum Windows
version supported by VS2015 and later is Windows Vista. Add some tweaks
to remove new compiler warnings. Remove documentation references to the
now obsolete msysGit.

Michael Paquier, somewhat edited by me, reviewed by Christian Ullrich.

Backpatch to 9.5
2016-04-29 08:09:07 -04:00
Peter Eisentraut 3019f432d6 pg_dump: Message style improvements
forgotten in b6dacc173b
2016-04-26 21:37:06 -04:00
Peter Eisentraut b6dacc173b pg_dump: Message style improvements 2016-04-25 17:16:59 -04:00
Peter Eisentraut 63417b4b2e Update GETTEXT_FILES after config and controldata refactoring 2016-04-24 20:58:11 -04:00
Robert Haas b4e0f18382 Add pg_dump support for the new PARALLEL option for aggregates.
This was an oversight in commit 41ea0c2376.

Fabrízio de Royes Mello, per a report from Tushar Ahuja
2016-04-20 23:06:06 -04:00
Tom Lane 9603a32594 Avoid code duplication in \crosstabview.
In commit 6f0d6a507 I added a duplicate copy of psqlscanslash's identifier
downcasing code, but actually it's not hard to split that out as a callable
subroutine and avoid the duplication.
2016-04-17 11:37:58 -04:00
Peter Eisentraut c313687673 psql: Add new gettext trigger 2016-04-15 20:23:41 -04:00
Tom Lane 6f0d6a5078 Rethink \crosstabview's argument parsing logic.
\crosstabview interpreted its arguments in an unusual way, including
doing case-insensitive matching of unquoted column names, which is
surely not the right thing.  Rip that out in favor of doing something
equivalent to the dequoting/case-folding rules used by other psql
commands.  To keep it simple, change the syntax so that the optional
sort column is specified as a separate argument, instead of the
also-quite-unusual syntax that attached it to the colH argument with
a colon.

Also, rework the error messages to be closer to project style.
2016-04-14 22:54:31 -04:00
Tom Lane 6cead413bb Fix pg_dump so pg_upgrade'ing an extension with simple opfamilies works.
As reported by Michael Feld, pg_upgrade'ing an installation having
extensions with operator families that contain just a single operator class
failed to reproduce the extension membership of those operator families.
This caused no immediate ill effects, but would create problems when later
trying to do a plain dump and restore, because the seemingly-not-part-of-
the-extension operator families would appear separately in the pg_dump
output, and then would conflict with the families created by loading the
extension.  This has been broken ever since extensions were introduced,
and many of the standard contrib extensions are affected, so it's a bit
astonishing nobody complained before.

The cause of the problem is a perhaps-ill-considered decision to omit
such operator families from pg_dump's output on the grounds that the
CREATE OPERATOR CLASS commands could recreate them, and having explicit
CREATE OPERATOR FAMILY commands would impede loading the dump script into
pre-8.3 servers.  Whatever the merits of that decision when 8.3 was being
written, it looks like a poor tradeoff now.  We can fix the pg_upgrade
problem simply by removing that code, so that the operator families are
dumped explicitly (and then will be properly made to be part of their
extensions).

Although this fixes the behavior of future pg_upgrade runs, it does nothing
to clean up existing installations that may have improperly-linked operator
families.  Given the small number of complaints to date, maybe we don't
need to worry about providing an automated solution for that; anyone who
needs to clean it up can do so with manual "ALTER EXTENSION ADD OPERATOR
FAMILY" commands, or even just ignore the duplicate-opfamily errors they
get during a pg_restore.  In any case we need this fix.

Back-patch to all supported branches.

Discussion: <20228.1460575691@sss.pgh.pa.us>
2016-04-13 18:58:14 -04:00
Tom Lane 7a5f8b5c59 Improve coding of column-name parsing in psql's new crosstabview.c.
Coverity complained about this code, not without reason because it was
rather messy.  Adjust it to not scribble on the passed string; that adds
one malloc/free cycle per column name, which is going to be insignificant
in context.  We can actually const-ify both the string argument and the
PGresult.

Daniel Verité, with some further cleanup by me
2016-04-12 12:52:42 -04:00
Tom Lane 074050f16a pg_dump: add missing "destroyPQExpBuffer(query)" in dumpForeignServer().
Coverity complained about this resource leak (why now, I don't know,
since it's been like that a long time).  Our general policy in pg_dump
is that PQExpBuffers are worth cleaning up, so do it here too.  But
don't bother with a back-patch, because it seems unlikely that very
many databases contain enough FOREIGN SERVER objects to notice.
2016-04-11 00:00:08 -04:00
Alvaro Herrera c09b18f21c Support \crosstabview in psql
\crosstabview is a completely different way to display results from a
query: instead of a vertical display of rows, the data values are placed
in a grid where the column and row headers come from the data itself,
similar to a spreadsheet.

The sort order of the horizontal header can be specified by using
another column in the query, and the vertical header determines its
ordering from the order in which they appear in the query.

This only allows displaying a single value in each cell.  If more than
one value correspond to the same cell, an error is thrown.  Merging of
values can be done in the query itself, if necessary.  This may be
revisited in the future.

Author: Daniel Verité
Reviewed-by: Pavel Stehule, Dean Rasheed
2016-04-08 20:23:18 -03:00
Stephen Frost 293007898d Reserve the "pg_" namespace for roles
This will prevent users from creating roles which begin with "pg_" and
will check for those roles before allowing an upgrade using pg_upgrade.

This will allow for default roles to be provided at initdb time.

Reviews by José Luis Tallón and Robert Haas
2016-04-08 16:56:27 -04:00
Stephen Frost fa6075e551 Fix improper usage of 'dump' bitmap
Now that 'dump' is a bitmap, we can't simply set it to 'true'.

Noticed while debugging the prior issue.
2016-04-08 16:30:02 -04:00
Stephen Frost 689f9a0588 In dumpTable, re-instate the skipping logic
Pretty sure I removed this based on some incorrect thinking that it was
no longer possible to reach this point for a table which will not be
dumped, but that's clearly wrong.

Pointed out on IRC by Erik Rijkers.
2016-04-08 15:00:44 -04:00
Teodor Sigaev 8b99edefca Revert CREATE INDEX ... INCLUDING ...
It's not ready yet, revert two commits
690c543550 - unstable test output
386e3d7609 - patch itself
2016-04-08 21:52:13 +03:00
Tom Lane 34c33a1f00 Add BSD authentication method.
Create a "bsd" auth method that works the same as "password" so far as
clients are concerned, but calls the BSD Authentication service to
check the password.  This is currently only available on OpenBSD.

Marisa Emerson, reviewed by Thomas Munro
2016-04-08 13:52:06 -04:00
Teodor Sigaev 386e3d7609 CREATE INDEX ... INCLUDING (column[, ...])
Now indexes (but only B-tree for now) can contain "extra" column(s) which
doesn't participate in index structure, they are just stored in leaf
tuples. It allows to use index only scan by using single index instead
of two or more indexes.

Author: Anastasia Lubennikova with minor editorializing by me
Reviewers: David Rowley, Peter Geoghegan, Jeff Janes
2016-04-08 19:45:59 +03:00
Robert Haas 25fe8b5f1a Add a 'parallel_degree' reloption.
The code that estimates what parallel degree should be uesd for the
scan of a relation is currently rather stupid, so add a parallel_degree
reloption that can be used to override the planner's rather limited
judgement.

Julien Rouhaud, reviewed by David Rowley, James Sewell, Amit Kapila,
and me.  Some further hacking by me.
2016-04-08 11:14:56 -04:00
Stephen Frost 23f34fa4ba In pg_dump, include pg_catalog and extension ACLs, if changed
Now that all of the infrastructure exists, add in the ability to
dump out the ACLs of the objects inside of pg_catalog or the ACLs
for objects which are members of extensions, but only if they have
been changed from their original values.

The original values are tracked in pg_init_privs.  When pg_dump'ing
9.6-and-above databases, we will dump out the ACLs for all objects
in pg_catalog and the ACLs for all extension members, where the ACL
has been changed from the original value which was set during either
initdb or CREATE EXTENSION.

This should not change dumps against pre-9.6 databases.

Reviews by Alexander Korotkov, Jose Luis Tallon
2016-04-06 21:45:32 -04:00
Stephen Frost d217b2c360 In pg_dump, split "dump" into "dump" and "dump_contains"
Historically, the "dump" component of the namespace has been used
to decide if the objects inside of the namespace should be dumped
also.  Given that "dump" is now a bitmask and may be partial, and
we may want to dump out all components of the namespace object but
only some of the components of objects contained in the namespace,
create a "dump_contains" bitmask which will represent what components
of the objects inside of a namespace should be dumped out.

No behavior change here, but in preparation for a change where we
will dump out just the ACLs of objects in pg_catalog, but we might
not dump out the ACL of the pg_catalog namespace itself (for instance,
when it hasn't been changed from the value set at initdb time).

Reviews by Alexander Korotkov, Jose Luis Tallon
2016-04-06 21:45:32 -04:00
Stephen Frost a9f0e8e5a2 In pg_dump, use a bitmap to represent what to include
pg_dump has historically used a simple boolean 'dump' value to indicate
if a given object should be included in the dump or not.  Instead, use
a bitmap which breaks down the components of an object into their
distinct pieces and use that bitmap to only include the components
requested.

This does not include any behavioral change, but is in preperation for
the change to dump out just ACLs for objects in pg_catalog.

Reviews by Alexander Korotkov, Jose Luis Tallon
2016-04-06 21:45:32 -04:00
Stephen Frost 6c268df127 Add new catalog called pg_init_privs
This new catalog holds the privileges which the system was
initialized with at initdb time, along with any permissions set
by extensions at CREATE EXTENSION time.  This allows pg_dump
(and any other similar use-cases) to detect when the privileges
set on initdb-created or extension-created objects have been
changed from what they were set to at initdb/extension-creation
time and handle those changes appropriately.

Reviews by Alexander Korotkov, Jose Luis Tallon
2016-04-06 21:45:32 -04:00
Peter Eisentraut 3b3fcc4eea pg_dump: Add table qualifications to some tags
Some object types have names that are only unique for one table.  But
for those we generally didn't put the table name into the dump TOC tag.
So it was impossible to identify these objects if the same name was used
for multiple tables.  This affects policies, column defaults,
constraints, triggers, and rules.

Fix by adding the table name to the TOC tag, so that it now reads
"$schema $table $object".

Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2016-04-06 12:13:11 -04:00
Simon Riggs 3fe3511d05 Generic Messages for Logical Decoding
API and mechanism to allow generic messages to be inserted into WAL that are
intended to be read by logical decoding plugins. This commit adds an optional
new callback to the logical decoding API.

Messages are either text or bytea. Messages can be transactional, or not, and
are identified by a prefix to allow multiple concurrent decoding plugins.

(Not to be confused with Generic WAL records, which are intended to allow crash
recovery of extensible objects.)

Author: Petr Jelinek and Andres Freund
Reviewers: Artur Zakirov, Tomas Vondra, Simon Riggs
Discussion: 5685F999.6010202@2ndquadrant.com
2016-04-06 10:05:41 +01:00
Tom Lane 2bbe9112ae Add a \gexec command to psql for evaluation of computed queries.
\gexec executes the just-entered query, like \g, but instead of printing
the results it takes each field as a SQL command to send to the server.
Computing a series of queries to be executed is a fairly common thing,
but up to now you always had to resort to kluges like writing the queries
to a file and then inputting the file.  Now it can be done with no
intermediate step.

The implementation is fairly straightforward except for its interaction
with FETCH_COUNT.  ExecQueryUsingCursor isn't capable of being called
recursively, and even if it were, its need to create a transaction
block interferes unpleasantly with the desired behavior of \gexec after
a failure of a generated query (i.e., that it can continue).  Therefore,
disable use of ExecQueryUsingCursor when doing the master \gexec query.
We can still apply it to individual generated queries, however, and there
might be some value in doing so.

While testing this feature's interaction with single-step mode, I (tgl) was
led to conclude that SendQuery needs to recognize SIGINT (cancel_pressed)
as a negative response to the single-step prompt.  Perhaps that's a
back-patchable bug fix, but for now I just included it here.

Corey Huinker, reviewed by Jim Nasby, Daniel Vérité, and myself
2016-04-04 15:25:16 -04:00
Tom Lane 3cc38ca7d2 Add psql \errverbose command to see last server error at full verbosity.
Often, upon getting an unexpected error in psql, one's first wish is that
the verbosity setting had been higher; for example, to be able to see the
schema-name field or the server code location info.  Up to now the only way
has been to adjust the VERBOSITY variable and repeat the failing query.
That's a pain, and it doesn't work if the error isn't reproducible.

This commit adds a psql feature that redisplays the most recent server
error at full verbosity, without needing to make any variable changes or
re-execute the failed command.  We just need to hang onto the latest error
PGresult in case the user executes \errverbose, and then apply libpq's
new PQresultVerboseErrorMessage() function to it.  This will consume
some trivial amount of psql memory, but otherwise the cost when the
feature isn't used should be negligible.

Alex Shulgin, reviewed by Daniel Vérité, some improvements by me
2016-04-03 12:29:55 -04:00
Alvaro Herrera 5cb882675a pgbench: Remove unused parameter
For some reason this parameter was introduced as unused in 3da0dfb4b1,
and has never been used for anything.  Remove it.

Author: Fabien Coelho
2016-04-01 17:11:18 -03:00
Teodor Sigaev 65578341af Add Generic WAL interface
This interface is designed to give an access to WAL for extensions which
could implement new access method, for example. Previously it was
impossible because restoring from custom WAL would need to access system
catalog to find a redo custom function. This patch suggests generic way
to describe changes on page with standart layout.

Bump XLOG_PAGE_MAGIC because of new record type.

Author: Alexander Korotkov with a help of Petr Jelinek, Markus Nullmeier and
	minor editorization by my
Reviewers: Petr Jelinek, Alvaro Herrera, Teodor Sigaev, Jim Nasby,
	Michael Paquier
2016-04-01 12:21:48 +03:00
Robert Haas 5fe5a2cee9 Allow aggregate transition states to be serialized and deserialized.
This is necessary infrastructure for supporting parallel aggregation
for aggregates whose transition type is "internal".  Such values
can't be passed between cooperating processes, because they are
just pointers.

David Rowley, reviewed by Tomas Vondra and by me.
2016-03-29 15:04:05 -04:00
Alvaro Herrera a1c935d3b7 pgbench: allow a script weight of zero
This refines the previous weight range and allows a script to be "turned
off" by passing a zero weight, which is useful when scripting multiple
pgbench runs.

I did not apply the suggested warning when a script uses zero weight; we
use the principle elsewhere that if there's nothing to be done, do
nothing quietly.

Adjust docs accordingly.

Author: Jeff Janes, Fabien Coelho
2016-03-29 14:47:10 -03:00
Robert Haas ad9566470b pgbench: Remove \setrandom.
You can now do the same thing via \set using the appropriate function,
either random(), random_gaussian(), or random_exponential(), depending
on the desired distribution.  This is not backward-compatible, but per
discussion, it's worth it to avoid having the old syntax hang around
forever.

Fabien Coelho, reviewed by Michael Paquier, and adjusted by me.
2016-03-29 12:08:49 -04:00
Tom Lane 656ee84890 Fix portability issues in 86c43f4e22.
INT64_MIN/MAX should be spelled PG_INT64_MIN/MAX, per well established
convention in our sources.  Less obviously, a symbol named DOUBLE causes
problems on Windows builds, so rename that to DOUBLE_CONST; and rename
INTEGER to INTEGER_CONST for consistency.

Also, get rid of incorrect/obsolete hand-munging of yycolumn, and fix
the grammar for float constants to handle expected cases such as ".1".

First two items by Michael Paquier, second two by me.
2016-03-29 00:53:53 -04:00
Robert Haas 86c43f4e22 pgbench: Support double constants and functions.
The new functions are pi(), random(), random_exponential(),
random_gaussian(), and sqrt().  I was worried that this would be
slower than before, but, if anything, it actually turns out to be
slightly faster, because we now express the built-in pgbench scripts
using fewer lines; each \setrandom can be merged into a subsequent
\set.

Fabien Coelho
2016-03-28 20:45:57 -04:00
Tom Lane 1f4e9da624 Sync tzload() and tzparse() APIs with IANA release tzcode2016c.
This brings us a bit closer to matching upstream, but since it affects
files outside src/timezone/, we might choose not to back-patch it.
Hence keep it separate from the main update patch.
2016-03-28 17:19:29 -04:00
Alvaro Herrera cad3edef4f pg_rewind: Improve internationalization
This is mostly cosmetic since two of the three changes are debug
messages, and the third one is just a progress indicator.

Author: Michaël Paquier
2016-03-28 14:33:00 -03:00
Alvaro Herrera 37732a2555 Fix minor leak in pg_dump for ACCESS METHOD.
Bug reported by Coverity.

Author: Michaël Paquier
2016-03-28 14:27:41 -03:00
Teodor Sigaev 559e7a0a6d psql tab-complete for CREATE/DROP ACCESS METHOD
Alexander Korotkov
2016-03-28 19:32:13 +03:00
Teodor Sigaev dabd255d58 Fix comment in pg_dump.
It was missed in 473b932870,
CREATE ACCESS METHOD

Alexander Korotkov
2016-03-28 19:17:28 +03:00
Andres Freund 408f043853 pg_rewind: fsync target data directory.
Previously pg_rewind did not fsync any files. That's problematic, given
that the target directory is modified. If the database was started
afterwards, 2ce439f33 luckily already caused the data directory to be
synced to disk at postmaster startup; reducing the scope of the problem.

To fix, use initdb -S, at the end of the pg_rewind run. It doesn't seem
worthwhile to duplicate the code into pg_rewind, and initdb -S is
already used that way by pg_upgrade.

Reported-By: Andres Freund
Author: Michael Paquier, somewhat edited by me
Discussion: 20160310034352.iuqgvpmg5qmnxtkz@alap3.anarazel.de
    CAB7nPqSytVG1o4S3S2pA1O=692ekurJ+fckW2PywEG3sNw54Ow@mail.gmail.com
Backpatch: 9.5, where pg_rewind was introduced
2016-03-27 23:46:25 +02:00
Andres Freund a6c845946d pg_rewind: Close backup_label file descriptor.
This was a relatively harmless leak, as createBackupLabel() is only
called once per pg_rewind invocation.

Author: Michael Paquier
Reported-By: Michael Paquier
Discussion: CAB7nPqRnOw30gOXe2_SPLjh37bgm4V+txbYAPwoXb97nGQ297w@mail.gmail.com
Backpatch: 9.5, where pg_rewind was introduced
2016-03-27 22:48:31 +02:00
Tom Lane 7caaeaf360 Link libpq after libpgfeutils to satisfy Windows linker.
Some of the non-MSVC Windows buildfarm members seem to need this to avoid
getting "undefined symbol" errors on libpgfeutils' references to libpq.
I could understand that if libpq were a static library, but surely it is
not?  Oh well, at least the extra reference is no more harmful than it is
for libpgcommon or libpgport.
2016-03-24 20:45:31 -04:00
Tom Lane c1156411ad Move psql's psqlscan.l into src/fe_utils.
This completes (at least for now) the project of getting rid of ad-hoc
linkages among the src/bin/ subdirectories.  Everything they share is now
in src/fe_utils/ and is included from a static library at link time.

A side benefit is that we can restore the FLEX_NO_BACKUP check for
psqlscanslash.l.  We might need to think of another way to do that check
if we ever need to build two lexers with that property in the same source
directory, but there's no foreseeable reason to need that.
2016-03-24 20:28:47 -04:00
Tom Lane d65bea26a8 Move psql's print.c and mbprint.c into src/fe_utils.
Just turning the crank ...
2016-03-24 18:27:28 -04:00
Tom Lane 588d963b00 Create src/fe_utils/, and move stuff into there from pg_dump's dumputils.
Per discussion, we want to create a static library and put the stuff into
it that until now has been shared across src/bin/ directories by ad-hoc
methods like symlinking a source file.  This commit creates the library and
populates it with a couple of files that contain the widely-useful portions
of pg_dump's dumputils.c file.  dumputils.c survives, because it has some
stuff that didn't seem appropriate for fe_utils, but it's significantly
smaller and is no longer referenced from any other directory.

Follow-on patches will move more stuff into fe_utils.

The Mkvcbuild.pm hacking here is just a best guess; we'll see how the
buildfarm likes it.
2016-03-24 15:55:57 -04:00
Alvaro Herrera 473b932870 Support CREATE ACCESS METHOD
This enables external code to create access methods.  This is useful so
that extensions can add their own access methods which can be formally
tracked for dependencies, so that DROP operates correctly.  Also, having
explicit support makes pg_dump work correctly.

Currently only index AMs are supported, but we expect different types to
be added in the future.

Authors: Alexander Korotkov, Petr Jelínek
Reviewed-By: Teodor Sigaev, Petr Jelínek, Jim Nasby
Commitfest-URL: https://commitfest.postgresql.org/9/353/
Discussion: https://www.postgresql.org/message-id/CAPpHfdsXwZmojm6Dx+TJnpYk27kT4o7Ri6X_4OSWcByu1Rm+VA@mail.gmail.com
2016-03-23 23:01:35 -03:00
Tom Lane 2c6af4f442 Move keywords.c/kwlookup.c into src/common/.
Now that we have src/common/ for code shared between frontend and backend,
we can get rid of (most of) the klugy ways that the keyword table and
keyword lookup code were formerly shared between different uses.
This is a first step towards a more general plan of getting rid of
special-purpose kluges for sharing code in src/bin/.

I chose to merge kwlookup.c back into keywords.c, as it once was, and
always has been so far as keywords.h is concerned.  We could have
kept them separate, but there is noplace that uses ScanKeywordLookup
without also wanting access to the backend's keyword list, so there
seems little point.

ecpg is still a bit weird, but at least now the trickiness is documented.

I think that the MSVC build script should require no adjustments beyond
what's done here ... but we'll soon find out.
2016-03-23 20:22:08 -04:00
Tom Lane b283096534 Allow the delay in psql's \watch command to be a fractional second.
Instead of just "2" seconds, allow eg. "2.5" seconds.  Per request
from Alvaro Herrera.  No docs change since the docs didn't say you
couldn't do this already.
2016-03-21 18:34:18 -04:00
Tom Lane dea2b5960a Improve header output from psql's \watch command.
Include the \pset title string if there is one, and shorten the prefab
part of the header to be "timestamp (every Ns)".  Per suggestion by
David Johnston.

Michael Paquier and Tom Lane
2016-03-21 18:18:13 -04:00
Tom Lane b6afae71aa Use %option bison-bridge in psql/pgbench lexers.
The point of this change is to use %pure-parser in pgbench's exprparse.y.
The immediate reason is that it turns out very ancient versions of bison
have a bug with the combination of a reentrant lexer and non-reentrant
parser.  We could consider dropping support for such ancient bisons; but
considering that we might well need exprparse.y to be reentrant some day,
it seems better to make it so right now than to move the portability
goalposts.  (AFAICT there's no particular performance consequence to this
change, either, so there's no good reason not to do it.)

Now, %pure-parser assumes that the called lexer is built with %option
bison-bridge.  Because we're assuming bitwise compatibility of yyscan_t
(yyguts_t) data structures among all the psql/pgbench lexers, that
requirement propagates back to psql's lexers as well.  But it's just a
few lines of change on that side too; and if psqlscan.l is to set the
baseline for a possibly-large family of lexers, it should err on the
side of including not omitting useful features.
2016-03-20 21:59:03 -04:00
Tom Lane 68ab8e8ba4 SQL commands in pgbench scripts are now ended by semicolons, not newlines.
To allow multiline SQL commands in scripts, adopt the same rules psql uses
to decide what is the end of a SQL command, to wit, an unquoted semicolon
not encased in parentheses.  Do this by importing the same flex lexer that
psql uses, since coping with stuff like dollar-quoted literals is hard to
get right without going the full nine yards.

This makes use of the infrastructure added in commit 0ea9efbe9e to
support independently-written flex lexers scanning the same PsqlScanState
input-buffer data structure.  Since that infrastructure isn't very
friendly to ad-hoc parsing code such as strtok(), improve exprscan.l
so that it can parse either whitespace-separated words or expression
tokens, on demand, and rewrite pgbench.c's backslash-command parsing
code to always use the lexer to fetch tokens.

It's still the case that pgbench backslash commands extend to the end
of the line, no more and no less.  That could be changed in a fairly
localized way now, and there was some interest in doing so, but it
seems like material for a separate patch.

In passing, make some marginal cleanups in syntax error reporting,
const-ify a few data structures that could use it, and run some of
this code through pgindent.

I can't tell whether the MSVC build scripts need to be taught explicitly
about the changes here or not, but the buildfarm will soon tell us.

Kyotaro Horiguchi and Tom Lane
2016-03-20 12:58:51 -04:00
Tom Lane 429ee5a822 Make pgbench's expression lexer reentrant.
This is a necessary preliminary step for making it play with psqlscan.l
given the way I set up the lexer input-buffer sharing mechanism in commit
0ea9efbe9e.

I've not tried to make it *actually* reentrant; there's still some static
variables laying about.  But flex thinks it's reentrant, and that's what
counts.

In support of that, fix exprparse.y to pass through the yyscan_t from the
caller.  Also do some minor code beautification, like not casting away
const.
2016-03-19 16:35:41 -04:00
Alvaro Herrera 1038bc91ca pgbench: Silence new compiler warnings
The original coding in 7bafffea64 and previous wasn't all that great
anyway.

Reported by Jeff Janes and Tom Lane
2016-03-19 16:16:39 -03:00
Tom Lane 21c8ee7946 Sync backend/parser/scan.l with bin/psql/psqlscan.l.
Make some minor formatting adjustments to make it easier to diff these
files and see that they indeed implement the same flex rules (at least
to the extent that we want them to be the same).

(Someday it'd be nice to make ecpg's pgc.l more easily diff'able too,
but today is not that day.)

Also run relevant parts of these files and psqlscanslash.l through
pgindent.

No actual behavioral changes here, just obsessive neatnik-ism.
2016-03-19 14:36:22 -04:00
Alvaro Herrera 7bafffea64 pgbench: Allow changing weights for scripts
Previously, all scripts had the same probability of being chosen when
multiple of them were specified via -b, -f, -N, -S.  With this commit,
-b and -f now search for an "@" in the script name and use the integer
found after it as the drawing probability for that script.

(One disadvantage is that if you have script whose names contain @, you
are now forced to specify "@1" at the end; otherwise the name's @ is
confused with a weight separator.  We don't expect many pgbench script
with @ in their names in the wild, so this shouldn't be too serious a
problem.)

While at it, rework the interface between addScript, process_file,
process_builtin, and findBuiltin.  It had gotten a bit out of hand with
recent commits.

Author: Fabien Coelho
Reviewed-By: Andres Freund, Robert Haas, Álvaro Herrera, Michaël Paquier
Discussion: http://www.postgresql.org/message-id/alpine.DEB.2.10.1603160721240.1666@sto
2016-03-19 12:32:42 -03:00
Tom Lane b46d9beb65 With ancient gcc, skip pg_attribute_printf() on function pointer.
Buildfarm results show that the ability to attach pg_attribute_printf
decoration to a function pointer appeared somewhere between gcc 2.95.3
and gcc 4.0.1.  Guess that it was there in 4.0.
2016-03-19 10:59:20 -04:00
Tom Lane ff0a7e6167 Use yylex_init not yylex_init_extra().
Older versions of flex don't have the latter.  Per buildfarm.
2016-03-19 01:02:18 -04:00
Tom Lane a3e39f8363 Suppress FLEX_NO_BACKUP check for psqlscanslash.l.
The existing infrastructure for FLEX_NO_BACKUP doesn't work reliably
when two lexers are built in parallel in the same directory.  We can
probably fix that, but as a short-term workaround, just don't make
the check for psqlscanslash.l.

Per buildfarm.
2016-03-19 00:43:46 -04:00
Tom Lane 0ea9efbe9e Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries.  Moreover, other programs can supply their own lexers
for backslash commands of their own choosing.  A follow-on patch will
use this in pgbench.

The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers.  The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer.  This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file.  (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)

Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states.  I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.

I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.

Kyotaro Horiguchi and Tom Lane
2016-03-19 00:24:55 -04:00
Tom Lane 27199058d9 Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables.  While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure.  If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files.  (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)

Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l.  I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.

Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead.  This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit.  (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-18 21:22:02 -04:00
Peter Eisentraut b555ed8102 Merge wal_level "archive" and "hot_standby" into new name "replica"
The distinction between "archive" and "hot_standby" existed only because
at the time "hot_standby" was added, there was some uncertainty about
stability.  This is now a long time ago.  We would like to move forward
with simplifying the replication configuration, but this distinction is
in the way, because a primary server cannot tell (without asking a
standby or predicting the future) which one of these would be the
appropriate level.

Pick a new name for the combined setting to make it clearer that it
covers all (non-logical) backup and replication uses.  The old values
are still accepted but are converted internally.

Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: David Steele <david@pgmasters.net>
2016-03-18 23:56:03 +01:00
Tom Lane 4e1d2a1708 Decouple psqlscan.l from surrounding program.
Remove assorted external references from psqlscan.l in preparation for
making it usable by other frontend programs.  This mostly involves
getting rid of direct calls to psql_error() and GetVariable() in favor
of introducing a callback-functions struct to encapsulate variable
fetching and error printing.  In addition, pass the current encoding
and standard-strings status as additional parameters to psql_scan_setup
instead of looking directly at "pset" or calling additional functions.

I did not bother to change some references to psql_error that are in
functions that will soon migrate to a psql-specific backslash-command
lexer.  Other than that, this version of psqlscan.l is capable of
compiling standalone.  It still depends on assorted src/common functions
as well as some encoding-related libpq functions, but we expect that
all programs using it will be happy with those dependencies.

Kyotaro Horiguchi, somewhat editorialized on by me
2016-03-18 15:05:59 -04:00
Tom Lane 3422feccca Clean up some misplaced #includes.
Random .h files have no business including postgres-fe.h (or postgres.h).
If that wasn't the first #include done by the calling .c file, it's the
.c file that's broken.  Noted while prepping Kyotaro Horiguchi's psql
lexer refactoring patch.
2016-03-18 13:43:17 -04:00
Tom Lane 47211af17a Fix "pg_bench -C -M prepared".
This didn't work because when we dropped and re-established a database
connection, we did not bother to reset session-specific state such as
the statements-are-prepared flags.

The st->prepared[] array certainly needs to be flushed, and I cleared a
couple of other fields as well that couldn't possibly retain meaningful
state for a new connection.

In passing, fix some bogus comments and strange field order choices.

Per report from Robins Tharakan.
2016-03-16 23:18:07 -04:00
Robert Haas 3aff33aa68 Fix typos.
Oskari Saarenmaa
2016-03-15 18:06:11 -04:00
Magnus Hagander a1aa8b7ea0 Fix order of MemSet arguments
Noted by Tomas Vondra
2016-03-13 13:11:06 +01:00
Tom Lane fc7a9dfddb Get rid of scribbling on a const variable in psql's print.c.
Commit a2dabf0e1d had the bright idea that it could modify a "const"
global variable if it merely casted away const from a pointer.  This does
not work on platforms where the compiler puts "const" variables into
read-only storage.  Depressingly, we evidently have no such platforms in
our buildfarm ... an oversight I have now remedied.  (The one platform
that is known to catch this is recent OS X with -fno-common.)

Per report from Chris Ruprecht.  Back-patch to 9.5 where the bogus
code was introduced.
2016-03-12 18:16:24 -05:00
Robert Haas 7087166a88 pg_upgrade: Convert old visibility map format to new format.
Commit a892234f83 added a second bit per
page to the visibility map, but pg_upgrade has been unaware of it up
until now.  Therefore, a pg_upgrade from an earlier major release of
PostgreSQL to any commit preceding this one and following the one
mentioned above would result in invalid visibility map contents on the
new cluster, very possibly leading to data corruption.  This plugs
that hole.

Masahiko Sawada, reviewed by Jeff Janes, Bruce Momjian, Simon Riggs,
Michael Paquier, Andres Freund, me, and others.
2016-03-11 12:34:20 -05:00
Robert Haas 69ab7b9d6c psql: Don't automatically use expanded format when there's 1 column.
Andreas Karlsson and Robert Haas
2016-03-11 08:04:01 -05:00
Magnus Hagander 38c83c9b75 Refactor receivelog.c parameters
Much cruft had accumulated over time with a large number of parameters
passed down between functions very deep. With this refactoring, instead
introduce a StreamCtl structure that holds the parameters, and pass around
a pointer to this structure instead. This makes it much easier to add or
remove fields that are needed deeper down in the implementation without
having to modify every function header in the file.

Patch by me after much nagging from Andres
Reviewed by Craig Ringer and Daniel Gustafsson
2016-03-11 11:15:12 +01:00
Robert Haas accf7616ff pgbench: When -T is used, don't wait for transactions beyond end of run.
At low rates, this can lead to pgbench taking significantly longer to
terminate than the user might expect.  Repair.

Fabien Coelho, reviewed by Aleksander Alekseev, Álvaro Herrera, and me.
2016-03-09 13:11:05 -05:00
Peter Eisentraut a40814d7aa Handle invalid libpq sockets in more places
Also, make error messages consistent.

From: Michael Paquier <michael.paquier@gmail.com>
2016-03-08 21:10:33 -05:00
Peter Eisentraut 92d4294d4b psql: Fix some strange code in SQL help creation
Struct QL_HELP used to be defined as static in the sql_help.h header
file, which is included in sql_help.c and help.c, thus creating two
separate instances of the struct.  This causes a warning from GCC 6,
because the struct is not used in sql_help.c.

Instead, declare the struct as extern in the header file and define it
in sql_help.c.  This also allows making a bunch of functions static
because they are no longer needed outside of sql_help.c.

Reviewed-by: Thomas Munro <thomas.munro@enterprisedb.com>
2016-03-08 19:41:51 -05:00
Robert Haas 6f56b41ac0 pg_upgrade: Remove converter plugin facility.
We've not found a use for this so far, and the current need, which
is to convert the visibility map to a new format, does not suit the
existing design anyway.  So just rip it out.

Author: Masahiko Sawada, slightly revised by me.
Discussion: 20160215211313.GB31273@momjian.us
2016-03-08 08:13:02 -05:00
Tom Lane b642e50aea Fix backwards test for Windows service-ness in pg_ctl.
A thinko in a96761391 caused pg_ctl to get it exactly backwards when
deciding whether to report problems to the Windows eventlog or to stderr.
Per bug #14001 from Manuel Mathar, who also identified the fix.
Like the previous patch, back-patch to all supported branches.
2016-03-07 10:40:44 -05:00
Tom Lane 94f1adccd3 Re-fix broken definition for function name in pgbench's exprscan.l.
Wups, my first try wasn't quite right either.  Too focused on fixing
the existing bug, not enough on not introducing new ones.
2016-03-06 21:45:34 -05:00
Tom Lane 3899caf772 Fix broken definition for function name in pgbench's exprscan.l.
As written, this would accept e.g. 123e9 as a function name.  Aside
from being mildly astonishing, that would come back to haunt us if
we ever try to add float constants to the expression syntax.  Insist
that function names start with letters (or at least non-digits).

In passing reset yyline as well as yycol when starting a new expression.
This variable is useless since it's used nowhere, but if we're going
to have it we should have it act sanely.
2016-03-06 21:04:25 -05:00
Joe Conway dc7d70ea05 Expose control file data via SQL accessible functions.
Add four new SQL accessible functions: pg_control_system(),
pg_control_checkpoint(), pg_control_recovery(), and pg_control_init()
which expose a subset of the control file data.

Along the way move the code to read and validate the control file to
src/common, where it can be shared by the new backend functions
and the original pg_controldata frontend program.

Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 11:10:19 -08:00
Robert Haas 9445db925e Fix query-based tab completion for multibyte characters.
The existing code confuses the byte length of the string (which is
relevant when passing it to pg_strncasecmp) with the character length
of the string (which is relevant when it is used with the SQL substring
function).  Separate those two concepts.

Report and patch by Kyotaro Horiguchi, reviewed by Thomas Munro and
reviewed and further revised by me.
2016-03-04 11:53:20 -05:00
Alvaro Herrera d561f1caec pgbench: accept unambiguous builtin prefixes for -b
This makes it easier to use "-b se" instead of typing the full "-b
select-only".

Author: Fabien Coelho
Reviewed-by: Michaël Paquier
2016-03-03 19:37:13 -03:00
Alvaro Herrera 2c83f435a3 Rework PostgresNode's psql method
This makes the psql() method much more capable: it captures both stdout
and stderr; it now returns the psql exit code rather than stdout; a
timeout can now be specified, as can ON_ERROR_STOP behavior; it gained a
new "on_error_die" (defaulting to off) parameter to raise an exception
if there's any problem.  Finally, additional parameters to psql can be
passed if there's need for further tweaking.

For convenience, a new safe_psql() method retains much of the old
behavior of psql(), except that it uses on_error_die on, so that
problems like syntax errors in SQL commands can be detected more easily.

Many existing TAP test files now use safe_psql, which is what is really
wanted.  A couple of ->psql() calls are now added in the commit_ts
tests, which verify that the right thing is happening on certain errors.
Some ->command_fails() calls in recovery tests that were verifying that
psql failed also became ->psql() calls now.

Author: Craig Ringer. Some tweaks by Álvaro Herrera
Reviewed-By: Michaël Paquier
2016-03-03 17:58:30 -03:00
Robert Haas 7e137f846d Extend pgbench's expression syntax to support a few built-in functions.
Fabien Coelho, reviewed mostly by Michael Paquier and me, but also by
Heikki Linnakangas, BeomYong Lee, Kyotaro Horiguchi, Oleksander
Shulgin, and Álvaro Herrera.
2016-03-01 13:08:30 -05:00
Alvaro Herrera 89ac7004da Move some code from RewindTest into PostgresNode
Some code in the RewindTest test suite is more generally useful than
just for that suite, so put it where other test suites can reach it.

Some postgresql.conf parameters change their default values when a
cluster is initialized with 'allows_streaming' than the previous
behavior; most notably, autovacuum is no longer turned off.

(Also, we no longer call pg_ctl promote with -w, but that flag doesn't
actually do anything in promote so there's no behavior change.)

Author: Michael Paquier
2016-02-26 13:24:22 -03:00
Noah Misch 25924ac47a Clean the last few TAP suite tmp_check directories.
Back-patch to 9.5, where the suites were introduced.
2016-02-24 23:41:54 -05:00
Noah Misch 5882ca6686 Call xlc __isync() after, not before, associated compare-and-swap.
Architecture reference material specifies this order, and s_lock.h
inline assembly agrees.  The former order failed to provide mutual
exclusion to lwlock.c and perhaps to other clients.  The two xlc
buildfarm members, hornet and mandrill, have failed sixteen times with
duplicate key errors involving pg_class_oid_index or pg_type_oid_index.
Back-patch to 9.5, where commit b64d92f1a5
introduced atomics.

Reviewed by Andres Freund and Tom Lane.
2016-02-19 22:47:50 -05:00
Peter Eisentraut a914a04142 pg_dump: Fix inconsistent sscanf() conversions
It was using %u to read a string that was earlier produced by snprintf with %d
into a signed integer variable.  This seems to work in practice but is
incorrect.

found by cppcheck
2016-02-18 20:12:38 -05:00
Bruce Momjian 3386f34cdc pg_upgrade: suppress creation of delete script
Suppress creation of the pg_upgrade delete script when the new data
directory is inside the old data directory.

Reported-by: IRC

Backpatch-through: 9.3, where delete script tests were added
2016-02-18 18:32:27 -05:00
Joe Conway fc8a81e3e7 Revert inadvertant change in pg_config behavior
In commit a5c43b88 the behavior of command line pg_config was
inadvertantly changed to include the config name when specific
configs are requested, similar to when none are requested and
all are emitted. This breaks scripts that expect to use
pg_config for e.g. PGXS. Revert the behavior to the previous.
2016-02-17 10:00:34 -08:00
Joe Conway a5c43b8869 Add new system view, pg_config
Move and refactor the underlying code for the pg_config client
application to src/common in support of sharing it with a new
system information SRF called pg_config() which makes the same
information available via SQL. Additionally wrap the SRF with a
new system view, as called pg_config.

Patch by me with extensive input and review by Michael Paquier
and additional review by Alvaro Herrera.
2016-02-17 09:12:06 -08:00
Alvaro Herrera 5df44d14ba pgbench: avoid FD_ISSET on an invalid file descriptor
The original code wasn't careful to test the file descriptor returned by
PQsocket() for an invalid socket.  If an invalid socket did turn up,
that would amount to calling FD_ISSET with fd = -1, whereby undefined
behavior can be invoked.

To fix, test file descriptor for validity and stop further processing if
that fails.

Problem noticed by Coverity.

There is an existing FD_ISSET callsite that does check for invalid
sockets beforehand, but the error message reported by it was
strerror(errno); in testing the aforementioned change, that turns out to
result in "bad socket: Success" which isn't terribly helpful.  Instead
use PQerrorMessage() in both places which is more likely to contain an
useful error message.

Backpatch-through: 9.1.
2016-02-15 20:33:43 -03:00
Bruce Momjian 13a6fa3634 pg_upgrade: Add C comment about NextXID delimiter
We don't test the catversion for the NextXID delimiter change, we just
test the string contents;  explain why.

Reported-by: Michael Paquier
2016-02-12 17:53:36 -05:00
Joe Conway 59a884e985 Change delimiter used for display of NextXID
NextXID has been rendered in the form of a pg_lsn even though it
really is not. This can cause confusion, so change the format from
%u/%u to %u:%u, per discussion on hackers.

Complaint by me, patch by me and Bruce, reviewed by Michael Paquier
and Alvaro. Applied to HEAD only.

Author: Joe Conway, Bruce Momjian
Reviewed-by: Michael Paquier, Alvaro Herrera
Backpatch-through: master
2016-02-12 14:23:59 -08:00
Alvaro Herrera 34f13cc484 pgbench: cleanup use of a "logfile" parameter
There is no reason to have the per-thread logfile file pointer as a
separate parameter in various functions: it's much simpler to put it in
the per-thread state struct instead, which is already being passed to
all functions that need the log file anyway.  Change the callsites in
which it was used as a boolean to test whether logging is active, so
that they use the use_log global variable instead.

No backpatch, even though this exists since commit a887c486d5 of March
2010, because this is just for cleanliness' sake and the surrounding
code has been modified a lot recently anyway.
2016-02-12 17:30:46 -03:00
Alvaro Herrera db94419ffd pgbench: fix segfault with empty sql file
Commit 1d0c3b3f8a introduced a bug that causes pgbench to crash if an
empty script file is specified.  Fix it by rejecting such files at
startup, which is the historical and intended behavior.

Reported-By: Jeff Janes
Discussion: https://www.postgresql.org/message-id/CAMkU=1zxKUbLPOt9hQWFp14pTc=V0cGo2GQBbn2GsK2Pu+8ZfA@mail.gmail.com
2016-02-12 17:14:45 -03:00
Noah Misch 2ffa869620 Accept pg_ctl timeout from the PGCTLTIMEOUT environment variable.
Many automated test suites call pg_ctl.  Buildfarm members axolotl,
hornet, mandrill, shearwater, sungazer and tern have failed when server
shutdown took longer than the pg_ctl default 60s timeout.  This addition
permits slow hosts to easily raise the timeout without us editing a
--timeout argument into every test suite pg_ctl call.  Back-patch to 9.1
(all supported versions) for the sake of automated testing.

Reviewed by Tom Lane.
2016-02-10 20:34:02 -05:00
Tom Lane c5e9b77127 Revert "Temporarily make pg_ctl and server shutdown a whole lot chattier."
This reverts commit 3971f64843 and a
couple of followon debugging commits; I think we've learned what we can
from them.
2016-02-10 16:01:04 -05:00
Tom Lane 3971f64843 Temporarily make pg_ctl and server shutdown a whole lot chattier.
This is a quick hack, due to be reverted when its purpose has been served,
to try to gather information about why some of the buildfarm critters
regularly fail with "postmaster does not shut down" complaints.  Maybe they
are just really overloaded, but maybe something else is going on.  Hence,
instrument pg_ctl to print the current time when it starts waiting for
postmaster shutdown and when it gives up, and add a lot of logging of the
current time in the server's checkpoint and shutdown code paths.

No attempt has been made to make this pretty.  I'm not even totally sure
if it will build on Windows, but we'll soon find out.
2016-02-08 18:43:11 -05:00
Tom Lane 0ed707e9b7 In pg_dump, ensure that view triggers are processed after view rules.
If a view is split into CREATE TABLE + CREATE RULE to break a circular
dependency, then any triggers on the view must be dumped/reloaded after
the CREATE RULE; else the backend may reject the CREATE TRIGGER because
it's the wrong type of trigger for a plain table.  This works all right
in plain dump/restore because of pg_dump's sorting heuristic that places
triggers after rules.  However, when using parallel restore, the ordering
must be enforced by a dependency --- and we didn't have one.

Fixing this is a mere matter of adding an addObjectDependency() call,
except that we need to be able to find all the triggers belonging to the
view relation, and there was no easy way to do that.  Add fields to
pg_dump's TableInfo struct to remember where the associated TriggerInfo
struct(s) are.

Per bug report from Dennis Kögel.  The failure can be exhibited at least
as far back as 9.1, so back-patch to all supported branches.
2016-02-04 00:26:10 -05:00
Tom Lane 2808a2e0f3 Remove printQueryOpt.quote field.
This field was included in the original definition of the printQueryOpt
struct in commit a45195a191, but it was not used anywhere in that
commit, nor since then.  Spotted by Dickson S. Guedes.
2016-02-02 15:26:30 -05:00
Magnus Hagander 23f3cc36ed Fix typo in comment 2016-02-02 13:49:02 +01:00
Alvaro Herrera 1d0c3b3f8a pgbench: allow per-script statistics
Provide per-script statistical info (count of transactions executed
under that script, average latency for the whole script) after a
multi-script run, adding an intermediate level of detail to existing
global stats and per-command stats.

Author: Fabien Coelho
Reviewer: Michaël Paquier, Álvaro Herrera
2016-02-01 15:55:33 +01:00
Robert Haas 64f5edca24 pgbench: Install guards against obscure overflow conditions.
Dividing INT_MIN by -1 or taking INT_MIN modulo -1 can sometimes
cause floating-point exceptions or otherwise misbehave.

Fabien Coelho and Michael Paquier
2016-02-01 08:23:41 -05:00
Fujii Masao 89611c4dfa Various fixes to "ALTER ... SET/RESET" tab completions
Add
- ALTER SYSTEM SET/RESET ... -> GUC variables
- ALTER TABLE ... SET WITH -> OIDS
- ALTER DATABASE/FUNCTION/ROLE/USER ... SET/RESET -> GUC variables
- ALTER DATABASE/FUNCTION/ROLE/USER ... SET ... -> FROM CURRENT/TO
- ALTER DATABASE/FUNCTION/ROLE/USER ... SET ... TO/= -> possible values

Author: Fujii Masao
Reviewed-by: Michael Paquier, Masahiko Sawada
2016-02-01 22:19:51 +09:00
Alvaro Herrera b603766496 pgbench: refactor handling of stats tracking
This doesn't add any functionality but just shuffles things around so
that it can be reused and improved later.

Author: Fabien Coelho
Reviewed-by: Michael Paquier, Álvaro Herrera
2016-01-29 13:05:08 +01:00
Tom Lane 7e22470471 Fix incorrect pattern-match processing in psql's \det command.
listForeignTables' invocation of processSQLNamePattern did not match up
with the other ones that handle potentially-schema-qualified names; it
failed to make use of pg_table_is_visible() and also passed the name
arguments in the wrong order.  Bug seems to have been aboriginal in commit
0d692a0dc9.  It accidentally sort of worked as long as you didn't
inquire too closely into the behavior, although the silliness was later
exposed by inconsistencies in the test queries added by 59efda3e50
(which I probably should have questioned at the time, but didn't).

Per bug #13899 from Reece Hart.  Patch by Reece Hart and Tom Lane.
Back-patch to all affected branches.
2016-01-29 10:28:02 +01:00
Robert Haas 025b2f3392 Fix cross-version pg_dump for aggregate combine functions.
Fixes a defect in commit a7de3dc5c3.

David Rowley, per report from Jeff Janes, who also checked that the
fix works.
2016-01-27 21:45:07 -05:00
Alvaro Herrera 8bea3d2219 pgbench: improve multi-script support
Previously, it was possible to specify one or several custom scripts to
run, or only one of the builtin scripts.  With this patch it is also
possible to specify to run the builtin scripts multiple times, using the
new -b option.  Also, unify the code for both cases; this eases future
pgbench improvements.

Author: Fabien Coelho
Review: Michaël Paquier, Álvaro Herrera
2016-01-27 02:54:22 +01:00
Alvaro Herrera 5b3cc1af2f Mostly mechanical cleanup of pgbench
pgindent for recent commits; also change some variables from int to
boolean, which is how they are really used.

Mostly submitted by Fabien Coelho; this is in preparation to commit
further patches to the file.
2016-01-27 02:11:34 +01:00
Kevin Grittner 879d71393d Various fixes to REFRESH MATERIALIZED VIEW tab completion.
Masahiko Sawada, Fujii Masao, Kevin Grittner
2016-01-26 08:45:08 -06:00
Peter Eisentraut 6ae4c8de00 psql: Improve completion of FDW DDL commands
Add
- ALTER FOREIGN DATA WRAPPER -> RENAME TO
- ALTER SERVER -> RENAME TO
- ALTER SERVER ... VERSION ... -> OPTIONS
- CREATE FOREIGN DATA WRAPPER -> OPTIONS
- CREATE SERVER -> OPTIONS
- CREATE|ALTER USER MAPPING -> OPTIONS

From: Andreas Karlsson <andreas@proxel.se>
2016-01-23 06:57:42 -05:00
Alvaro Herrera df43fcf457 pg_dump: Fix quoting of domain constraint names
The original code was adding double quotes to an already-quoted
identifier, leading to nonsensical results.  Remove the quoting call.

I introduced the broken code in 7eca575d1c of 9.5 era, so backpatch to
9.5.

Report and patch by Elvis Pranskevichus
Reviewed by Michael Paquier
2016-01-22 20:04:35 -03:00
Simon Riggs c80b31d557 Refactor headers to split out standby defs
Jeff Janes
2016-01-20 18:51:34 -08:00
Peter Eisentraut d0f2f53cd6 psql: Add tab completion for COPY with query
From: Andreas Karlsson <andreas@proxel.se>
2016-01-20 21:27:46 -05:00
Robert Haas a7de3dc5c3 Support multi-stage aggregation.
Aggregate nodes now have two new modes: a "partial" mode where they
output the unfinalized transition state, and a "finalize" mode where
they accept unfinalized transition states rather than individual
values as input.

These new modes are not used anywhere yet, but they will be necessary
for parallel aggregation.  The infrastructure also figures to be
useful for cases where we want to aggregate local data and remote
data via the FDW interface, and want to bring back partial aggregates
from the remote side that can then be combined with locally generated
partial aggregates to produce the final value.  It may also be useful
even when neither FDWs nor parallelism are in play, as explained in
the comments in nodeAgg.c.

David Rowley and Simon Riggs, reviewed by KaiGai Kohei, Heikki
Linnakangas, Haribabu Kommi, and me.
2016-01-20 13:46:50 -05:00
Alvaro Herrera c8642d909f PostgresNode: Add names to nodes
This makes the log files easier to follow when investigating a test
failure.

Author: Michael Paquier
Review: Noah Misch
2016-01-20 14:13:11 -03:00
Andrew Dunstan 53c949c1be Remove Cygwin-specific code from pg_ctl
This code has been there for a long time, but it's never really been
needed. Cygwin has its own utility for registering, unregistering,
stopping and starting Windows services, and that's what's used in the
Cygwin postgres packages. So now pg_ctl for Cygwin looks like it is for
any Unix platform.

Michael Paquier and me
2016-01-19 07:31:18 -05:00
Tom Lane 57ce9acc04 Remove dead code in pg_dump.
Coverity quite reasonably complained that this check for fout==NULL
occurred after we'd already dereferenced fout.  However, the check
is just dead code since there is no code path by which CreateArchive
can return a null pointer.  Errors such as can't-open-that-file are
reported down inside CreateArchive, and control doesn't return.
So let's silence the warning by removing the dead code, rather than
continuing to pretend it does something.

Coverity didn't complain about this before 5b5fea2a1, so back-patch
to 9.5 like that patch.
2016-01-17 11:38:40 -05:00
Peter Eisentraut 4189e3d659 psql: Add completion support for DROP INDEX CONCURRENTLY
based on patch by Kyotaro Horiguchi
2016-01-16 20:46:14 -05:00
Robert Haas 23c2dd03d5 Fix spelling mistakes.
Same patch submitted independently by David Rowley and Peter Geoghegan.
2016-01-14 23:16:40 -05:00
Tom Lane e72d7d8531 Handle extension members when first setting object dump flags in pg_dump.
pg_dump's original approach to handling extension member objects was to
run around and clear (or set) their dump flags rather late in its data
collection process.  Unfortunately, quite a lot of code expects those flags
to be valid before that; which was an entirely reasonable expectation
before we added extensions.  In particular, this explains Karsten Hilbert's
recent report of pg_upgrade failing on a database in which an extension
has been installed into the pg_catalog schema.  Its objects are initially
marked as not-to-be-dumped on the strength of their schema, and later we
change them to must-dump because we're doing a binary upgrade of their
extension; but we've already skipped essential tasks like making associated
DO_SHELL_TYPE objects.

To fix, collect extension membership data first, and incorporate it in the
initial setting of the dump flags, so that those are once again correct
from the get-go.  This has the undesirable side effect of slightly
lengthening the time taken before pg_dump acquires table locks, but testing
suggests that the increase in that window is not very much.

Along the way, get rid of ugly special-case logic for deciding whether
to dump procedural languages, FDWs, and foreign servers; dump decisions
for those are now correct up-front, too.

In 9.3 and up, this also fixes erroneous logic about when to dump event
triggers (basically, they were *always* dumped before).  In 9.5 and up,
transform objects had that problem too.

Since this problem came in with extensions, back-patch to all supported
versions.
2016-01-13 18:55:27 -05:00
Tom Lane 5b5fea2a11 Access pg_dump's options structs through Archive struct, not directly.
Rather than passing around DumpOptions and RestoreOptions as separate
arguments, add fields to struct Archive to carry pointers to these objects,
and access them through those fields when needed.  There already was a
RestoreOptions pointer in Archive, though for no obvious reason it was part
of the "private" struct rather than out where pg_dump.c could see it.

Doing this allows reversion of quite a lot of parameter-addition changes
made in commit 0eea8047bf, which is a good thing IMO because this will
reduce the code delta between 9.4 and 9.5, probably easing a few future
back-patch efforts.  Moreover, the previous commit only added a DumpOptions
argument to functions that had to have it at the time, which means we could
anticipate still more code churn (and more back-patch hazard) as the
requirement spread further.  I'd hit exactly that problem in my upcoming
patch to fix extension membership marking, which is what motivated me to
do this.
2016-01-13 17:48:33 -05:00
Tom Lane 26905e009b Run pgindent on src/bin/pg_dump/*
To ease doing indent fixups on a couple of patches I have in progress.
2016-01-13 15:48:54 -05:00
Peter Eisentraut b1bfb28b58 psql: Improve CREATE INDEX CONCURRENTLY tab completion
The completion of CREATE INDEX CONCURRENTLY was lacking in several ways
compared to a plain CREATE INDEX command:

- CREATE INDEX <name> ON completes table names, but didn't with
  CONCURRENTLY.

- CREATE INDEX completes ON and existing index names, but with
  CONCURRENTLY it only completed ON.

- CREATE INDEX <name> completes ON, but didn't with CONCURRENTLY.

These are now all fixed.
2016-01-12 20:54:27 -05:00
Peter Eisentraut bc56d5898d psql: Fix CREATE INDEX tab completion
The previous code supported a syntax like CREATE INDEX name
CONCURRENTLY, which never existed.  Mistake introduced in commit
37ec19a15c.  Remove the addition of
CONCURRENTLY at that point.
2016-01-12 20:54:27 -05:00
Peter Eisentraut 7032703009 psql: Update tab completion comment
This just updates a comment to match the code.

from Michael Paquier
2016-01-12 20:54:27 -05:00
Alvaro Herrera 4631721166 Revert "Blind attempt at a Cygwin fix"
This reverts commit e9282e9532, which blew
up in a pretty spectacular way.  Re-introduce the original code while we
search for a real fix.
2016-01-08 13:18:40 -03:00
Tom Lane b41fb65056 Fix unobvious interaction between -X switch and subdirectory creation.
Turns out the only reason initdb -X worked is that pg_mkdir_p won't
whine if you point it at something that's a symlink to a directory.
Otherwise, the attempt to create pg_xlog/ just like all the other
subdirectories would have failed.  Let's be a little more explicit
about what's happening.  Oversight in my patch for bug #13853
(mea culpa for not testing -X ...)
2016-01-07 18:20:57 -05:00
Tom Lane 33b054bc79 Use plain mkdir() not pg_mkdir_p() to create subdirectories of PGDATA.
When we're creating subdirectories of PGDATA during initdb, we know darn
well that the parent directory exists (or should exist) and that the new
subdirectory doesn't (or shouldn't).  There is therefore no need to use
anything more complicated than mkdir().  Using pg_mkdir_p() just opens us
up to unexpected failure modes, such as the one exhibited in bug #13853
from Nuri Boardman.  It's not very clear why pg_mkdir_p() went wrong there,
but it is clear that we didn't need to be trying to create parent
directories in the first place.  We're not even saving any code, as proven
by the fact that this patch nets out at minus five lines.

Since this is a response to a field bug report, back-patch to all branches.
2016-01-07 15:22:24 -05:00
Alvaro Herrera a967613911 Windows: Make pg_ctl reliably detect service status
pg_ctl is using isatty() to verify whether the process is running in a
terminal, and if not it sends its output to Windows' Event Log ... which
does the wrong thing when the output has been redirected to a pipe, as
reported in bug #13592.

To fix, make pg_ctl use the code we already have to detect service-ness:
in the master branch, move src/backend/port/win32/security.c to src/port
(with suitable tweaks so that it runs properly in backend and frontend
environments); pg_ctl already has access to pgport so it Just Works.  In
older branches, that's likely to cause trouble, so instead duplicate the
required code in pg_ctl.c.

Author: Michael Paquier
Bug report and diagnosis: Egon Kocjan
Backpatch: all supported branches
2016-01-07 11:59:08 -03:00
Tom Lane dad08994b2 In initdb's post-bootstrap phase, drop temp tables explicitly.
Although these temp tables will get removed from template1 at the end of
the standalone-backend run, that's too late to keep them from getting
copied into the template0 and postgres databases, now that we use only a
single backend run for the whole sequence.  While no real harm is done
by the extra copies (since they'd be deleted on first use of the temp
schema), it's still unsightly, and it would mean some wasted cycles for
every database creation for the life of the installation.

Oversight in commit c4a8812cf6.  Noticed by Amit Langote.
2016-01-06 12:25:32 -05:00
Tom Lane 3343ea9e8e Sort $(wildcard) output where needed for reproducible build output.
The order of inclusion of .o files makes a difference in linker output;
not a functional difference, but still a bitwise difference, which annoys
some packagers who would like reproducible builds.

Report and patch by Christoph Berg
2016-01-05 15:47:05 -05:00
Alvaro Herrera 4aecd22d3c Make pg_receivexlog silent with 9.3 and older servers
A pointless and confusing error message is shown to the user when
attempting to identify a 9.3 or older remote server with a 9.5/9.6
pg_receivexlog, because the return signature of IDENTIFY_SYSTEM was
changed in 9.4.  There's no good reason for the warning message, so
shuffle code around to keep it quiet.

(pg_recvlogical is also affected by this commit, but since it obviously
cannot work with 9.3 that doesn't actually matter much.)

Backpatch to 9.5.

Reported by Marco Nenciarini, who also wrote the initial patch.  Further
tweaked by Robert Haas and Fujii Masao; reviewed by Michael Paquier and
Craig Ringer.
2016-01-05 17:25:12 -03:00
Tom Lane 4f18010af1 Convert psql's tab completion for backslash commands to the new style.
This requires adding some more infrastructure to handle both case-sensitive
and case-insensitive matching, as well as the ability to match a prefix of
a previous word.  So it ends up being about a wash line-count-wise, but
it's just as big a readability win here as in the SQL tab completion rules.

Michael Paquier, some adjustments by me
2016-01-05 12:00:13 -05:00
Tom Lane 9b181b0363 In psql's tab completion, change most TailMatches patterns to Matches.
In the refactoring in commit d37b816dc9,
we mostly kept to the original design whereby only the last few words
on the line were matched to identify a completable pattern.  However,
after commit d854118c8d, there's really
no reason to do it like that: where it's sensible, we can use patterns
that expect to match the entire input line.  And mostly, it's sensible.
Matching the entire line greatly reduces the odds of a false match that
leads to offering irrelevant completions.  Moreover (though I've not
tried to measure this), it should make tab completion faster since
many of the patterns will be discarded after a single integer comparison
that finds that the wrong number of words appear on the line.

There are certain identifiable places where we still need to use
TailMatches because the statement in question is allowed to appear
embedded in a larger statement.  These are just a small minority of
the existing patterns, though, so the benefit of switching where
possible is large.

It's possible that this patch has removed some within-line matching
behaviors that are in fact desirable, but we can put those back when
we get complaints.  Most of the removed behaviors are certainly silly.

Michael Paquier, with some further adjustments by me
2016-01-04 20:08:08 -05:00
Robert Haas 8978eb03a8 Fix typo in comment.
Masahiko Sawada
2016-01-04 10:12:44 -05:00
Tom Lane b416c0bb62 Teach pg_dump to quote reloption values safely.
Commit c7e27becd2 fixed this on the backend side, but we neglected
the fact that several code paths in pg_dump were printing reloptions
values that had not gotten massaged by ruleutils.  Apply essentially the
same quoting logic in those places, too.
2016-01-02 19:04:45 -05:00
Bruce Momjian ee94300446 Update copyright for 2016
Backpatch certain files through 9.1
2016-01-02 13:33:40 -05:00