Commit Graph

5753 Commits

Author SHA1 Message Date
Thomas Munro 7f338369ca Allow invisible PROMPT2 in psql.
Keep track of the visible width of PROMPT1, and provide %w as a way
for PROMPT2 to generate the same number of spaces.

Author: Thomas Munro, with ideas from others
Reviewed-by: Tom Lane (earlier version)
Discussion: https://postgr.es/m/CA%2BhUKG%2BzGd7RigjWbxwhzGW59gUpf76ydQECeGdEdodH6nd__A%40mail.gmail.com
2019-11-19 15:56:21 +13:00
Tom Lane 29aeda6e4e Handle arrays and ranges in pg_upgrade's test for non-upgradable types.
pg_upgrade needs to check whether certain non-upgradable data types
appear anywhere on-disk in the source cluster.  It knew that it has
to check for these types being contained inside domains and composite
types; but it somehow overlooked that they could be contained in
arrays and ranges, too.  Extend the existing recursive-containment
query to handle those cases.

We probably should have noticed this oversight while working on
commit 0ccfc2822 and follow-ups, but we failed to :-(.  The whole
thing's possibly a bit overdesigned, since we don't really expect
that any of these types will appear on disk; but if we're going to
the effort of doing a recursive search then it's silly not to cover
all the possibilities.

While at it, refactor so that we have only one copy of the search
logic, not three-and-counting.  Also, to keep the branches looking
more alike, back-patch the output wording change of commit 1634d3615.

Back-patch to all supported branches.

Discussion: https://postgr.es/m/31473.1573412838@sss.pgh.pa.us
2019-11-13 11:35:37 -05:00
Amit Kapila 1379fd537f Introduce the 'force' option for the Drop Database command.
This new option terminates the other sessions connected to the target
database and then drop it.  To terminate other sessions, the current user
must have desired permissions (same as pg_terminate_backend()).  We don't
allow to terminate the sessions if prepared transactions, active logical
replication slots or subscriptions are present in the target database.

Author: Pavel Stehule with changes by me
Reviewed-by: Dilip Kumar, Vignesh C, Ibrar Ahmed, Anthony Nowocien,
Ryan Lambert and Amit Kapila
Discussion: https://postgr.es/m/CAP_rwwmLJJbn70vLOZFpxGw3XD7nLB_7+NKz46H5EOO2k5H7OQ@mail.gmail.com
2019-11-13 08:25:33 +05:30
Tom Lane a7145f6bc8 Fix integer-overflow edge case detection in interval_mul and pgbench.
This patch adopts the overflow check logic introduced by commit cbdb8b4c0
into two more places.  interval_mul() failed to notice if it computed a
new microseconds value that was one more than INT64_MAX, and pgbench's
double-to-int64 logic had the same sorts of edge-case problems that
cbdb8b4c0 fixed in the core code.

To make this easier to get right in future, put the guts of the checks
into new macros in c.h, and add commentary about how to use the macros
correctly.

Back-patch to all supported branches, as we did with the previous fix.

Yuya Watari

Discussion: https://postgr.es/m/CAJ2pMkbkkFw2hb9Qb1Zj8d06EhWAQXFLy73St4qWv6aX=vqnjw@mail.gmail.com
2019-11-07 11:22:58 -05:00
Fujii Masao a386942bd2 Add "G" (server-side data generation) as an initialization step in pgbench.
This commit allows --init-steps option in pgbench to accept "G" character
meaning server-side data generation as an initialization step.
With "G", only limited queries are sent from pgbench client and
then data is actually generated in the server. This might make
the initialization phase faster if the bandwidth between pgbench client
and the server is low.

Author: Fabien Coelho
Reviewed-by: Anna Endo, Ibrar Ahmed, Fujii Masao
Discussion: https://postgr.es/m/alpine.DEB.2.21.1904061826420.3678@lancre
2019-11-06 11:02:30 +09:00
Andres Freund 26aaf97b68 Make StringInfo available to frontend code.
There's plenty places in frontend code that could benefit from a
string buffer implementation. Some because it yields simpler and
faster code, and some others because of the desire to share code
between backend and frontend.

While there is a string buffer implementation available to frontend
code, libpq's PQExpBuffer, it is clunkier than stringinfo, it
introduces a libpq dependency, doesn't allow for sharing between
frontend and backend code, and has a higher API/ABI stability
requirement due to being exposed via libpq.

Therefore it seems best to just making StringInfo being usable by
frontend code. There's not much to do for that, except for rewriting
two subsequent elog/ereport calls into others types of error
reporting, and deciding on a maximum string length.

For the maximum string size I decided to privately define MaxAllocSize
to the same value as used in the backend. It seems likely that we'll
want to reconsider this for both backend and frontend code in the not
too far away future.

For now I've left stringinfo.h in lib/, rather than common/, to reduce
the likelihood of unnecessary breakage. We could alternatively decide
to provide a redirecting stringinfo.h in lib/, or just not provide
compatibility.

Author: Andres Freund
Reviewed-By: Kyotaro Horiguchi, Daniel Gustafsson
Discussion: https://postgr.es/m/20190920051857.2fhnvhvx4qdddviz@alap3.anarazel.de
2019-11-05 14:56:40 -08:00
Andres Freund 01368e5d9d Split all OBJS style lines in makefiles into one-line-per-entry style.
When maintaining or merging patches, one of the most common sources
for conflicts are the list of objects in makefiles. Especially when
the split across lines has been changed on both sides, which is
somewhat common due to attempting to stay below 80 columns, those
conflicts are unnecessarily laborious to resolve.

By splitting, and alphabetically sorting, OBJS style lines into one
object per line, conflicts should be less frequent, and easier to
resolve when they still occur.

Author: Andres Freund
Discussion: https://postgr.es/m/20191029200901.vww4idgcxv74cwes@alap3.anarazel.de
2019-11-05 14:41:07 -08:00
Tom Lane 5102f39440 Stabilize pg_dump output order for similarly-named triggers and policies.
The code only compared two triggers' names and namespaces (the latter
being the owning table's schema).  This could result in falling back
to an OID-based sort of similarly-named triggers on different tables.
We prefer to avoid that, so add a comparison of the table names too.
(The sort order is thus table namespace, trigger name, table name,
which is a bit odd, but it doesn't seem worth contorting the code
to work around that.)

Likewise for policy objects, in 9.5 and up.

Complaint and fix by Benjie Gillam.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/CAMThMzEEt2mvBbPgCaZ1Ap1N-moGn=Edxmadddjq89WG4NpPtQ@mail.gmail.com
2019-11-04 16:25:05 -05:00
Andres Freund e4d92126fd pg_waldump: Fix --bkp-details to not issue spurious newlines for FPWs.
The additional newline seems to have accidentally been introduced in
2c03216d83, in 9.5. The newline is only issued when an FPW is
present for the block reference.

While there could be an argument that removing the newlines in the
back branches could cause a problem for somebody parsing the
pg_waldump output, the likelihood of that seems small enough. It seems
at least equally likely that the randomness of when newlines are
issued causes problems.

Author: Andres Freund
Discussion: https://postgr.es/m/20191029233341.4gnyau7e5v2lh5sc@alap3.anarazel.de
Backpatch: 9.5, like 2c03216d83.
2019-10-29 22:53:05 -07:00
Andres Freund e0f76f204c pg_waldump: Fix small memory leak when rmgr->rm_identify returns NULL.
This got broken in 604f7956b9, shortly after rm_identify's
introduction.

Author: Andres Freund
Discussion: https://postgr.es/m/20191029233341.4gnyau7e5v2lh5sc@alap3.anarazel.de
Backpatch: 9.5, where rm_identify was introduced
2019-10-29 19:18:07 -07:00
Michael Paquier 6ca86bb7e9 Fix typos in the code
Author: Vignesh C
Reviewed-by: Dilip Kumar, Michael Paquier
Discussion: https://postgr.es/m/CALDaNm0ni+GAOe4+fbXiOxNrVudajMYmhJFtXGX-zBPoN8ixhw@mail.gmail.com
2019-10-30 10:03:00 +09:00
Tom Lane f88544904e On Windows, use COMSPEC to find the location of cmd.exe.
Historically, psql consulted COMSPEC to spawn a shell in its \! command,
but we just invoked "cmd" when spawning shells in pg_ctl and pg_regress.
It seems better to rely on the environment variable, if it's set,
in all cases.

It's debatable whether this is a bug fix or just a behavioral change,
so no back-patch.

Juan José Santamaría Flecha

Discussion: https://postgr.es/m/16080-5d7f03222469f717@postgresql.org
2019-10-28 14:15:03 -04:00
Michael Paquier 51970fa8df Fix initialization of fake LSN for unlogged relations
9155580 has changed the value of the first fake LSN for unlogged
relations from 1 to FirstNormalUnloggedLSN (aka 1000), GiST requiring a
non-zero LSN on some pages to allow an interlocking logic to work, but
its value was still initialized to 1 at the beginning of recovery or
after running pg_resetwal.  This fixes the initialization for both code
paths.

Author: Takayuki Tsunakawa
Reviewed-by: Dilip Kumar, Kyotaro Horiguchi, Michael Paquier
Discussion: https://postgr.es/m/OSBPR01MB2503CE851940C17DE44AE3D9FE6F0@OSBPR01MB2503.jpnprd01.prod.outlook.com
Backpatch-through: 12
2019-10-27 13:54:12 +09:00
Amit Kapila dddf4cdc33 Make the order of the header file includes consistent in non-backend modules.
Similar to commit 7e735035f2, this commit makes the order of header file
inclusion consistent for non-backend modules.

In passing, fix the case where we were using angle brackets (<>) for the
local module includes instead of quotes ("").

Author: Vignesh C
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/CALDaNm2Sznv8RR6Ex-iJO6xAdsxgWhCoETkaYX=+9DW3q0QCfA@mail.gmail.com
2019-10-25 07:41:52 +05:30
Bruce Momjian 0c089d36e2 pg_upgrade: adjust error output to use new database list format
Commit a524f50d0f added
old_11_check_for_sql_identifier_data_type_usage(), but it did not use
the clearer database error list format added to the master branch in
commit 1634d36157.  This commit fixes that.

Backpatch-through: master
2019-10-23 18:06:38 -04:00
Michael Paquier 4fa5edcd1a Remove last traces of --adduser/--no-adduser in createuser
8ae0d47 marked those options as obsolete back in 2005, with the options
removed from the documentation.  This removes the last references to
both options in the code which were kept around for compatibility
purposes with past commands.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/5da284a2-62d9-e338-88d1-26ee5009d93e@gmail.com
2019-10-23 12:27:03 +09:00
Alvaro Herrera 1752e35163 Fix parallel restore of FKs to partitioned tables
When an FK constraint is created, it needs the index on the referenced
table to exist and be valid.  When doing parallel pg_restore and the
referenced table was partitioned, this condition can sometimes not be
met, because pg_dump didn't emit sufficient object dependencies to
ensure so; this means that parallel pg_restore would fail in certain
conditions.  Fix by having pg_dump make the FK constraint object
dependent on the partition attachment objects for the constraint's
referenced index.

This has been broken since f56f8f8da6, so backpatch to Postgres 12.

Discussion: https://postgr.es/m/20191005224333.GA9738@alvherre.pgsql
2019-10-17 09:58:01 +02:00
Tomas Vondra a524f50d0f Improve the check for pg_catalog.unknown data type in pg_upgrade
The pg_upgrade check for pg_catalog.unknown type when upgrading from 9.6
had a couple of issues with domains and composite types - it detected
even composite types unused in objects with storage. So for example this
was enough to trigger an unnecessary pg_upgrade failure:

  CREATE TYPE unknown_composite AS (u pg_catalog.unknown)

On the other hand, this only happened with composite types directly on
the pg_catalog.unknown data type, but not with a domain. So this was not
detected

  CREATE DOMAIN unknown_domain AS pg_catalog.unknown;
  CREATE TYPE unknown_composite_2 AS (u unknown_domain);

unlike the first example. These false positives and inconsistencies are
unfortunate, but what's worse we've failed to detected objects using the
pg_catalog.unknown type through a domain. So we missed cases like this

  CREATE TABLE t (u unknown_composite_2);

The consequence is clusters broken after a pg_upgrade.

This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 10, where the of pg_catalog.unknown data type was restricted.

Author: Tomas Vondra
Backpatch-to: 10-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
2019-10-16 13:23:18 +02:00
Tomas Vondra 8d48e6a724 Improve the check for pg_catalog.line data type in pg_upgrade
The pg_upgrade check for pg_catalog.line data type when upgrading from
9.3 had a couple of issues with domains and composite types. Firstly, it
triggered false positives for composite types unused in objects with
storage. This was enough to trigger an unnecessary pg_upgrade failure:

  CREATE TYPE line_composite AS (l pg_catalog.line)

On the other hand, this only happened with composite types directly on
the pg_catalog.line data type, but not with a domain. So this was not
detected

  CREATE DOMAIN line_domain AS pg_catalog.line;
  CREATE TYPE line_composite_2 AS (l line_domain);

unlike the first example. These false positives and inconsistencies are
unfortunate, but what's worse we've failed to detected objects using the
pg_catalog.line data type through a domain. So we missed cases like this

  CREATE TABLE t (l line_composite_2);

The consequence is clusters broken after a pg_upgrade.

This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. 9.3 did not
support domains on composite types, but we can still have multi-level
composite types.

Backpatch all the way to 9.4, where the format for pg_catalog.line data
type changed.

Author: Tomas Vondra
Backpatch-to: 9.4-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
2019-10-16 13:23:14 +02:00
Thomas Munro d5ac14f9cc Use libc version as a collation version on glibc systems.
Using glibc's version string to detect potential collation definition
changes is not 100% reliable, but it's better than nothing.  Currently
this affects only collations explicitly provided by "libc".  More work
will be needed to handle the default collation.

Author: Thomas Munro, based on a suggestion from Christoph Berg
Reviewed-by: Peter Eisentraut
Discussion: https://postgr.es/m/4b76c6d4-ae5e-0dc6-7d0d-b5c796a07e34%402ndquadrant.com
2019-10-16 17:28:24 +13:00
Tomas Vondra 3a0e857394 Correct reference to pg_catalog.regtype in pg_upgrade query
The recursive CTE added in 0ccfc28223 referenced pg_catalog.regtype,
without the schema part, unlike all other queries in pg_upgrade.

Backpatch-to: 12
2019-10-15 00:25:04 +02:00
Tomas Vondra 0ccfc28223 Check for tables with sql_identifier during pg_upgrade
Commit 7c15cef86d changed sql_identifier data type to be based on name
instead of varchar.  Unfortunately, this breaks on-disk format for this
data type.  Luckily, that should be a very rare problem, as this data
type is used only in information_schema views, so this only affects user
objects (tables, materialized views and indexes).  One way to end in
such situation is to do CTAS with a query on those system views.

There are two options to deal with this - we can either abort pg_upgrade
if there are user objects with sql_identifier columns in pg_upgrade, or
we could replace the sql_identifier type with varchar.  Considering how
rare the issue is expected to be, and the complexity of replacing the
data type (e.g. in matviews), we've decided to go with the simple check.

The query is somewhat complex - the sql_identifier data type may be used
indirectly - through a domain, a composite type or both, possibly in
multiple levels.  Detecting this requires a recursive CTE.

Backpatch to 12, where the sql_identifier definition changed.

Reported-by: Hans Buschmann
Author: Tomas Vondra
Reviewed-by: Tom Lane
Backpatch-to: 12
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
2019-10-14 22:31:56 +02:00
Tom Lane f38291e927 Revert "Hack pg_ctl to report postmaster's exit status."
This reverts commit 6a5084eed4.
We learned what we needed to know from that.
2019-10-13 12:56:16 -04:00
Peter Eisentraut 3b5d3721c2 pg_upgrade: Clean up some redundant code
No need to call exit() after pg_fatal().  Clean up a few stragglers
for consistency.
2019-10-10 10:51:11 +02:00
Bruce Momjian 1634d36157 pg_upgrade: clarify the database names in error files
Previously, the "Database:" label in the error file was unclear if the
label was a status report or the problem was _in_ the database.  New
text is "In database:".

Reported-by: Justin Pryzby

Discussion: https://postgr.es/m/20191002172337.GC9680@telsasoft.com

Backpatch-through: head
2019-10-08 22:16:48 -04:00
Peter Eisentraut 38d8dce61f Remove some code for old unsupported versions of MSVC
As of d9dd406fe2, we require MSVC 2013,
which means _MSC_VER >= 1800.  This means that conditionals about
older versions of _MSC_VER can be removed or simplified.

Previous code was also in some cases handling MinGW, where _MSC_VER is
not defined at all, incorrectly, such as in pg_ctl.c and win32_port.h,
leading to some compiler warnings.  This should now be handled better.

Reviewed-by: Michael Paquier <michael@paquier.xyz>
2019-10-08 10:50:54 +02:00
Michael Paquier 55ba56415b Improve test coverage of pg_rewind
This includes new TAP tests for a couple of areas not covered yet and
some improvements:
- More coverage for --no-ensure-shutdown, the enforced recovery step and
--dry-run.
- Failures with option combinations and basic option checks.
- Removal of a duplicated comment.

Author: Alexey Kondratov, Michael Paquier
Discussion: https://postgr.es/m/20191007010651.GD14532@paquier.xyz
2019-10-08 11:46:30 +09:00
Tom Lane 6a5084eed4 Hack pg_ctl to report postmaster's exit status.
Temporarily change pg_ctl so that the postmaster's exit status will
be printed (to the postmaster's stdout).  This is to help identify
the cause of intermittent "postmaster exited during a parallel
transaction" failures seen on a couple of buildfarm members.  This
change degrades pg_ctl's functionality in a couple of minor ways,
so we'll revert it once we've obtained the desired info.

Discussion: https://postgr.es/m/18537.1570421268@sss.pgh.pa.us
2019-10-07 10:39:07 -04:00
Michael Paquier caa078353e Improve handling and coverage of --no-ensure-shutdown in pg_rewind
This includes a couple of changes around the new behavior of pg_rewind
which enforces recovery to happen once on a cluster not shut down
cleanly:
- Some comments and documentation improvements.
- Shutdown the cluster to rewind with immediate mode in all the tests,
this allows to check after the forced recovery behavior which is wanted
as new default.
- Use -F for the forced recovery step, so as postgres does not use
fsync.  This was useless as a final sync is done once the tool is done.

Author: Michael Paquier
Reviewed-by: Alexey Kondratov
Discussion: https://postgr.es/m/20191004083721.GA1829@paquier.xyz
2019-10-07 09:07:22 +09:00
Tom Lane d82f3909da Avoid use of wildcard in pg_waldump's .gitignore.
This would be all right, maybe, if it didn't also match a file that
definitely should not be ignored.  We don't add rmgrs so often that
manual maintenance of this file list is impractical, so just write
out the list.

(I find the equivalent wildcard use in the Makefile pretty lazy and
unsafe as well, but will leave that alone until it actually causes a
problem.)

Per bug #16042 from Denis Stuchalin.

Discussion: https://postgr.es/m/16042-c174ee692ac21cbd@postgresql.org
2019-10-05 12:26:55 -04:00
Michael Paquier 6837632b75 Fix issues in pg_rewind with --no-ensure-shutdown/--write-recovery-conf
This fixes two issues with recent features added in pg_rewind:
- --dry-run should do nothing on the target directory, but 927474c
forgot to consider that for --write-recovery-conf.
- --no-ensure-shutdown was not actually working.  There is no test
coverage for this option yet, but a subsequent patch will add that.

Author: Alexey Kondratov
Discussion: https://postgr.es/m/7ca88204-3e0b-2f4c-c8af-acadc4b266e5@postgrespro.ru
2019-10-04 16:18:29 +09:00
Michael Paquier 6f3823b035 Fix --dry-run mode of pg_rewind
Even if --dry-run mode was specified, the control file was getting
updated, preventing follow-up runs of pg_rewind to work properly on the
target data folder.  The origin of the problem came from the refactoring
done by ce6afc6.

Author: Alexey Kondratov
Discussion: https://postgr.es/m/7ca88204-3e0b-2f4c-c8af-acadc4b266e5@postgrespro.ru
Backpatch-through: 12
2019-10-04 09:14:51 +09:00
Amit Kapila b1c1aa5318 pgbench: add --partitions and --partition-method options.
These new options allow users to partition the pgbench_accounts table by
specifying the number of partitions and partitioning method.  The values
allowed for partitioning method are range and hash.

This feature allows users to measure the overhead of partitioning if any.

Author: Fabien COELHO
Reviewed-by: Amit Kapila, Amit Langote, Dilip Kumar, Asif Rehman, and
Alvaro Herrera
Discussion: https://postgr.es/m/alpine.DEB.2.21.1907230826190.7008@lancre
2019-10-03 08:27:07 +05:30
Andres Freund 5daf682cfc Fix pg_rewind link order issue introduced in 927474ce1a.
The aforementioned commit orders the link to pgfeutils after libpq,
which can fail because pgfeutils uses symbols from libpq.

Per buildfarm animal jacana.

Author: Andres Freund
Discussion: https://postgr.es/m/20190930192013.r3wievljua2n3tbb@alap3.anarazel.de
2019-09-30 12:43:09 -07:00
Alvaro Herrera 7524c78874 pg_rewind: test new --write-recovery-conf functionality
Author: Alexey Kondratov
Reviewed-by: Paul Guo
Discussion: https://postgr.es/m/2f726102-3f1e-bf16-061e-501919473ace@postgrespro.ru
2019-09-30 14:04:00 -03:00
Alvaro Herrera 927474ce1a pg_rewind: Allow writing recovery configuration
This is provided with a new switch --write-recovery-conf and reuses the
pg_basebackup code.

Author: Paul Guo, Jimmy Yih, Ashwin Agrawal
Reviewed-by: Alexey Kondratov, Michaël Paquier, Álvaro Herrera
Discussion: https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
2019-09-30 12:57:35 -03:00
Alvaro Herrera 5adafaf176 Have pg_rewind run crash recovery before rewinding
If we don't do this, the rewind fails if the server wasn't cleanly shut
down, which seems unhelpful serving no purpose.

Also provide a new option --no-ensure-shutdown to suppress this
behavior, for alleged advanced usage that prefers to avoid the crash
recovery.

Authors: Paul Guo, Jimmy Yih, Ashwin Agrawal
Reviewed-by: Álvaro Herrera
Discussion: https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
2019-09-27 16:40:01 -03:00
Michael Paquier 4b011cad27 Add tab completion for EXPLAIN (SETTINGS) in psql
Author: Justin Pryzby
Reviewed-by: Tatsuro Yamada
Discussion: https://postgr.es/m/20190927022051.GC24334@telsasoft.com
Backpatch-through: 12
2019-09-27 12:53:43 +09:00
Alvaro Herrera caba97a9d9 Split out recovery confing-writing code from pg_basebackup
... into a new file, fe_utils/recovery_gen.c.

This can later be used by pg_rewind.

Authors: Paul Guo, Jimmy Yih, Ashwin Agrawal.  A few tweaks by Álvaro Herrera
Reviewed-by: Michaël Paquier
Discussion: https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
2019-09-25 14:35:24 -03:00
Alvaro Herrera 709d003fbd Rework WAL-reading supporting structs
The state-tracking of WAL reading in various places was pretty messy,
mostly because the ancient physical-replication WAL reading code wasn't
using the XLogReader abstraction.  This led to some untidy code.  Make
it prettier by creating two additional supporting structs,
WALSegmentContext and WALOpenSegment which keep track of WAL-reading
state.  This makes code cleaner, as well as supports more future
cleanup.

Author: Antonin Houska
Reviewed-by: Álvaro Herrera and (older versions) Robert Haas
Discussion: https://postgr.es/m/14984.1554998742@spoje.net
2019-09-24 16:39:53 -03:00
Alvaro Herrera 32200c19dd pg_upgrade/test.sh: Quote sed(1) argument
Lack of quotes results in failure to run the test under older Solaris.

Author: Marina Polyakova, Victor Wagner
Discussion: https://postgr.es/m/feba89f89e8925b3535cb7d72b9e05e1@postgrespro.ru
2019-09-18 11:27:11 -03:00
Fujii Masao fc16778873 Add tab completion for CREATE OR REPLACE in psql.
Author: Shenhao Wang
Discussion: https://postgr.es/m/63580B24E208E3429D94153A03C68E0901AA8002D5@G08CNEXMBPEKD02.g08.fujitsu.local
2019-09-13 18:16:40 +09:00
Michael Paquier 8a0deae8d9 Fix comment in psql's describe.c
Procedures are supported since v11 and \dfp can be used since this
version, but it was not mentioned as a supported option in the
description of describeFunctions() which handles \df in psql.

Extracted from a larger patch.

Author: Fabien Coelho
Discussion: https://postgr.es/m/alpine.DEB.2.21.1908281618520.28828@lancre
2019-09-11 15:17:35 +09:00
Tomas Vondra d06215d03b Allow setting statistics target for extended statistics
When building statistics, we need to decide how many rows to sample and
how accurate the resulting statistics should be. Until now, it was not
possible to explicitly define statistics target for extended statistics
objects, the value was always computed from the per-attribute targets
with a fallback to the system-wide default statistics target.

That's a bit inconvenient, as it ties together the statistics target set
for per-column and extended statistics. In some cases it may be useful
to require larger sample / higher accuracy for extended statics (or the
other way around), but with this approach that's not possible.

So this commit introduces a new command, allowing to specify statistics
target for individual extended statistics objects, overriding the value
derived from per-attribute targets (and the system default).

  ALTER STATISTICS stat_name SET STATISTICS target_value;

When determining statistics target for an extended statistics object we
first look at this explicitly set value. When this value is -1, we fall
back to the old formula, looking at the per-attribute targets first and
then the system default. This means the behavior is backwards compatible
with older PostgreSQL releases.

Author: Tomas Vondra
Discussion: https://postgr.es/m/20190618213357.vli3i23vpkset2xd@development
Reviewed-by: Kirk Jamison, Dean Rasheed
2019-09-11 00:25:51 +02:00
Andrew Dunstan 73ff3a0abb Prevent msys2 conversion of "cmd /c" switch to a file path
Modern versions of msys2 have changed the treatment of "cmd /c" so that
the runtime will try to convert the switch to a native file path. This
patch adds a setting to inhibit that behaviour.

Discussion: https://postgr.es/m/3227042f-cfcc-745a-57dd-fb8c471f8ddf@2ndQuadrant.com

Backpatch to all live branches.
2019-09-09 09:04:07 -04:00
Peter Eisentraut c5bc7050af Message style fixes 2019-09-06 22:54:02 +02:00
Robert Haas 8b94dab066 Split tuptoaster.c into three separate files.
detoast.c/h contain functions required to detoast a datum, partially
or completely, plus a few other utility functions for examining the
size of toasted datums.

toast_internals.c/h contain functions that are used internally to the
TOAST subsystem but which (mostly) do not need to be accessed from
outside.

heaptoast.c/h contains code that is intrinsically specific to the
heap AM, either because it operates on HeapTuples or is based on the
layout of a heap page.

detoast.c and toast_internals.c are placed in
src/backend/access/common rather than src/backend/access/heap.  At
present, both files still have dependencies on the heap, but that will
be improved in a future commit.

Patch by me, reviewed and tested by Prabhat Sabu, Thomas Munro,
Andres Freund, and Álvaro Herrera.

Discussion: http://postgr.es/m/CA+TgmoZv-=2iWM4jcw5ZhJeL18HF96+W1yJeYrnGMYdkFFnEpQ@mail.gmail.com
2019-09-05 13:15:10 -04:00
Michael Paquier 522baf1484 Delay fsyncs of pg_basebackup until the end of backup
Since the addition of fsync requests in bc34223 to make base backup data
consistent on disk once pg_basebackup finishes, each tablespace tar file
is individually flushed once completed, with an additional flush of the
parent directory when the base backup finishes.  While holding a
connection to the server, a fsync request taking a long time may cause a
failure of the base backup, which is annoying for any integration.  A
recent example of breakage can involve tcp_user_timeout, but
wal_sender_timeout can cause similar problems.

While reviewing the code, there was a second issue causing too many
fsync requests to be done for the same WAL data.  As recursive fsyncs
are done at the end of the backup for both the plain and tar formats
from the base target directory where everything is written, it is fine
to disable fsyncs when fetching or streaming WAL.

Reported-by: Ryohei Takahashi
Author: Michael Paquier
Reviewed-by: Ryohei Takahashi
Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com
Backpatch-through: 10
2019-09-04 13:21:11 +09:00
Peter Eisentraut 10f5544896 Clarify pg_dump documentation
Clarify in the help output and documentation that -n, -t etc. take a
"pattern" rather than a "schema" or "table" etc.  This was especially
confusing now that the new pg_dumpall --exclude-database option was
documented with "pattern" and the others not, even though they all
behave the same.

Discussion: https://www.postgresql.org/message-id/flat/b85f3fa1-b350-38d1-1893-4f7911bd7310%402ndquadrant.com
2019-09-03 14:25:26 +02:00
Peter Eisentraut bde8c2d319 Improve base backup protocol documentation
Document that the tablespace sizes are in units of kilobytes.  Make
the pg_basebackup source code a bit clearer about this, too.

Reviewed-by: Magnus Hagander <magnus@hagander.net>
2019-09-03 11:59:36 +02:00
Peter Eisentraut 1d7a6e3eb4 pg_checksums: Handle read and write returns correctly
The read() return was not checking for errors, the write() return was
not checking for short writes.

Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://www.postgresql.org/message-id/flat/5de61b6b-8be9-7771-0048-860328efe027%402ndquadrant.com
2019-09-03 08:30:21 +02:00
Tom Lane aef3623858 Handle corner cases correctly in psql's reconnection logic.
After an unexpected connection loss and successful reconnection,
psql neglected to resynchronize its internal state about the server,
such as server version.  Ordinarily we'd be reconnecting to the same
server and so this isn't really necessary, but there are scenarios
where we do need to update --- one example is where we have a list
of possible connection targets and they're not all alike.

Define "resynchronize" as including connection_warnings(), so that
this case acts the same as \connect.  This seems useful; for example,
if the server version did change, the user might wish to know that.
An attuned user might also notice that the new connection isn't
SSL-encrypted, for example, though this approach isn't especially
in-your-face about such changes.  Although this part is a behavioral
change, it only affects interactive sessions, so it should not break
any applications.

Also, in do_connect, make sure that we desynchronize correctly when
abandoning an old connection in non-interactive mode.

These problems evidently are the result of people patching only one
of the two places where psql deals with connection changes, so insert
some cross-referencing comments in hopes of forestalling future bugs
of the same ilk.

Lastly, in Windows builds, issue codepage mismatch warnings only at
startup, not during reconnections.  psql's codepage can't change
during a reconnect, so complaining about it again seems like useless
noise.

Peter Billen and Tom Lane.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/CAMTXbE8e6U=EBQfNSe01Ej17CBStGiudMAGSOPaw-ALxM-5jXg@mail.gmail.com
2019-09-02 14:03:05 -04:00
Peter Eisentraut 9684e42695 Error out on too many command-line arguments
Fix up oid2name, pg_upgrade, and pgbench to error out on too many
command-line arguments.  This makes it match the behavior of other
PostgreSQL programs.

Author: Peter Eisentraut, Ibrar Ahmed
Discussion: https://www.postgresql.org/message-id/flat/f2554627-04e7-383a-ef01-ab99bb6a291c%402ndquadrant.com
2019-08-29 18:49:41 +02:00
Michael Paquier be182e4f9e Disable timeouts when running pg_rewind with online source cluster
In this case, the transfer uses a libpq connection, which is subject to
the timeout parameters set at system level, and this can make the rewind
operation suddenly canceled which is not good for automation.  One
workaround to such issues would be to use PGOPTIONS to enforce the
wanted timeout parameters, but that's annoying, and for example pg_dump,
which can run potentially long-running queries disables all types of
timeouts.

lock_timeout and statement_timeout are the ones which can cause problems
now.  Note that pg_rewind does not use transactions, so disabling
idle_in_transaction_session_timeout is optional, but it feels safer to
do so for the future.

This is back-patched down to 9.5.  idle_in_transaction_session_timeout
is only present since 9.6.

Author: Alexander Kukushkin
Discussion: https://postgr.es/m/CAFh8B=krcVXksxiwVQh1SoY+ziJ-JC=6FcuoBL3yce_40Es5_g@mail.gmail.com
Backpatch-through: 9.5
2019-08-28 11:47:35 +09:00
Tom Lane d4b2425441 Add missing newline in help output.
Daniel Gustafsson

Discussion: https://postgr.es/m/F2FB03F2-B112-4E51-842E-12C50DCA2F4A@yesql.se
2019-08-27 15:14:55 -04:00
Michael Paquier 9acda73118 Fix failure of --jobs with reindexdb and vacuumdb on Windows
FD_SETSIZE needs to be declared before winsock2.h, or it is possible to
run into buffer overflow issues when using --jobs.  This is similar to
pgbench's solution done in a23c641.

This has been introduced by 71d84ef, and older versions have been using
the default value of FD_SETSIZE, defined at 64.

Per buildfarm member jacana, but this impacts all Windows animals
running the TAP tests.  I have reproduced the failure locally to check
the patch.

Author: Michael Paquier
Reviewed-by: Andrew Dunstan
Discussion: https://postgr.es/m/20190826054000.GE7005@paquier.xyz
Backpatch-through: 9.5
2019-08-27 09:11:31 +09:00
Peter Eisentraut f269033881 Fix gettext triggers specification
In cc8d415117, the arguments of
warn_or_exit_horribly() were changed but this was not updated.
2019-08-26 19:06:01 +02:00
Andrew Dunstan c62b84437f Adjust to latest Msys2 kernel release number
Previously 'uname -r' on Msys2 reported a kernele release starting with
2. The latest version starts with 3. In commit 1638623f we specifically
looked for one starting with 2. This is now changed to look for any
digit between 2 and 9.

backpatch to release 10.
2019-08-26 08:11:27 -04:00
Andrew Dunstan acb96eb7d2 Treat MINGW and MSYS the same in pg_upgrade test script
On msys2, 'uname -s' reports a string starting MSYS instead on MINGW
as happens on msys1. Treat these both the same way. This reverts
608a710195 in favor of a more general solution.

Backpatch to all live branches.
2019-08-26 07:44:34 -04:00
Michael Paquier 71d84efba7 Fix error handling of vacuumdb and reindexdb when running out of fds
When trying to use a high number of jobs, vacuumdb (and more recently
reindexdb) has only checked for a maximum number of jobs used, causing
confusing failures when running out of file descriptors when the jobs
open connections to Postgres.  This commit changes the error handling so
as we do not check anymore for a maximum number of allowed jobs when
parsing the option value with FD_SETSIZE, but check instead if a file
descriptor is within the supported range when opening the connections
for the jobs so as this is detected at the earliest time possible.

Also, improve the error message to give a hint about the number of jobs
recommended, using a wording given by the reviewers of the patch.

Reported-by: Andres Freund
Author: Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera, Tom Lane
Discussion: https://postgr.es/m/20190818001858.ho3ev4z57fqhs7a5@alap3.anarazel.de
Backpatch-through: 9.5
2019-08-26 11:14:18 +09:00
Tom Lane 6338fa3e71 Avoid platform-specific null pointer dereference in psql.
POSIX permits getopt() to advance optind beyond argc when the last
argv entry is an option that requires an argument and hasn't got one.
It seems that no major platforms actually do that, but musl does,
so that something like "psql -f" would crash with that libc.
Add a check that optind is in range before trying to look at the
possibly-bogus option.

Report and fix by Quentin Rameau.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/20190825100617.GA6087@fifth.space
2019-08-25 15:04:04 -04:00
Michael Paquier 56f8f9624b Fix compilation failure of vacuumdb and reindexdb with OpenBSD
FD_SETSIZE is included in sys/select.h per POSIX, and this header
inclusion has been moved to scripts_parallel.c as of 5f38403 without
moving the variable, causing a compilation failure on recent versions of
OpenBSD (6.6 was the version used in the report).

In order to take care of the failure, move FD_SETSIZE directly to
scripts_parallel.c with a wrapper controlling the maximum number of
parallel slots supported, based on a suggestion by Andres Freund.

While on it, reduce the maximum number to be less than FD_SETSIZE,
leaving some room for stdin, stdout and such as they consume some file
descriptors.

The buildfarm did not complain about that, as it happens to only be
an issue on recent versions of OpenBSD and there is no coverage in this
area.  51c3e9f fixed a similar set of issues.

Bug: #15964
Reported-by: Sean Farrell
Discussion: https://postgr.es/m/15964-c1753bdfed722e04@postgresql.org
2019-08-20 16:10:20 +09:00
Michael Paquier 71851e9ab7 Fix tab completion for CREATE TYPE in psql
Oversight in 7bdc655.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/5da8e325-c665-da95-21e0-c8a99ea61fbf@gmail.com
2019-08-19 16:33:24 +09:00
Michael Paquier c96581abe4 Fix inconsistencies and typos in the tree, take 11
This fixes various typos in docs and comments, and removes some orphaned
definitions.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/5da8e325-c665-da95-21e0-c8a99ea61fbf@gmail.com
2019-08-19 16:21:39 +09:00
Tom Lane 5f110933e1 Fix failure-to-compile-standalone in scripts_parallel.h.
Needs libpq-fe.h for references to PGConn.

Discussion: https://postgr.es/m/17463.1566153454@sss.pgh.pa.us
2019-08-18 18:01:01 -04:00
Peter Eisentraut fded4773eb initdb: Remove obsolete locale handling
The method of passing LC_COLLATE and LC_CTYPE to the backend during
initdb is obsolete as of 61d9674988.
This can all be removed.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/eeaf2f99-a1a6-8aca-3f43-9ab0b2fb112a%402ndquadrant.com
2019-08-14 06:51:13 +02:00
Tom Lane 31d43710fb Un-break pg_dump for pre-8.3 source servers.
Commit 07b39083c inserted an unconditional reference to pg_opfamily,
which of course fails on servers predating that catalog.  Fortunately,
the case it's trying to solve can't occur on such old servers (AFAIK).
Hence, just skip the additional code when the source predates 8.3.

Per bug #15955 from sly.  Back-patch to all supported branches,
like the previous patch.

Discussion: https://postgr.es/m/15955-1daa2e676e903d87@postgresql.org
2019-08-13 16:58:32 -04:00
Michael Paquier 66bde49d96 Fix inconsistencies and typos in the tree, take 10
This addresses some issues with unnecessary code comments, fixes various
typos in docs and comments, and removes some orphaned structures and
definitions.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/9aabc775-5494-b372-8bcb-4dfc0bd37c68@gmail.com
2019-08-13 13:53:41 +09:00
Michael Paquier b8f2da0ac5 Refactor logic to remove trailing CR/LF characters from strings
b654714 has reworked the way trailing CR/LF characters are removed from
strings.  This commit introduces a new routine in common/string.c and
refactors the code so as the logic is in a single place, mostly.

Author: Michael Paquier
Reviewed-by: Bruce Momjian
Discussion: https://postgr.es/m/20190801031820.GF29334@paquier.xyz
2019-08-09 11:05:14 +09:00
Peter Eisentraut 43211c2a02 initdb: Use varargs macro for PG_CMD_PRINTF
I left PG_CMD_PUTS around even though it could be handled by
PG_CMD_PRINTF since PG_CMD_PUTS is sometimes called with non-literal
arguments, and so that would create a potential problem if such a
string contained percent signs.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
2019-08-08 08:47:55 +02:00
Michael Paquier 05ba8370b8 Fix tab completion for ALTER LANGUAGE in psql
OWNER_TO was used for the completion, which is not a supported grammar,
but OWNER TO is.

This error has been introduced by d37b816, so backpatch down to 9.6.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/7ab243e0-116d-3e44-d120-76b3df7abefd@gmail.com
Backpatch-through: 9.6
2019-08-05 14:27:20 +09:00
Michael Paquier 8548ddc61b Fix inconsistencies and typos in the tree, take 9
This addresses more issues with code comments, variable names and
unreferenced variables.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/7ab243e0-116d-3e44-d120-76b3df7abefd@gmail.com
2019-08-05 12:14:58 +09:00
Tom Lane 07b39083c2 Fix pg_dump's handling of dependencies for custom opclasses.
Since pg_dump doesn't treat the member operators and functions of operator
classes/families (that is, the pg_amop and pg_amproc entries, not the
underlying operators/functions) as separate dumpable objects, it missed
their dependency information.  I think this was safe when the code was
designed, because the default object sorting rule emits operators and
functions before opclasses, and there were no dependency types that could
mess that up.  However, the introduction of range types in 9.2 broke it:
now a type can have a dependency on an opclass, allowing dependency rules
to push the opclass before the type and hence before custom operators.
Lacking any information showing that it shouldn't do so, pg_dump emitted
the objects in the wrong order.

Fix by teaching getDependencies() to translate pg_depend entries for
pg_amop/amproc rows to look like dependencies for their parent opfamily.

I added a regression test for this in HEAD/v12, but not further back;
life is too short to fight with 002_pg_dump.pl.

Per bug #15934 from Tom Gottfried.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/15934-58b8c8ab7a09ea15@postgresql.org
2019-07-31 15:42:49 -04:00
Heikki Linnakangas d8b094dabb Print WAL position correctly in pg_rewind error message.
This has been wrong ever since pg_rewind was added. The if-branch just
above this, where we print the same error with an extra message supplied
by XLogReadRecord() got this right, but the variable name was wrong in the
else-branch. As a consequence, the error printed the WAL position as
0/0 if there was an error reading a WAL file.

Backpatch to 9.5, where pg_rewind was added.
2019-07-30 21:14:14 +03:00
Michael Paquier 04cf0bfc90 Fix memory leak coming from simple lists built in reindexdb
When building a list of relations for a parallel processing of a schema
or a database (or just a single-entry list for the non-parallel case
with the database name), the list is allocated and built on-the-fly for
each database processed, leaking after one database-level reindex is
done.  This accumulates leaks when processing all databases, and could
become a visible issue with thousands of relations.

This is fixed by introducing a new routine in simple_list.c to free all
the elements in a simple list made of strings or OIDs.  The header of
the list may be using a variable declaration or an allocated pointer,
so we don't have a routine to free this part to keep the interface
simple.

Per report from coverity for an issue introduced by 5ab892c, and
valgrind complains about the leak as well.  The idea to introduce a new
routine in simple_list.c is from Tom Lane.

Author: Michael Paquier
Reviewed-by: Tom Lane
2019-07-30 10:54:48 +09:00
Peter Eisentraut 1e2fddfa33 Handle fsync failures in pg_receivewal and pg_recvlogical
It is not safe to simply report an fsync error and continue.  We must
exit the program instead.

Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sehrope Sarkuni <sehrope@jackdb.com>
Discussion: https://www.postgresql.org/message-id/flat/9b49fe44-8f3e-eca9-5914-29e9e99030bf@2ndquadrant.com
2019-07-29 07:54:57 +02:00
Michael Paquier eb43f3d193 Fix inconsistencies and typos in the tree
This is numbered take 8, and addresses again a set of issues with code
comments, variable names and unreferenced variables.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/b137b5eb-9c95-9c2f-586e-38aba7d59788@gmail.com
2019-07-29 12:28:30 +09:00
Michael Paquier 5ab892c391 Add support for --jobs in reindexdb
When doing a schema-level or a database-level operation, a list of
relations to build is created which gets processed in parallel using
multiple connections, based on the recent refactoring for parallel slots
in src/bin/scripts/.  System catalogs are processed first in a
serialized fashion to prevent deadlocks, followed by the rest done in
parallel.

This new option is not compatible with --system as reindexing system
catalogs in parallel can lead to deadlocks, and with --index as there is
no conflict handling for indexes rebuilt in parallel depending in the
same relation.

Author: Julien Rouhaud
Reviewed-by: Sergei Kornilov, Michael Paquier
Discussion: https://postgr.es/m/CAOBaU_YrnH_Jqo46NhaJ7uRBiWWEcS40VNRQxgFbqYo9kApUsg@mail.gmail.com
2019-07-27 22:21:18 +09:00
Peter Eisentraut 959f6d6a18 pg_upgrade: Default new bindir to pg_upgrade location
Make the directory where the pg_upgrade binary resides the default for
new bindir, as running the pg_upgrade binary from where the new
cluster is installed is a very common scenario.  Setting this as the
defauly bindir for the new cluster will remove the need to provide it
explicitly via -B in many cases.

To support directories being missing from option parsing, extend the
directory check with a missingOk mode where the path must be filled at
a later point before being used.  Also move the exec_path check to
earlier in setup to make sure we know the new cluster bindir when we
scan for required executables.

This removes the exec_path from the OSInfo struct as it is not used
anywhere.

Author: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/9328.1552952117@sss.pgh.pa.us
2019-07-27 08:19:04 +02:00
Peter Eisentraut 0befb4f313 pg_upgrade: Check all used executables
Expand the validate_exec() calls to cover all the used binaries.

Author: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/9328.1552952117@sss.pgh.pa.us
2019-07-27 07:48:08 +02:00
Peter Eisentraut 28cb0555c1 Fix typo in pg_upgrade file header
Author: Daniel Gustafsson <daniel@yesql.se>
2019-07-27 07:46:02 +02:00
Alvaro Herrera 0994cfc0ac Don't uselessly escape a string that doesn't need escaping
Per gripe from Ian Barwick

Co-authored-by: Ian Barwick <ian@2ndquadrant.com>
Discussion: https://postgr.es/m/CABvVfJWNnNKb8cHsTLhkTsvL1+G6BVcV+57+w1JZ61p8YGPdWQ@mail.gmail.com
2019-07-26 17:46:40 -04:00
Tom Lane 3754113f33 Avoid choosing "localtime" or "posixrules" as TimeZone during initdb.
Some platforms create a file named "localtime" in the system
timezone directory, making it a copy or link to the active time
zone file.  If Postgres is built with --with-system-tzdata, initdb
will see that file as an exact match to localtime(3)'s behavior,
and it may decide that "localtime" is the most preferred spelling of
the active zone.  That's a very bad choice though, because it's
neither informative, nor portable, nor stable if someone changes
the system timezone setting.  Extend the preference logic added by
commit e3846a00c so that we will prefer any other zone file that
matches localtime's behavior over "localtime".

On the same logic, also discriminate against "posixrules", which
is another not-really-a-zone file that is often present in the
timezone directory.  (Since we install "posixrules" but not
"localtime", this change can affect the behavior of Postgres
with or without --with-system-tzdata.)

Note that this change doesn't prevent anyone from choosing these
pseudo-zones if they really want to (i.e., by setting TZ for initdb,
or modifying the timezone GUC later on).  It just prevents initdb
from preferring these zone names when there are multiple matches to
localtime's behavior.

Since we generally prefer to keep timezone-related behavior the
same in all branches, and since this is arguably a bug fix,
back-patch to all supported branches.

Discussion: https://postgr.es/m/CADT4RqCCnj6FKLisvT8tTPfTP4azPhhDFJqDF1JfBbOH5w4oyQ@mail.gmail.com
Discussion: https://postgr.es/m/27991.1560984458@sss.pgh.pa.us
2019-07-26 12:45:32 -04:00
Tom Lane b654714f9b Fix failures to ignore \r when reading Windows-style newlines.
libpq failed to ignore Windows-style newlines in connection service files.
This normally wasn't a problem on Windows itself, because fgets() would
convert \r\n to just \n.  But if libpq were running inside a program that
changes the default fopen mode to binary, it would see the \r's and think
they were data.  In any case, it's project policy to ignore \r in text
files unconditionally, because people sometimes try to use files with
DOS-style newlines on Unix machines, where the C library won't hide that
from us.

Hence, adjust parseServiceFile() to ignore \r as well as \n at the end of
the line.  In HEAD, go a little further and make it ignore all trailing
whitespace, to match what it's always done with leading whitespace.

In HEAD, also run around and fix up everyplace where we have
newline-chomping code to make all those places look consistent and
uniformly drop \r.  It is not clear whether any of those changes are
fixing live bugs.  Most of the non-cosmetic changes are in places that
are reading popen output, and the jury is still out as to whether popen
on Windows can return \r\n.  (The Windows-specific code in pipe_read_line
seems to think so, but our lack of support for this elsewhere suggests
maybe it's not a problem in practice.)  Hence, I desisted from applying
those changes to back branches, except in run_ssl_passphrase_command()
which is new enough and little-tested enough that we'd probably not have
heard about any problems there.

Tom Lane and Michael Paquier, per bug #15827 from Jorge Gustavo Rocha.
Back-patch the parseServiceFile() change to all supported branches,
and the run_ssl_passphrase_command() change to v11 where that was added.

Discussion: https://postgr.es/m/15827-e6ba53a3a7ed543c@postgresql.org
2019-07-25 12:11:17 -04:00
Tom Lane 4e784f3514 Fix infelicities in describeOneTableDetails' partitioned-table handling.
describeOneTableDetails issued a partition-constraint-fetching query
for every table, even ones it knows perfectly well are not partitions.

To add insult to injury, it then proceeded to leak the empty PGresult
if the table wasn't a partition.  Doing that a lot of times might
amount to a meaningful leak, so this seems like a back-patchable bug.

Fix that, and also fix a related PGresult leak in the partition-parent
case (though that leak would occur only if we got no row, which is
unexpected).

Minor code beautification too, to make this code look more like the
pre-existing code around it.

Back-patch the whole change into v12.  However, the fact that we already
know whether the table is a partition dates only to commit 1af25ca0c;
back-patching the relevant changes from that is probably more churn
than is justified in released branches.  Hence, in v11 and v10, just
do the minimum to fix the PGresult leaks.

Noted while messing around with adjacent code for yesterday's \d
improvements.
2019-07-24 18:14:43 -04:00
Andrew Dunstan 1a721248f3 Don't assume expr is available in pgbench tests
Windows hosts do not normally come with expr, so instead of using that
to test the \setshell command, use echo instead, which is fairly
universally available.

Backpatch to release 11, where this came in.

Problem found by me, patch by Fabien Coelho.
2019-07-24 11:41:39 -04:00
Tom Lane 24f62e93f3 Improve psql's \d output for partitioned indexes.
Include partitioning information much as we do for partitioned tables.
(However, \d+ doesn't show the partition bounds, because those are
not stored for indexes.)

In passing, fix a couple of queries to look less messy in -E output.

Also, add some tests for \d on tables with nondefault tablespaces.
(Somebody previously added a rather silly number of tests for \d
on partitioned indexes, yet completely neglected other cases.)

Justin Pryzby, reviewed by Fabien Coelho

Discussion: https://postgr.es/m/20190422154902.GH14223@telsasoft.com
2019-07-23 17:04:21 -04:00
Tom Lane eb5472da9f Improve psql's \d output for TOAST tables.
Add the name of the owning table to the footers for a TOAST table.
Also, show all the same footers as for a regular table (in practice,
this adds the index and perhaps the tablespace and access method).

Justin Pryzby, reviewed by Fabien Coelho

Discussion: https://postgr.es/m/20190422154902.GH14223@telsasoft.com
2019-07-23 15:25:56 -04:00
Peter Eisentraut 06140c201b Add CREATE DATABASE LOCALE option
This sets both LC_COLLATE and LC_CTYPE with one option.  Similar
behavior is already supported in initdb, CREATE COLLATION, and
createdb.

Reviewed-by: Fabien COELHO <coelho@cri.ensmp.fr>
Discussion: https://www.postgresql.org/message-id/flat/d9d5043a-dc70-da8a-0166-1e218e6e34d4%402ndquadrant.com
2019-07-23 14:47:24 +02:00
Michael Paquier 3cae75f420 Remove more progname references in vacuumdb.c
Oversight in 5f384037.

Author: Álvaro Herrera
Discussion: https://postgr.es/m/20190722151806.GA22634@alvherre.pgsql
2019-07-23 14:29:34 +09:00
Tom Lane a0555ddab9 Install dependencies to prevent dropping partition key columns.
The logic in ATExecDropColumn that rejects dropping partition key
columns is quite an inadequate defense, because it doesn't execute
in cases where a column needs to be dropped due to cascade from
something that only the column, not the whole partitioned table,
depends on.  That leaves us with a badly broken partitioned table;
even an attempt to load its relcache entry will fail.

We really need to have explicit pg_depend entries that show that the
column can't be dropped without dropping the whole table.  Hence,
add those entries.  In v12 and HEAD, bump catversion to ensure that
partitioned tables will have such entries.  We can't do that in
released branches of course, so in v10 and v11 this patch affords
protection only to partitioned tables created after the patch is
installed.  Given the lack of field complaints (this bug was found
by fuzz-testing not by end users), that's probably good enough.

In passing, fix ATExecDropColumn and ATPrepAlterColumnType
messages to be more specific about which partition key column
they're complaining about.

Per report from Manuel Rigger.  Back-patch to v10 where partitioned
tables were added.

Discussion: https://postgr.es/m/CA+u7OA4JKCPFrdrAbOs7XBiCyD61XJxeNav4LefkSmBLQ-Vobg@mail.gmail.com
Discussion: https://postgr.es/m/31920.1562526703@sss.pgh.pa.us
2019-07-22 14:55:40 -04:00
Peter Eisentraut 7961886580 Revert "initdb: Change authentication defaults"
This reverts commit 09f08930f0.

The buildfarm client needs some adjustments first.
2019-07-22 19:28:25 +02:00
Peter Eisentraut 09f08930f0 initdb: Change authentication defaults
Change the defaults for the pg_hba.conf generated by initdb to "peer"
for local (if supported, else "md5") and "md5" for host.

(Changing from "md5" to SCRAM is left as a separate exercise.)

"peer" is currently not supported on AIX, HP-UX, and Windows.  Users
on those operating systems will now either have to provide a password
to initdb or choose a different authentication method when running
initdb.

Reviewed-by: Julien Rouhaud <rjuju123@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/bec17f0a-ddb1-8b95-5e69-368d9d0a3390%40postgresql.org
2019-07-22 15:14:27 +02:00
Michael Paquier 90317ab7e6 Fix compilation warning of pg_basebackup with MinGW
Several buildfarm members have been complaining about that with gcc,
like jacana.  Weirdly enough, Visual Studio's compilers do not find this
issue.

Author: Michael Paquier
Reviewed-by: Andrew Dunstan
Discussion: https://postgr.es/m/20190719050830.GK1859@paquier.xyz
2019-07-21 22:27:11 +09:00
Michael Paquier 5f3840370b Refactor parallelization processing code in src/bin/scripts/
The existing facility of vacuumdb to handle parallel connections into a
given database with an authentication set is moved to a common file in
src/bin/scripts/, named scripts_parallel.c.  This introduces a set of
routines to initialize, wait and terminate a set of connections,
simplifying a bit the code of vacuumdb on the way.  More routines
related to result handling and database connection are moved to
common.c.

The initial plan is to use that for reindexdb, but it could be applied
to other tools like clusterdb.

While on it, clean up a set of variables "progname" which were defined
as routine arguments for error messages.  Since most of the callers have
switched to pg_log_error() and such there is no need for this variable.

Author: Julien Rouhaud
Reviewed-by: Michael Paquier, Álvaro Herrera
Discussion: https://postgr.es/m/CAOBaU_YrnH_Jqo46NhaJ7uRBiWWEcS40VNRQxgFbqYo9kApUsg@mail.gmail.com
2019-07-19 09:31:58 +09:00
Michael Paquier 0896ae561b Fix inconsistencies and typos in the tree
This is numbered take 7, and addresses a set of issues around:
- Fixes for typos and incorrect reference names.
- Removal of unneeded comments.
- Removal of unreferenced functions and structures.
- Fixes regarding variable name consistency.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/10bfd4ac-3e7c-40ab-2b2e-355ed15495e8@gmail.com
2019-07-16 13:23:53 +09:00
Thomas Munro 5823677acc Provide pgbench --show-script to dump built-in scripts.
Author: Fabien Coelho
Reviewed-by: Ibrar Ahmed
Discussion: https://postgr.es/m/alpine.DEB.2.21.1904081737390.5867%40lancre
2019-07-16 11:57:49 +12:00
Thomas Munro ce8f946764 Report the time taken by pgbench initialization steps.
Author: Fabien Coelho
Reviewed-by: Ibrar Ahmed
Discussion: https://postgr.es/m/alpine.DEB.2.21.1904061810510.3678%40lancre
2019-07-16 11:37:37 +12:00
Noah Misch 8a0cbb8852 Revive test of concurrent OID generation.
Commit 578b229718 replaced it with a
concurrent "nextval" test.  That version does not detect PostgreSQL's
incompatibility with xlc 13.1.3, so bring back an OID-based test that
does.  Back-patch to v12, where that commit first appeared.

Discussion: https://postgr.es/m/20190707170035.GA1485546@rfd.leadboat.com
2019-07-13 13:34:22 -07:00
Thomas Munro 5b51bbfbd5 Fix tab completion for UPDATE.
Previously it suggested an extra "=" after "SET x=".

Reported-by: Kyotaro Horiguchi
Discussion: https://postgr.es/m/CA%2BhUKGLk%3D0yLDjfviONJLzcHEzygj%3Dx6VbGH43LnXbBUvQb52g%40mail.gmail.com
2019-07-13 16:08:13 +12:00
Thomas Munro 7bdc6556fb Tab completion for CREATE TYPE.
Author: Thomas Munro
Reviewed-by: Kyotaro Horiguchi
Discussion: https://postgr.es/m/CA%2BhUKGLk%3D0yLDjfviONJLzcHEzygj%3Dx6VbGH43LnXbBUvQb52g%40mail.gmail.com
2019-07-13 16:07:47 +12:00
Peter Eisentraut e435c1e7d9 Message style improvements 2019-07-09 15:47:09 +02:00
Bruce Momjian 481837783b Remove unused C structure member
Remove quote_all_identifiers from struct _dumpOptions.

Discussion: https://postgr.es/m/d3d92ce9-78a4-8adb-0393-d3deeec29f7e@postgrespro.ru

Author: Arthur Zakirov

Backpatch-through: head
2019-07-08 19:31:16 -04:00
Michael Paquier 6b8548964b Fix inconsistencies in the code
This addresses a couple of issues in the code:
- Typos and inconsistencies in comments and function declarations.
- Removal of unreferenced function declarations.
- Removal of unnecessary compile flags.
- A cleanup error in regressplans.sh.

Author: Alexander Lakhin
Discussion: https://postgr.es/m/0c991fdf-2670-1997-c027-772a420c4604@gmail.com
2019-07-08 13:15:09 +09:00
Michael Paquier d1a040543b Remove more unreferenced function declarations
Author: Masahiko Sawada
Discussion: https://postgr.es/m/CAD21AoDuAYsRb3Q9aobkFZ6DZMWxsyg4HOmgkwgeWNfSkTwGxw@mail.gmail.com
2019-07-07 09:58:33 +09:00
Peter Eisentraut 3f3542621f psql: Fix logging output format
In normal interactive mode, psql's log messages accidentally got a
"psql:" prefix that was not supposed to be there.  This only happened
if there was no .psqlrc file being read, so it wasn't discovered for a
while.  Fix this by adding the appropriate logging format
configuration call in the right code path.

Discussion: https://www.postgresql.org/message-id/7586.1560540361@sss.pgh.pa.us
2019-07-06 15:03:17 +02:00
Tom Lane 02e95a5049 Add \warn command to psql.
This is like \echo except that the text is sent to stderr not stdout.

In passing, fix a pre-existing bug in \echo and \qecho: per documentation
the -n switch should only be recognized when it is the first argument,
but actually any argument matching "-n" was treated as a switch.
(Should we back-patch that?)

David Fetter (bug fix by me), reviewed by Fabien Coelho

Discussion: https://postgr.es/m/20190421183115.GA4311@fetter.org
2019-07-05 12:32:36 -04:00
Peter Eisentraut 6a1cd8b923 Unwind some workarounds for lack of portable int64 format specifier
Because there is no portable int64/uint64 format specifier and we
can't stick macros like INT64_FORMAT into the middle of a translatable
string, we have been using various workarounds that put the number to
be printed into a string buffer first.  Now that we always use our own
sprintf(), we can rely on %lld and %llu to work, so we can use those.

This patch undoes this workaround in a few places where it was
egregiously verbose.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/CAH2-Wz%3DWbNxc5ob5NJ9yqo2RMJ0q4HXDS30GVCobeCvC9A1L9A%40mail.gmail.com
2019-07-04 17:01:43 +02:00
Peter Eisentraut 7b925e1270 Sync our Snowball stemmer dictionaries with current upstream
The main change is a new stemmer for Greek.  There are minor changes
in the Danish and French stemmers.

Author: Panagiotis Mavrogiorgos <pmav99@gmail.com>
2019-07-04 13:26:48 +02:00
Michael Paquier d5ab9a891c Simplify TAP tests of pg_dump for connection strings
The last set of scenarios did an initialization of nodes followed by an
extra command to set up the authentication policy with pg_regress
--config-auth.  This configuration step can be integrated directly using
the option auth_extra from PostgresNode::init when initializing the
node, saving from one extra command.  On Windows, this also restricts
more pg_ident.conf for the SSPI user mapping by removing the entry of
the OS user running the test, which is not needed anyway.

Note that IPC::Run mishandles double quotes, hence the restore user name
is changed to map with that.  This was already done in the test as a
later step, but not in a consistent way, causing the switch to use
auth_extra to fail.

Found while reviewing ca129e5.

Discussion: https://postgr.es/m/20190703062024.GD3084@paquier.xyz
2019-07-04 11:33:42 +09:00
David Rowley 8abc13a889 Use appendStringInfoString and appendPQExpBufferStr where possible
This changes various places where appendPQExpBuffer was used in places
where it was possible to use appendPQExpBufferStr, and likewise for
appendStringInfo and appendStringInfoString.  This is really just a
stylistic improvement, but there are also small performance gains to be
had from doing this.

Discussion: http://postgr.es/m/CAKJS1f9P=M-3ULmPvr8iCno8yvfDViHibJjpriHU8+SXUgeZ=w@mail.gmail.com
2019-07-04 13:01:13 +12:00
Tom Lane 9a2ea61832 Show table persistence in psql's \dt+ and related commands.
In verbose mode, listTables() now emits a "Persistence" column
showing whether the table/index/view/etc is permanent, temporary,
or unlogged.

David Fetter, reviewed by Fabien Coelho and Rafia Sabih

Discussion: https://postgr.es/m/20190423005642.GZ28936@fetter.org
2019-07-03 11:46:34 -04:00
Peter Eisentraut 84c41ae81b Fix accidentally swapped error message arguments
Author: Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-07-02 23:44:30 +01:00
Peter Eisentraut 24c7000f64 Remove redundant newlines from error messages
These are no longer needed/allowed with the new logging API.
2019-07-02 23:18:43 +01:00
Tom Lane b4771d7c7f Don't treat complete_from_const as equivalent to complete_from_list.
Commit 4f3b38fe2 supposed that complete_from_const() is equivalent to
the one-element-list case of complete_from_list(), but that's not
really true at all.  complete_from_const() supposes that the completion
is certain enough to justify wiping out whatever the user typed, while
complete_from_list() will only provide completions that match the
word-so-far.

In practice, given the lame parsing technology used by tab-complete.c,
it's fairly hard to believe that we're *ever* certain enough about
a completion to justify auto-correcting user input that doesn't match.

Hence, remove the inappropriate unification of the two cases.
As things now stand, complete_from_const() is used only for the
situation where we have no matches and we need to keep readline
from applying its default complete-with-file-names behavior.

This (mis?) behavior actually exists much further back, but
I'm hesitant to change it in released branches.  It's not too
late for v12, though, especially seeing that the aforesaid
commit is new in v12.

Per gripe from Ken Tanzer.

Discussion: https://postgr.es/m/CAD3a31XpXzrZA9TT3BqLSHghdTK+=cXjNCE+oL2Zn4+oWoc=qA@mail.gmail.com
2019-07-02 14:04:42 -04:00
Tom Lane 0ec3e13c69 Fix tab completion of "SET variable TO|=" to not offer bogus completions.
Don't think that the context "UPDATE tab SET var =" is a GUC-setting
command.

If we have "SET var =" but the "var" is not a known GUC variable,
don't offer any completions.  The most likely explanation is that
we've misparsed the context and it's not really a GUC-setting command.

Per gripe from Ken Tanzer.  Back-patch to 9.6.  The issue exists
further back, but before 9.6 the code looks very different and it
doesn't actually know whether the "var" name matches anything,
so I desisted from trying to fix it.

Discussion: https://postgr.es/m/CAD3a31XpXzrZA9TT3BqLSHghdTK+=cXjNCE+oL2Zn4+oWoc=qA@mail.gmail.com
2019-07-02 13:35:14 -04:00
Tom Lane 4d6603f28d Simplify psql \d's rule for ordering the indexes of a table.
The previous rule was "primary key (if any) first, then other unique
indexes in name order, then all other indexes in name order".
But the preference for unique indexes seems a bit obsolete since the
introduction of exclusion constraints.   It's no longer the case
that unique indexes are the only ones that constrain what data can
be in the table, and it's hard to see what other rationale there is
for separating out unique indexes.  Other new features like the
possibility for some indexes to be INVALID (hence, not constraining
anything) make this even shakier.

Hence, simplify the sort order to be "primary key (if any) first,
then all other indexes in name order".

No documentation change, since this was never documented anyway.
A couple of existing regression test cases change output, though.

Discussion: https://postgr.es/m/14422.1561474929@sss.pgh.pa.us
2019-07-02 12:32:49 -04:00
Michael Paquier 9adda24543 Refactor code of reindexdb for query generation
This merges the portion related to REINDEX SYSTEM into the routine
already available for all the other reindex types, making the query
generation cleaner.  While on it, change the handling of the reindex
types using an enum, which allows to get rid of the hardcoded strings
used directly in the query generation present for the same purpose (aka
"TABLE", "DATABASE", etc.).

Per discussion with Julien Rouhaud, Tom Lane, Alvaro Herrera and me.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/CAOBaU_bSmSik_WRK9niDnm-3NkNZky6+uKxkmQwvthZvMWpS5A@mail.gmail.com
2019-07-02 11:36:53 +09:00
Tom Lane 9e1c9f9594 pgindent run prior to branching v12.
pgperltidy and reformat-dat-files too, though the latter didn't
find anything to change.
2019-07-01 12:37:52 -04:00
Peter Eisentraut 95bbe5d82e Convert some stragglers to new frontend logging API 2019-07-01 13:34:31 +02:00
Peter Eisentraut 1b29e990e3 Add missing serial commas 2019-07-01 13:07:14 +02:00
Tom Lane 681cca86f5 Blind attempt to fix SSPI-auth case in 010_dump_connstr.pl.
Up to now, pg_regress --config-auth had a hard-wired assumption
that the target cluster uses the default bootstrap superuser name.
pg_dump's 010_dump_connstr.pl TAP test uses non-default superuser
names, and was klugily getting around the restriction by listing
the desired superuser name as a role to "create".  This is pretty
confusing (or at least, it confused me).  Let's make it clearer by
allowing --config-auth mode to be told the bootstrap superuser name.
Repurpose the existing --user switch for that, since it has no
other function in --config-auth mode.

Per buildfarm.  I don't have an environment at hand in which I can
test this fix, but the buildfarm should soon show if it works.

Discussion: https://postgr.es/m/3142.1561840611@sss.pgh.pa.us
2019-06-30 13:34:45 -04:00
Tom Lane ca129e58c0 Fix regression tests to use only global names beginning with "regress_".
In commit 18555b132 we tentatively established a rule that regression
tests should use names containing "regression" for databases, and names
starting with "regress_" for all other globally-visible object names, so
as to circumscribe the side-effects that "make installcheck" could have on
an existing installation.  However, no enforcement mechanism was created,
so it's unsurprising that some new violations have crept in since then.

In fact, a whole new *category* of violations has crept in, to wit we now
also have globally-visible subscription and replication origin names, and
"make installcheck" could very easily clobber user-created objects of
those types.  So it's past time to do something about this.

This commit sanitizes the tests enough that they will pass (i.e. not
generate any visible warnings) with the enforcement mechanism I'll add
in the next commit.  There are some TAP tests that still trigger the
warnings, but the warnings do not cause test failure.  Since these tests
do not actually run against a pre-existing installation, there's no need
to worry whether they could conflict with user-created objects.

The problem with rolenames.sql testing special role names like "user"
is still there, and is dealt with only very cosmetically in this patch
(by hiding the warnings :-().  What we actually need to do to be safe is
to take that test script out of "make installcheck" altogether, but that
seems like material for a separate patch.

Discussion: https://postgr.es/m/16638.1468620817@sss.pgh.pa.us
2019-06-29 11:09:03 -04:00
Michael Paquier ce59b75d44 Add toast-level reloption for vacuum_index_cleanup
a96c41f has introduced the option for heap, but it still lacked the
variant to control the behavior for toast relations.

While on it, refactor the tests so as they stress more scenarios with
the various values that vacuum_index_cleanup can use.  It would be
useful to couple those tests with pageinspect to check that pages are
actually cleaned up, but this is left for later.

Author: Masahiko Sawada, Michael Paquier
Reviewed-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAD21AoCqs8iN04RX=i1KtLSaX5RrTEM04b7NHYps4+rqtpWNEg@mail.gmail.com
2019-06-25 09:09:27 +09:00
Noah Misch 660a2b1903 Consolidate methods for translating a Perl path to a Windows path.
This fixes some TAP suites when using msys Perl and a builddir located
in an msys mount point other than "/".  For example, builddir=/c/pg
exhibited the problem, since /c/pg falls in mount point "/c".
Back-patch to 9.6, where tests first started to perform such
translations.  In back branches, offer both new and old APIs.

Reviewed by Andrew Dunstan.

Discussion: https://postgr.es/m/20190610045838.GA238501@rfd.leadboat.com
2019-06-21 20:34:23 -07:00
Peter Eisentraut cd917ffb9a pg_upgrade: Improve error messages
Make wording more accurate and add strerror() information.

Discussion: https://www.postgresql.org/message-id/24c8bd05-aed1-6301-919d-8acbabdb8c24@2ndquadrant.com
2019-06-19 21:50:24 +02:00
Peter Eisentraut aba78ab4a9 pg_upgrade: Improve invalid option handling
Currently, calling pg_upgrade with an invalid command-line option
aborts pg_upgrade but leaves a pg_upgrade_internal.log file lying
around.  Reorder things a bit so that that file is not created until
all the options have been parsed.

Discussion: https://www.postgresql.org/message-id/24c8bd05-aed1-6301-919d-8acbabdb8c24@2ndquadrant.com
2019-06-19 21:50:24 +02:00
Peter Eisentraut 91acff7a53 Translation updates
Source-Git-URL: https://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 1a710c413ce4c4cd081843e563cde256bb95f490
2019-06-17 15:30:20 +02:00
Michael Paquier 3412030205 Fix more typos and inconsistencies in the tree
Author: Alexander Lakhin
Discussion: https://postgr.es/m/0a5419ea-1452-a4e6-72ff-545b1a5a8076@gmail.com
2019-06-17 16:13:16 +09:00
Andrew Gierth e3846a00c2 Prefer timezone name "UTC" over alternative spellings.
tzdb 2019a made "UCT" a link to the "UTC" zone rather than a separate
zone with its own abbreviation. Unfortunately, our code for choosing a
timezone in initdb has an arbitrary preference for names earlier in
the alphabet, and so it would choose the spelling "UCT" over "UTC"
when the system is running on a UTC zone.

Commit 23bd3cec6 was backpatched in order to address this issue, but
that code helps only when /etc/localtime exists as a symlink, and does
nothing to help on systems where /etc/localtime is a copy of a zone
file (as is the standard setup on FreeBSD and probably some other
platforms too) or when /etc/localtime is simply absent (giving UTC as
the default).

Accordingly, add a preference for the spelling "UTC", such that if
multiple zone names have equally good content matches, we prefer that
name before applying the existing arbitrary rules. Also add a slightly
lower preference for "Etc/UTC"; lower because that preserves the
previous behaviour of choosing the shorter name, but letting us still
choose "Etc/UTC" over "Etc/UCT" when both exist but "UTC" does
not (not common, but I've seen it happen).

Backpatch all the way, because the tzdb change that sparked this issue
is in those branches too.
2019-06-15 18:15:23 +01:00
Alvaro Herrera a193cbec11 Add pg_dumpall --rows-per-insert
Commit 7e413a0f82 added that option to pg_dump, but neglected to teach
pg_dumpall how to pass it along.  Repair.

Author: Fabien Coelho
Reported-by: Peter Eisentraut
Reviewed-by: David Rowley
Discussion: https://postgr.es/m/45f50c59-ddbb-8cf2-eedb-81003f603528@2ndquadrant.com
2019-06-14 18:21:52 -04:00
Alvaro Herrera 313f56ce2d Tweak libpq's PQhost, PQhostaddr, and psql's \connect
Fixes some problems introduced by 6e5f8d489acc:

* When reusing conninfo data from the previous connection in \connect,
  the host address should only be reused if it was specified as
  hostaddr; if it wasn't, then 'host' is resolved afresh.  We were
  reusing the same IP address, which ignores a possible DNS change
  as well as any other addresses that the name resolves to than the
  one that was used in the original connection.

* PQhost, PQhostaddr: Don't present user-specified hostaddr when we have
  an inet_net_ntop-produced equivalent address.  The latter has been
  put in canonical format, which is cleaner (so it produces "127.0.0.1"
  when given "host=2130706433", for example).

* Document the hostaddr-reusing aspect of \connect.

* Fix some code comments

Author: Fabien Coelho
Reported-by: Noah Misch
Discussion: https://postgr.es/m/20190527203713.GA58392@gust.leadboat.com
2019-06-14 18:02:26 -04:00
Alvaro Herrera b976845815 Fix double-word typos
Discussion: https://postgr.es/m/20190612184527.GA24266@alvherre.pgsql
Reviewed-by: Michaël Paquier
2019-06-13 10:03:56 -04:00
Alvaro Herrera 33a53130a8 Make pg_dump emit ATTACH PARTITION instead of PARTITION OF (reprise)
Using PARTITION OF can result in column ordering being changed from the
database being dumped, if the partition uses a column layout different
from the parent's.  It's not pg_dump's job to editorialize on table
definitions, so this is not acceptable; back-patch all the way back to
pg10, where partitioned tables where introduced.

This change also ensures that partitions end up in the correct
tablespace, if different from the parent's; this is an oversight in
ca4103025d (in pg12 only).  Partitioned indexes (in pg11) don't have
this problem, because they're already created as independent indexes and
attached to their parents afterwards.

This change also has the advantage that the partition is restorable from
the dump (as a standalone table) even if its parent table isn't
restored.

The original commits (3b23552ad8 in branch master) failed to cover
subsidiary column elements correctly, such as NOT NULL constraint and
CHECK constraints, as reported by Rushabh Lathia (initially as a failure
to restore serial columns).  They were reverted.  This recapitulation
commit fixes those problems.

Add some pg_dump tests to verify these things more exhaustively,
including constraints with legacy-inheritance tables, which were not
tested originally.  In branches 10 and 11, add a local constraint to the
pg_dump test partition that was added by commit 2d7eeb1b14 to master.

Author: Álvaro Herrera, David Rowley
Reviewed-by: Álvaro Herrera
Discussion: https://postgr.es/m/CAKJS1f_1c260nOt_vBJ067AZ3JXptXVRohDVMLEBmudX1YEx-A@mail.gmail.com
Discussion: https://postgr.es/m/20190423185007.GA27954@alvherre.pgsql
Discussion: https://postgr.es/m/CAGPqQf0iQV=PPOv2Btog9J9AwOQp6HmuVd6SbGTR_v3Zp2XT1w@mail.gmail.com
2019-06-10 18:56:23 -04:00
Noah Misch f2c71cb71f Stop using spelling "nonexistant".
The documentation used "nonexistent" exclusively, and the source tree
used it three times as often as "nonexistant".
2019-06-08 10:12:26 -07:00
Noah Misch 31d250e049 Update stale comments, and fix comment typos. 2019-06-08 10:12:26 -07:00
Michael Paquier 84d4de97e8 Improve some comments in pg_checksums about the needed clean shutdown
It was not clear from the code why it is necessary.  And we need a clean
shutdown to avoid random checksum failures caused by torn pages.

Author: Masahiko Sawada
Discussion: https://postgr.es/m/CAD21AoDum5MbAb7F=pa9dOD1W2tukuDMPzWT7NjZceNoWB_6Qw@mail.gmail.com
2019-06-07 20:48:39 +09:00
Michael Paquier 438e51987d Add forgotten translatable string in pg_checksums.c
Oversight in commit 280e5f1.

Author: Kyotaro Horiguchi
Discussion: https://postgr.es/m/20190606.200612.110313249.horikyota.ntt@gmail.com
2019-06-06 22:10:52 +09:00
Peter Eisentraut f65eced251 pg_waldump: Fix invalid option handling
Previously, running pg_waldump with an invalid option (pg_waldump
--foo) would print the help output and exit successfully.  This was
because it tried to process the option letter '?' as a normal option,
but that letter is used by getopt() to report an invalid option.

To fix, process help and version options separately, like we do
everywhere else.  Also add a basic test suite for pg_waldump and run
the basic option handling tests, which would have caught this.
2019-06-06 10:14:25 +02:00
Michael Paquier 1fb6f62a84 Fix typos in various places
Author: Andrea Gelmini
Reviewed-by: Michael Paquier, Justin Pryzby
Discussion: https://postgr.es/m/20190528181718.GA39034@glet
2019-06-03 13:44:03 +09:00
Michael Paquier 0240a00fbd Fix some issues and improve psql completion for access methods
The following issues have been spotted:
- CREATE INDEX .. USING suggests both index and table AMs, but it should
consider only index AMs.
- CREATE TABLE .. USING has no completion support.  USING was not being
included in the completion list where it should, and follow-up
suggestions for table AMs have been missing as well.
- CREATE ACCESS METHOD .. TYPE suggests only INDEX, with TABLE missing.

Author: Michael Paquier
Discussion: https://postgr.es/m/20190601191007.GC1905@paquier.xyz
2019-06-03 11:02:32 +09:00
Tom Lane 7640f93123 Fix assorted header files that failed to compile standalone.
We have a longstanding project convention that all .h files should
be includable with no prerequisites other than postgres.h.  This is
tested/relied-on by cpluspluscheck.  However, cpluspluscheck has not
historically been applied to most headers outside the src/include
tree, with the predictable consequence that some of them don't work.
Fix that, usually by adding missing #include dependencies.

The change in printf_hack.h might require some explanation: without
it, my C++ compiler whines that the function is unused.  There's
not so many call sites that "inline" is going to cost much, and
besides all the callers are in test code that we really don't care
about the size of.

There's no actual bugs being fixed here, so I see no need to back-patch.

Discussion: https://postgr.es/m/b517ec3918d645eb950505eac8dd434e@gaz-is.ru
2019-05-31 11:45:33 -04:00
Michael Paquier fc115d0f9f Rework options of pg_checksums options for filenode handling
This makes the tool consistent with the option set of oid2name, which
has been historically using -f for filenodes, and has more recently
gained long options and --filenode via 1aaf532.

Reported-by: Peter Eisentraut
Author: Fabien Coelho
Discussion: https://postgr.es/m/97045260-fb9e-e145-a950-cf7d28c4eaea@2ndquadrant.com
2019-05-30 16:58:17 -04:00
Noah Misch 40b132c1af In the pg_upgrade test suite, don't write to src/test/regress.
When this suite runs installcheck, redirect file creations from
src/test/regress to src/bin/pg_upgrade/tmp_check/regress.  This closes a
race condition in "make -j check-world".  If the pg_upgrade suite wrote
to a given src/test/regress/results file in parallel with the regular
src/test/regress invocation writing it, a test failed spuriously.  Even
without parallelism, in "make -k check-world", the suite finishing
second overwrote the other's regression.diffs.  This revealed test
"largeobject" assuming @abs_builddir@ is getcwd(), so fix that, too.

Buildfarm client REL_10, released fifty-four days ago, supports saving
regression.diffs from its new location.  When an older client reports a
pg_upgradeCheck failure, it will no longer include regression.diffs.
Back-patch to 9.5, where pg_upgrade moved to src/bin.

Reviewed (in earlier versions) by Andrew Dunstan.

Discussion: https://postgr.es/m/20181224034411.GA3224776@rfd.leadboat.com
2019-05-28 12:59:00 -07:00
Noah Misch 10b72deafe In the pg_upgrade test suite, remove and recreate "tmp_check".
This allows "vcregress upgradecheck" to pass twice in immediate
succession, and it's more like how $(prove_check) works.  Back-patch to
9.5, where pg_upgrade moved to src/bin.

Discussion: https://postgr.es/m/20190520012436.GA1480421@rfd.leadboat.com
2019-05-28 12:58:30 -07:00
Peter Eisentraut a94fd139df Fix comment
This code block was copied/adapted from other similar places but
somehow the comment placement was changed so that it makes less sense.
2019-05-28 08:26:24 -04:00
Amit Kapila 9679345f3c Fix typos.
Reported-by: Alexander Lakhin
Author: Alexander Lakhin
Reviewed-by: Amit Kapila and Tom Lane
Discussion: https://postgr.es/m/7208de98-add8-8537-91c0-f8b089e2928c@gmail.com
2019-05-26 18:28:18 +05:30
Thomas Munro 7988cb446d Fix typos.
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/CA%2BhUKGJFWXmtYo6Frd77RR8YXCHz7hJ2mRy5aHV%3D7fJOqDnBHA%40mail.gmail.com
2019-05-24 12:00:59 +12:00
Michael Paquier 54487d1560 Fix table dump in pg_dump[all] with backends older than 9.5
The access method name "amname" can be dumped as of 3b925e90, but
queries for backends older than 9.5 forgot to map it to a dummy NULL
value, causing the column to not be mapped to a number.  As a result,
pg_dump was throwing some spurious errors in its stderr output coming
from libpq:
pg_dump: column number -1 is out of range 0..36

Fix this issue by adding a mapping of "amname" to NULL to all the older
queries.

Discussion: https://postgr.es/m/20190522083038.GA16837@paquier.xyz
Author: Michael Paquier
Reviewed-by: Dmitry Dolgov, Andres Freund, Tom Lane
2019-05-24 08:19:21 +09:00
Andres Freund f916221df5 pg_upgrade: Make test.sh's installcheck use to-be-upgraded version's bindir.
On master (after 700538) the old version's installed psql was used -
even when the old version might not actually be installed / might be
installed into a temporary directory. As commonly the case when just
executing make check for pg_upgrade, as $oldbindir is just the current
version's $bindir.

In the back branches, with --install specified, psql from the new
version's temporary installation was used, without --install (e.g for
NO_TEMP_INSTALL, cf 47b3c26642), the new version's installed psql was
used (which might or might not exist).

Author: Andres Freund
Discussion: https://postgr.es/m/20190522175150.c26f4jkqytahajdg@alap3.anarazel.de
2019-05-23 14:46:52 -07:00
Michael Paquier 156c0c2dff Fix ordering of GRANT commands in pg_dumpall for tablespaces
This uses a method similar to 68a7c24f and now b8c6014 (applied for
database creation), which guarantees that GRANT commands using the WITH
GRANT OPTION are dumped in a way so as cascading dependencies are
respected.  Note that tablespaces do not have support for initial
privileges via pg_init_privs, so the same method needs to be applied
again.  It would be nice to merge all the logic generating ACL queries
in dumps under the same banner, but this requires extending the support
of pg_init_privs to objects that cannot use it yet, so this is left as
future work.

Discussion: https://postgr.es/m/20190522071555.GB1278@paquier.xyz
Author: Michael Paquier
Reviewed-by: Nathan Bossart
Backpatch-through: 9.6
2019-05-23 10:48:17 +09:00
Michael Paquier 657c2384c6 Remove -o/--oids from pg_dumpall
This has been forgotten in 578b229, which has removed support for WITH
OIDS.

Discussion: https://postgr.es/m/CALAY4q99FcFCoG6ddke0V-AksGe82L_+bhDWgEfgZBakB840zA@mail.gmail.com
Author: Surafel Temesgen
2019-05-23 09:36:28 +09:00
Tom Lane db6e2b4c52 Initial pgperltidy run for v12.
Make all the perl code look nice, too (for some value of "nice").
2019-05-22 13:36:19 -04:00
Tom Lane 8255c7a5ee Phase 2 pgindent run for v12.
Switch to 2.1 version of pg_bsd_indent.  This formats
multiline function declarations "correctly", that is with
additional lines of parameter declarations indented to match
where the first line's left parenthesis is.

Discussion: https://postgr.es/m/CAEepm=0P3FeTXRcU5B2W3jv3PgRVZ-kGUXLGfd42FFhUROO3ug@mail.gmail.com
2019-05-22 13:04:48 -04:00
Tom Lane be76af171c Initial pgindent run for v12.
This is still using the 2.0 version of pg_bsd_indent.
I thought it would be good to commit this separately,
so as to document the differences between 2.0 and 2.1 behavior.

Discussion: https://postgr.es/m/16296.1558103386@sss.pgh.pa.us
2019-05-22 12:55:34 -04:00
Fujii Masao 03de5187d5 Mention ANALYZE boolean options in documentation.
Commit 41b54ba78e allowed not only VACUUM but also ANALYZE options
to take a boolean argument. But it forgot to update the documentation
for ANALYZE. This commit adds the descriptions about those ANALYZE
boolean options into the documentation.

This patch also updates tab-completion for ANALYZE boolean options.

Reported-by: Kyotaro Horiguchi
Author: Fujii Masao
Reviewed-by: Masahiko Sawada, Michael Paquier
Discussion: https://postgr.es/m/CAHGQGwHTUt-kuwgiwe8f0AvTnB+ySqJWh95jvmh-qcoKW9YA9g@mail.gmail.com
2019-05-23 01:18:16 +09:00
Michael Paquier b8c6014a65 Fix ordering of GRANT commands in pg_dump for database creation
This uses a method similar to 68a7c24f, which guarantees that GRANT
commands using the WITH GRANT OPTION are dumped in a way so as cascading
dependencies are respected.  As databases do not have support for
initial privileges via pg_init_privs, we need to repeat again the same
ACL reordering method.

ACL for databases have been moved from pg_dumpall to pg_dump in v11, so
this impacts pg_dump for v11 and above, and pg_dumpall for v9.6 and
v10.

Discussion: https://postgr.es/m/15788-4e18847520ebcc75@postgresql.org
Author: Nathan Bossart
Reviewed-by: Haribabu Kommi
Backpatch-through: 9.6
2019-05-22 14:48:00 +09:00
Tom Lane 5eb4a51cb1 Un-break pg_upgrade regression test.
Commit 5af2e976d removed a bit too much from the test.sh invocation.
Per buildfarm.
2019-05-21 23:51:19 -04:00
Andres Freund 5af2e976d7 pg_upgrade: Avoid check target accidentally breaking make's --output-sync.
When $(MAKE) is present in a rule, make assumes that target is a
submake, and it doesn't need to buffer its output. But in this case
it's a shell script that needs buffered output. Avoid that heuristic,
by referring to $(MAKE) via an indirection.

Discussion: https://postgr.es/m/20190521004717.qsktdsugj3shagco@alap3.anarazel.de
2019-05-21 15:03:53 -07:00
Andres Freund 7005389b2a pg_upgrade: Don't use separate installation for test.
For pg_upgrade's test we (unless prevented by the caller via via
NO_TEMP_INSTALL) built a separate installation. That causes an
unnecessary slowdown after the infrastructure introduced by
dcae5facca (and unnecessarily duplicates code).

Author: Andres Freund
Reviewed-By: Tom Lane
Discussion:
    https://postgr.es/m/20190521191918.z7kwnrlj45mk2k67@alap3.anarazel.de
    https://postgr.es/m/20190521195209.qfzwfxvymguuwlu5@alap3.anarazel.de
2019-05-21 15:03:53 -07:00
Tom Lane eb9812f272 Make pg_upgrade's test.sh less chatty.
The use of "set -x" to echo a subset of the test's commands might've
been a good idea during development of this test, but it's been stable
for long enough now that the extra output isn't very useful.  Also
our project expectations have been trending towards less output in
non-error cases; the fact that "set -x" produces output on stderr
is particularly annoying from that standpoint.  So get rid of it.

Also, pass "-A trust" to initdb explicitly so that it won't issue
a warning about "trust" being an insecure default.  This matches
what the TAP tests have done for a long time, and again gets rid
of some noise on stderr.

Discussion: https://postgr.es/m/21766.1558397960@sss.pgh.pa.us
2019-05-21 13:11:57 -04:00
Peter Eisentraut 3c439a58df Translation updates
Source-Git-URL: https://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: a20bf6b8a5b4e32450967055eb5b07cee4704edd
2019-05-20 16:00:53 +02:00
Noah Misch ae35e1c9d7 Revert "In the pg_upgrade test suite, don't write to src/test/regress."
This reverts commit bd1592e857.  It had
multiple defects.

Discussion: https://postgr.es/m/12717.1558304356@sss.pgh.pa.us
2019-05-19 15:24:42 -07:00
Noah Misch bd1592e857 In the pg_upgrade test suite, don't write to src/test/regress.
When this suite runs installcheck, redirect file creations from
src/test/regress to src/bin/pg_upgrade/tmp_check/regress.  This closes a
race condition in "make -j check-world".  If the pg_upgrade suite wrote
to a given src/test/regress/results file in parallel with the regular
src/test/regress invocation writing it, a test failed spuriously.  Even
without parallelism, in "make -k check-world", the suite finishing
second overwrote the other's regression.diffs.  This revealed test
"largeobject" assuming @abs_builddir@ is getcwd(), so fix that, too.

Buildfarm client REL_10, released forty-five days ago, supports saving
regression.diffs from its new location.  When an older client reports a
pg_upgradeCheck failure, it will no longer include regression.diffs.
Back-patch to 9.5, where pg_upgrade moved to src/bin.

Reviewed by Andrew Dunstan.

Discussion: https://postgr.es/m/20181224034411.GA3224776@rfd.leadboat.com
2019-05-19 14:36:44 -07:00
Tom Lane 9d5c22d28f Improve logrotate test so that it meaningfully exercises syslogger.
Discussion of bug #15804 reveals that this test didn't really prove
that the syslogger child process ever launched successfully, much
less did anything.  It was only checking that the expected log file
gets created, and that's done in the postmaster.  Moreover, the
test assumed it could rename the log file, which is likely to fail
on Windows (cf. commit d611175e5).

Instead, use the default log file name pattern, which should result
in a new file name being chosen after 1 second, and verify that
rotation has occurred by checking for a new file name.  Also add code
to test that messages actually do propagate through the syslogger.

In theory this version of the test should work on Windows, so
revert d611175e5.

Discussion: https://postgr.es/m/15804-3721117bf40fb654@postgresql.org
2019-05-19 13:55:39 -04:00
Tom Lane fc9a62af3f Move logging.h and logging.c from src/fe_utils/ to src/common/.
The original placement of this module in src/fe_utils/ is ill-considered,
because several src/common/ modules have dependencies on it, meaning that
libpgcommon and libpgfeutils now have mutual dependencies.  That makes it
pointless to have distinct libraries at all.  The intended design is that
libpgcommon is lower-level than libpgfeutils, so only dependencies from
the latter to the former are acceptable.

We already have the precedent that fe_memutils and a couple of other
modules in src/common/ are frontend-only, so it's not stretching anything
out of whack to treat logging.c as a frontend-only module in src/common/.
To the extent that such modules help provide a common frontend/backend
environment for the rest of common/ to use, it's a reasonable design.
(logging.c does not yet provide an ereport() emulation, but one can
dream.)

Hence, move these files over, and revert basically all of the build-system
changes made by commit cc8d41511.  There are no places that need to grow
new dependencies on libpgcommon, further reinforcing the idea that this
is the right solution.

Discussion: https://postgr.es/m/a912ffff-f6e4-778a-c86a-cf5c47a12933@2ndquadrant.com
2019-05-14 14:20:10 -04:00
Tom Lane 53ddefbaf8 Remove pg_rewind's private logging.h/logging.c files.
The existence of these files became rather confusing with the
introduction of a widely-known logging.h header in commit cc8d41511.
(Indeed, there's already some duplicative #includes here, perhaps
betraying such confusion.)  The only thing left in them, after that
commit, is a progress-reporting function that's neither general-purpose
nor tied in any way to other logging infrastructure.  Hence, let's just
move that function to pg_rewind.c, and get rid of the separate files.

Discussion: https://postgr.es/m/3971.1557787914@sss.pgh.pa.us
2019-05-14 13:11:23 -04:00
Noah Misch d02768ddd1 Fail pgwin32_message_to_UTF16() for SQL_ASCII messages.
The function had been interpreting SQL_ASCII messages as UTF8, throwing
an error when they were invalid UTF8.  The new behavior is consistent
with pg_do_encoding_conversion().  This affects LOG_DESTINATION_STDERR
and LOG_DESTINATION_EVENTLOG, which will send untranslated bytes to
write() and ReportEventA().  On buildfarm member bowerbird, enabling
log_connections caused an error whenever the role name was not valid
UTF8.  Back-patch to 9.4 (all supported versions).

Discussion: https://postgr.es/m/20190512015615.GD1124997@rfd.leadboat.com
2019-05-12 10:33:05 -07:00
Noah Misch 54c2ecb567 Honor TEMP_CONFIG in TAP suites.
The buildfarm client uses TEMP_CONFIG to implement its extra_config
setting.  Except for stats_temp_directory, extra_config now applies to
TAP suites; extra_config values seen in the past month are compatible
with this.  Back-patch to 9.6, where PostgresNode was introduced, so the
buildfarm can rely on it sooner.

Reviewed by Andrew Dunstan and Tom Lane.

Discussion: https://postgr.es/m/20181229021950.GA3302966@rfd.leadboat.com
2019-05-11 00:22:38 -07:00
Michael Paquier e51bad8fb4 Fix error reporting in reindexdb
When failing to reindex a table or an index, reindexdb would generate an
extra error message related to a database failure, which is misleading.

Backpatch all the way down, as this has been introduced by 85e9a5a0.

Discussion: https://postgr.es/m/CAOBaU_Yo61RwNO3cW6WVYWwH7EYMPuexhKqufb2nFGOdunbcHw@mail.gmail.com
Author: Julien Rouhaud
Reviewed-by: Daniel Gustafsson, Álvaro Herrera, Tom Lane, Michael
Paquier
Backpatch-through: 9.4
2019-05-11 13:00:54 +09:00
Peter Eisentraut cd805f46d8 pg_controldata: Add common gettext flags
So it picks up strings in pg_log_* calls.  This was forgotten when it
was added to all other relevant subdirectories.
2019-05-09 09:16:59 +02:00
Michael Paquier 3ae3c18b36 Fix error status of vacuumdb when multiple jobs are used
When running a batch of VACUUM or ANALYZE commands on a given database,
there were cases where it is possible to have vacuumdb not report an
error where it actually should, leading to incorrect status results.

Author: Julien Rouhaud
Reviewed-by: Amit Kapila, Michael Paquier
Discussion: https://postgr.es/m/CAOBaU_ZuTwz7CtqLYJ1Ouuh272bTQPLN8b1bAPk0bCBm4PDMTQ@mail.gmail.com
Backpatch-through: 9.5
2019-05-09 10:29:10 +09:00
Fujii Masao b84dbc8eb8 Add TRUNCATE parameter to VACUUM.
This commit adds new parameter to VACUUM command, TRUNCATE,
which specifies that VACUUM should attempt to truncate off
any empty pages at the end of the table and allow the disk space
for the truncated pages to be returned to the operating system.

This parameter, if specified, overrides the vacuum_truncate
reloption. If neither the reloption nor the VACUUM option is
used, the default is true, as before.

Author: Fujii Masao
Reviewed-by: Julien Rouhaud, Masahiko Sawada
Discussion: https://postgr.es/m/CAD21AoD+qtrSDL=GSma4Wd3kLYLeRC0hPna-YAdkDeV4z156vg@mail.gmail.com
2019-05-08 02:10:33 +09:00
Amit Kapila 7db0cde6b5 Revert "Avoid the creation of the free space map for small heap relations".
This feature was using a process local map to track the first few blocks
in the relation.  The map was reset each time we get the block with enough
freespace.  It was discussed that it would be better to track this map on
a per-relation basis in relcache and then invalidate the same whenever
vacuum frees up some space in the page or when FSM is created.  The new
design would be better both in terms of API design and performance.

List of commits reverted, in reverse chronological order:

06c8a5090e  Improve code comments in b0eaa4c51b.
13e8643bfc  During pg_upgrade, conditionally skip transfer of FSMs.
6f918159a9  Add more tests for FSM.
9c32e4c350  Clear the local map when not used.
29d108cdec  Update the documentation for FSM behavior..
08ecdfe7e5  Make FSM test portable.
b0eaa4c51b  Avoid creation of the free space map for small heap relations.

Discussion: https://postgr.es/m/20190416180452.3pm6uegx54iitbt5@alap3.anarazel.de
2019-05-07 09:30:24 +05:30
Michael Paquier af82f95abb Remove some code related to 7.3 and older servers from tools of src/bin/
This code was broken as of 582edc3, and is most likely not used anymore.
Note that pg_dump supports servers down to 8.0, and psql has code to
support servers down to 7.4.

Author: Julien Rouhaud
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/CAOBaU_Y5y=zo3+2gf+2NJC1pvMYPcbRXoQaPXx=U7+C8Qh4CzQ@mail.gmail.com
2019-05-07 09:39:39 +09:00
Alvaro Herrera a1ec7402e9 Revert "Make pg_dump emit ATTACH PARTITION instead of PARTITION OF"
... and fallout (from branches 10, 11 and master).  The change was
ill-considered, and it broke a few normal use cases; since we don't have
time to fix it, we'll try again after this week's minor releases.

Reported-by: Rushabh Lathia
Discussion: https://postgr.es/m/CAGPqQf0iQV=PPOv2Btog9J9AwOQp6HmuVd6SbGTR_v3Zp2XT1w@mail.gmail.com
2019-05-06 12:23:49 -04:00
Michael Paquier 84e4570da9 Fix set of issues with memory-allocation system calls in frontend code
Like the backend, the frontend has wrappers on top of malloc() and such
whose use is recommended.  Particularly, it is possible to do memory
allocation without issuing an error.  Some binaries missed the use of
those wrappers, so let's fix the gap for consistency.

This also fixes two latent bugs:
- In pg_dump/pg_dumpall when parsing an ACL item, on an out-of-memory
error for strdup(), the code considered the failure as a ACL parsing
problem instead of an actual OOM.
- In pg_waldump, an OOM when building the target directory string would
cause a crash.

Author: Daniel Gustafsson
Discussion: https://postgr.es/m/gY0y9xenfoBPc-Tufsr2Zg-MmkrJslm0Tw_CMg4p_j58-k_PXNC0klMdkKQkg61BkXC9_uWo-DcUzfxnHqpkpoR5jjVZrPHqKYikcHIiONhg=@yesql.se
2019-05-04 16:32:19 +09:00
Michael Paquier 9c592896d9 Fix some typos
Author: Daniel Gustafsson
Discussion: https://postgr.es/m/42kEeWei6VxLGh12QbR08hiI5Pm-c3XgbK7qj393PSttEhVbnnQoFXHKzXjPRZLUpndWAfHIuZuUqGZBzyXadmEUCSqm9xphWur_I8vESMA=@yesql.se
2019-04-29 23:52:42 +09:00
Alvaro Herrera ffbce803e6 Message fixes 2019-04-29 10:05:45 -04:00
Noah Misch 90e7f31773 Use preprocessor conditions compatible with Emacs indent.
Emacs wrongly indented hundreds of subsequent lines.
2019-04-28 12:56:53 -07:00
Alvaro Herrera 7fcdb5e002 pg_dump: store unused attribs as NULL instead of '\0'
Commit f831d4accd changed pg_dump to emit (and pg_restore to
understand) NULLs for unused members in ArchiveEntry structs, as a side
effect of some code beautification.  That broke pg_restore of dumps
generated with older pg_dump, however, so it was reverted in
19455c9f56.  Since the archiver version number has been bumped in
3b925e905d, we can put it back.

Author: Dmitry Dolgov
Discussion: https://postgr.es/m/CA+q6zcXx0XHqLsFJLaUU2j5BDiBAHig=YRoBC_YVq7VJGvzBEA@mail.gmail.com
2019-04-26 12:05:17 -04:00
Fujii Masao c247ae0922 Fix file path in comment. 2019-04-25 23:49:37 +09:00
Alvaro Herrera 3b23552ad8 Make pg_dump emit ATTACH PARTITION instead of PARTITION OF
Using PARTITION OF can result in column ordering being changed from the
database being dumped, if the partition uses a column layout different
from the parent's.  It's not pg_dump's job to editorialize on table
definitions, so this is not acceptable; back-patch all the way back to
pg10, where partitioned tables where introduced.

This change also ensures that partitions end up in the correct
tablespace, if different from the parent's; this is an oversight in
ca4103025d (in pg12 only).  Partitioned indexes (in pg11) don't have
this problem, because they're already created as independent indexes and
attached to their parents afterwards.

This change also has the advantage that the partition is restorable from
the dump (as a standalone table) even if its parent table isn't
restored.

Author: David Rowley
Reviewed-by: Álvaro Herrera
Discussion: https://postgr.es/m/CAKJS1f_1c260nOt_vBJ067AZ3JXptXVRohDVMLEBmudX1YEx-A@mail.gmail.com
Discussion: https://postgr.es/m/20190423185007.GA27954@alvherre.pgsql
2019-04-24 15:30:37 -04:00
Andres Freund 4d01835927 pg_dump: Remove stray option parsing support for -o.
I (Andres) missed this in 578b229718, the removal of WITH OIDS
support.

Author: Daniel Verite
Discussion: https://postgr.es/m/f06e9735-3717-4904-8c95-47d0b9c3bb10@manitou-mail.org
2019-04-17 17:28:02 -07:00
Alvaro Herrera 421a2c4832 Tie loose ends in psql's new \dP command
* Remove one unnecessary pg_class join in SQL command.  Not needed,
  because we use a regclass cast instead.

* Doc: refer to "partitioned relations" rather than specifically tables,
  since indexes are also displayed.

* Rename "On table" column to "Table", for consistency with \di.

Author: Justin Pryzby
Discussion: https://postgr.es/m/20190407212525.GB10080@telsasoft.com
2019-04-17 18:38:49 -04:00
Alvaro Herrera b036982db7 psql: display tablespace for partitioned indexes
Nothing was shown previously.
2019-04-17 18:17:43 -04:00
Tom Lane 1a75c1d0c5 Fix unportable code in pgbench.
The buildfarm points out that UINT64_FORMAT might not work with sscanf;
it's calibrated for our printf implementation, which might not agree
with the platform-supplied sscanf.  Fall back to just accepting an
unsigned long, which is already more than the documentation promises.

Oversight in e6c3ba7fb; back-patch to v11, as that was.
2019-04-17 17:30:29 -04:00
Michael Paquier d9f543e9e9 Switch TAP tests of pg_rewind to use non-superuser role, take two
Up to now the tests of pg_rewind have been using a superuser for all its
tests (which is the default of many tests actually, and something that
ought to be reviewed) when involving an online source server, still it
is possible to use a non-superuser role to do that as long as this role
is granted permissions to execute all the source-side functions used for
the rewind.  This is possible since v11, and was already documented as
of bfc8068.

PostgresNode::init is extended so as callers of this routine can add
extra options to configure the authentication of a new node, which gets
used by this commit, and allows the tests to work properly on Windows
where SSPI is used.

This will allow to catch up easily any change in pg_rewind if the tool
begins to use more backend-side functions, so as the properties
introduced by v11 are kept.

Per suggestion from Peter Eisentraut.

Author: Michael Paquier
Reviewed-by: Magnus Hagander
Discussion: https://postgr.es/m/20190411041336.GM2728@paquier.xyz
2019-04-14 18:47:51 +09:00
Michael Paquier db8db624e8 Revert "Switch TAP tests of pg_rewind to use a role with minimal permissions"
This reverts commit d4e2a84, which added a new user with limited
permissions to run the TAP tests of pg_rewind.  Buildfarm machine
members on Windows jacana and bowerbird have been complaining about
that, the new role not being able to run the rewind because SSPI is not
configured to allow it.

Fixing the test requires passing down directly the new user to
pg_regress with --create-role so as SSPI can work properly.

Reported-by: Andrew Dunstan
Discussion: https://postgr.es/m/3cd43d33-f415-cc41-ade3-7230ab15b2c9@2ndQuadrant.com
2019-04-13 13:20:21 +09:00
Michael Paquier d4e2a843e6 Switch TAP tests of pg_rewind to use a role with minimal permissions
Up to now the tests of pg_rewind have been using a superuser for all the
tests (which is the default of many tests actually, and something that
ought to be reviewed) when involving an online source server, still it
is possible to use a non-superuser role to do that as long as this role
is granted permissions to execute all the source-side functions used for
the rewind.  This is possible since v11, and was already documented as
of bfc8068.

This will allow to catch up easily any change in pg_rewind if the tool
begins to use more backend-side functions, so as the properties
introduced by v11 are kept.

Per suggestion from Peter Eisentraut.

Author: Michael Paquier
Reviewed-by: Magnus Hagander
Discussion: https://postgr.es/m/20190411041336.GM2728@paquier.xyz
2019-04-12 10:46:43 +09:00
Alvaro Herrera 65d857d92c Fix declaration after statement
This style is frowned upon.  I inadvertently introduced one in commit
fe0e0b4fc7.  (My compiler does not complain about it, even though
-Wdeclaration-after-statement is specified.  Weird.)

Author: Masahiko Sawada
2019-04-10 22:31:40 -04:00
Peter Eisentraut 765525c8c2 pg_restore: Make not verbose by default
This was accidentally changed in
cc8d415117.

Reported-by: Christoph Berg <myon@debian.org>
2019-04-10 11:45:00 +02:00
Alvaro Herrera fe0e0b4fc7 Fix memory leak in pgbench
Commit 25ee70511e introduced a memory leak in pgbench: some PGresult
structs were not being freed during error bailout, because we're now
doing more PQgetResult() calls than previously.  Since there's more
cleanup code outside the discard_response() routine than in it, refactor
the cleanup code, removing the routine.

This has little effect currently, since we abandon processing after
hitting errors, but if we ever get further pgbench features (such as
testing for serializable transactions), it'll matter.

Per Coverity.

Reviewed-by: Michaël Paquier
2019-04-09 12:46:34 -04:00
Fujii Masao 119dcfad98 Add vacuum_truncate reloption.
vacuum_truncate controls whether vacuum tries to truncate off
any empty pages at the end of the table. Previously vacuum always
tried to do the truncation. However, the truncation could cause
some problems; for example, ACCESS EXCLUSIVE lock needs to
be taken on the table during the truncation and can cause
the query cancellation on the standby even if hot_standby_feedback
is true. Setting this reloption to false can be helpful to avoid
such problems.

Author: Tsunakawa Takayuki
Reviewed-By: Julien Rouhaud, Masahiko Sawada, Michael Paquier, Kirk Jamison and Fujii Masao
Discussion: https://postgr.es/m/CAHGQGwE5UqFqSq1=kV3QtTUtXphTdyHA-8rAj4A=Y+e4kyp3BQ@mail.gmail.com
2019-04-08 16:43:57 +09:00
Alvaro Herrera 1c5d9270e3 psql \dP: list partitioned tables and indexes
The new command lists partitioned relations (tables and/or indexes),
possibly with their sizes, possibly including partitioned partitions;
their parents (if not top-level); if indexes show the tables they belong
to; and their descriptions.

While there are various possible improvements to this, having it in this
form is already a great improvement over not having any way to obtain
this report.

Author: Pavel Stěhule, with help from Mathias Brossard, Amit Langote and
	Justin Pryzby.
Reviewed-by: Amit Langote, Mathias Brossard, Melanie Plageman,
	Michaël Paquier, Álvaro Herrera
2019-04-07 15:07:21 -04:00
Tom Lane 7bac3acab4 Add a "SQLSTATE-only" error verbosity option to libpq and psql.
This is intended for use mostly in test scripts for external tools,
which could do without cross-PG-version variations in error message
wording.  Of course, the SQLSTATE isn't guaranteed stable either, but
it should be more so than the error message text.

Note: there's a bit of an ABI change for libpq here, but it seems
OK because if somebody compiles against a newer version of libpq-fe.h,
and then tries to pass PQERRORS_SQLSTATE to PQsetErrorVerbosity()
of an older libpq library, it will be accepted and then act like
PQERRORS_DEFAULT, thanks to the way the tests in pqBuildErrorMessage3
have historically been phrased.  That seems acceptable.

Didier Gautheron, reviewed by Dagfinn Ilmari Mannsåker

Discussion: https://postgr.es/m/CAJRYxuKyj4zA+JGVrtx8OWAuBfE-_wN4sUMK4H49EuPed=mOBw@mail.gmail.com
2019-04-04 17:22:02 -04:00
Alvaro Herrera 413ccaa74d pg_restore: Require "-f -" to mean stdout
The previous convention that stdout was selected by default when nothing
is specified was just too error-prone.

After a suggestion from Andrew Gierth.
Author: Euler Taveira
Reviewed-by: Yoshikazu Imai, José Arthur Benetasso Villanova
Discussion: https://postgr.es/m/87sgwrmhdv.fsf@news-spur.riddles.org.uk
2019-04-04 16:54:15 -03:00
Robert Haas a96c41feec Allow VACUUM to be run with index cleanup disabled.
This commit adds a new reloption, vacuum_index_cleanup, which
controls whether index cleanup is performed for a particular
relation by default.  It also adds a new option to the VACUUM
command, INDEX_CLEANUP, which can be used to override the
reloption.  If neither the reloption nor the VACUUM option is
used, the default is true, as before.

Masahiko Sawada, reviewed and tested by Nathan Bossart, Alvaro
Herrera, Kyotaro Horiguchi, Darafei Praliaskouski, and me.
The wording of the documentation is mostly due to me.

Discussion: http://postgr.es/m/CAD21AoAt5R3DNUZSjOoXDUY=naYPUOuffVsRzuTYMz29yLzQCA@mail.gmail.com
2019-04-04 15:04:43 -04:00
Tom Lane d8c0bd9fef Remove now-unnecessary thread pointer arguments in pgbench.
Not required after nuking the zipfian thread-local cache.

Also add a comment about hazardous pointer punning in threadRun(),
and avoid using "thread" to refer to the threads array as a whole.

Fabien Coelho and Tom Lane, per suggestion from Alvaro Herrera

Discussion: https://postgr.es/m/alpine.DEB.2.21.1904032126060.7997@lancre
2019-04-03 17:16:09 -04:00
Stephen Frost b0b39f72b9 GSSAPI encryption support
On both the frontend and backend, prepare for GSSAPI encryption
support by moving common code for error handling into a separate file.
Fix a TODO for handling multiple status messages in the process.
Eliminate the OIDs, which have not been needed for some time.

Add frontend and backend encryption support functions.  Keep the
context initiation for authentication-only separate on both the
frontend and backend in order to avoid concerns about changing the
requested flags to include encryption support.

In postmaster, pull GSSAPI authorization checking into a shared
function.  Also share the initiator name between the encryption and
non-encryption codepaths.

For HBA, add "hostgssenc" and "hostnogssenc" entries that behave
similarly to their SSL counterparts.  "hostgssenc" requires either
"gss", "trust", or "reject" for its authentication.

Similarly, add a "gssencmode" parameter to libpq.  Supported values are
"disable", "require", and "prefer".  Notably, negotiation will only be
attempted if credentials can be acquired.  Move credential acquisition
into its own function to support this behavior.

Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring
if GSSAPI authentication was used, what principal was used, and if
encryption is being used on the connection.

Finally, add documentation for everything new, and update existing
documentation on connection security.

Thanks to Michael Paquier for the Windows fixes.

Author: Robbie Harwood, with changes to the read/write functions by me.
Reviewed in various forms and at different times by: Michael Paquier,
   Andres Freund, David Steele.
Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com
2019-04-03 15:02:33 -04:00