Commit Graph

443 Commits

Author SHA1 Message Date
Alvaro Herrera
73c986adde Keep track of transaction commit timestamps
Transactions can now set their commit timestamp directly as they commit,
or an external transaction commit timestamp can be fed from an outside
system using the new function TransactionTreeSetCommitTsData().  This
data is crash-safe, and truncated at Xid freeze point, same as pg_clog.

This module is disabled by default because it causes a performance hit,
but can be enabled in postgresql.conf requiring only a server restart.

A new test in src/test/modules is included.

Catalog version bumped due to the new subdirectory within PGDATA and a
couple of new SQL functions.

Authors: Álvaro Herrera and Petr Jelínek

Reviewed to varying degrees by Michael Paquier, Andres Freund, Robert
Haas, Amit Kapila, Fujii Masao, Jaime Casanova, Simon Riggs, Steven
Singer, Peter Eisentraut
2014-12-03 11:53:02 -03:00
Bruce Momjian
dc9c612767 pg_upgrade: prefix Unix shell script name output with "./"
This more clearly suggests the current directory.  While this also works
on Windows, it might be confusing.

Report by Christoph Berg
2014-10-11 18:38:41 -04:00
Heikki Linnakangas
733be2a5cd Remove unnecessary initialization of local variables.
Oops, forgot these in the prveious commit.
2014-10-10 13:00:53 +03:00
Heikki Linnakangas
33755e8edf Change the way encoding and locale checks are done in pg_upgrade.
Lc_collate and lc_ctype have been per-database settings since server version
8.4, but pg_upgrade was still treating them as cluster-wide options. It
fetched the values for the template0 databases in old and new cluster, and
compared them. That's backwards; the encoding and locale of the template0
database doesn't matter, as template0 is guaranteed to contain only ASCII
characters. But if there are any other databases that exist on both clusters
(in particular template1 and postgres databases), their encodings and
locales must be compatible.

Also, make the locale comparison more lenient. If the locale names are not
equal, try to canonicalize both of them by passing them to setlocale(). We
used to do that only when upgrading from 9.1 or below, but it seems like a
good idea even with newer versions. If we change the canonical form of a
locale, this allows pg_upgrade to still work. I'm about to do just that to
fix bug #11431, by mapping a locale name that contains non-ASCII characters
to a pure-ASCII alias of the same locale.

No backpatching, because earlier versions of pg_upgrade still support
upgrading from 8.3 servers. That would be more complicated, so it doesn't
seem worth it, given that we haven't received any complaints about this
from users.
2014-10-10 10:39:32 +03:00
Bruce Momjian
35419aeb83 pg_upgrade: have pg_upgrade fail for old 9.4 JSONB format
Backpatch through 9.4
2014-09-29 20:19:59 -04:00
Bruce Momjian
c3c75fcd7a pg_upgrade: adjust C comments 2014-09-11 18:44:00 -04:00
Bruce Momjian
acc8e41681 pg_upgrade: compare control version, not catalog version
Also modify test for the possibility the large object value might not
exist in the old cluster.

Fix for commit e1598a15f4
2014-09-10 20:22:10 -04:00
Bruce Momjian
e1598a15f4 pg_upgrade: check for large object size compatibility 2014-09-10 19:23:36 -04:00
Bruce Momjian
a74a4aa23b pg_upgrade: preserve the timestamp epoch
This is useful for replication tools like Slony and Skytools.

Report by Sergey Konoplev
2014-09-05 19:19:41 -04:00
Peter Eisentraut
2d759341d9 Fix whitespace 2014-08-26 17:26:45 -04:00
Bruce Momjian
5d16332e96 pg_upgrade: use CTE query rather than temp table
Now that 8.3 is not supported, we can use a CTE and not temp tables.
This allows for auto-oid assignment protection in a future patch.
2014-08-25 21:22:08 -04:00
Bruce Momjian
6cb74a67e2 revert "Throw error for ALTER TABLE RESET of an invalid option"
Reverts commits 73d78e11a0 and
b0488e5c4f.  Also reverts pg_upgrade
changes.
2014-08-25 20:07:37 -04:00
Bruce Momjian
2209b3923a pg_upgrade: remove support for 8.3 old clusters
This trims down the code, and is in preparation for hardening pg_upgrade
against auto-oid assignment.
2014-08-25 20:05:07 -04:00
Bruce Momjian
b0488e5c4f pg_upgrade: add SET log_min_error_statement = warning
Add to commit 73d78e11a0
2014-08-25 17:24:33 -04:00
Bruce Momjian
73d78e11a0 Throw error for ALTER TABLE RESET of an invalid option
Also adjust pg_upgrade to not use this method for optional TOAST table
creation.

Patch by Fabrízio de Royes Mello
2014-08-25 17:06:40 -04:00
Bruce Momjian
ebe30ad59b pg_ctl, pg_upgrade: allow multiple -o/-O options, append them
Report by Pavel Raiskup
2014-08-25 16:30:26 -04:00
Bruce Momjian
7567d94910 pg_upgrade: adjust logging to use QUERY_ALLOC lengths
Allows the logging to print the entire text of failed queries, rather
than a truncated version.
2014-08-20 13:04:45 -04:00
Peter Eisentraut
9ee9ca545b Fix whitespace 2014-08-15 14:52:06 -04:00
Bruce Momjian
ea9b8abd6f pg_upgrade: fix define adjusted during testing
from commit e4c3c99ac3
2014-08-15 12:28:18 -04:00
Bruce Momjian
e4c3c99ac3 pg_upgrade: error if run from top of new PGDATA on Windows
Print a clear error message in this case, rather than wait for initdb
--sync-only to generate a "Permission denied" error.
2014-08-15 12:04:03 -04:00
Bruce Momjian
4c6780fd17 pg_upgrade: prevent oid conflicts with new-cluster TOAST tables
Previously, TOAST tables only required in the new cluster could cause
oid conflicts if they were auto-numbered and a later conflicting oid had
to be assigned.

Backpatch through 9.3
2014-08-07 14:56:13 -04:00
Bruce Momjian
ee646df59a pg_upgrade: assume user is install user
The user specified to the upgrade was effectively the install user, but
that was not clearly stated in the comments, documentation, or error
messages.
2014-08-04 11:56:16 -04:00
Bruce Momjian
b4bd6f6b05 pg_upgrade: remove reference to autovacuum_multixact_freeze_max_age
autovacuum_multixact_freeze_max_age was added as a pg_ctl start
parameter in 9.3.X to prevent autovacuum from running.  However, only
some 9.3.X releases have autovacuum_multixact_freeze_max_age as it was
added in a minor PG 9.3 release.  It also isn't needed because -b turns
off autovacuum in 9.1+.

Without this fix, trying to upgrade from an early 9.3 release to 9.4
would fail.

Report by EDB

Backpatch through 9.3
2014-08-04 11:46:05 -04:00
Bruce Momjian
d826d8ecce pg_upgrade: improve C comment wording 2014-07-29 22:09:03 -04:00
Peter Eisentraut
e0da6ec618 pg_upgrade: Fix spacing in help output 2014-07-15 15:08:16 -04:00
Bruce Momjian
3088cc3704 pg_upgrade: allow upgrades for new-only TOAST tables
Previously, when calculations on the need for toast tables changed,
pg_upgrade could not handle cases where the new cluster needed a TOAST
table and the old cluster did not.  (It already handled the opposite
case.)  This fixes the "OID mismatch" error typically generated in this
case.

Backpatch through 9.2
2014-07-07 13:24:08 -04:00
Noah Misch
333b7db8b3 Consistently pass an "unsigned char" to ctype.h functions.
The isxdigit() calls relied on undefined behavior.  The isascii() call
was well-defined, but our prevailing style is to include the cast.
Back-patch to 9.4, where the isxdigit() calls were introduced.
2014-07-06 00:29:51 -04:00
Bruce Momjian
a61daa14d5 pg_upgrade: preserve database and relation minmxid values
Also set these values for pre-9.3 old clusters that don't have values to
preserve.

Analysis by Alvaro

Backpatch through 9.3
2014-07-02 15:29:38 -04:00
Bruce Momjian
89cf2d5203 pg_upgrade: no need to remove "members" files for pre-9.3 upgrades
Per analysis by Alvaro

Backpatch through 9.3
2014-07-02 13:11:05 -04:00
Tom Lane
fbb1d7d73f Allow CREATE/ALTER DATABASE to manipulate datistemplate and datallowconn.
Historically these database properties could be manipulated only by
manually updating pg_database, which is error-prone and only possible for
superusers.  But there seems no good reason not to allow database owners to
set them for their databases, so invent CREATE/ALTER DATABASE options to do
that.  Adjust a couple of places that were doing it the hard way to use the
commands instead.

Vik Fearing, reviewed by Pavel Stehule
2014-07-01 20:10:38 -04:00
Bruce Momjian
55863274d9 pg_upgrade: update C comments about pg_dumpall
There were some C comments that hadn't been updated from the switch of
using only pg_dumpall to using pg_dump and pg_dumpall, so update them.
Also, don't bother using --schema-only for pg_dumpall --globals-only.

Backpatch through 9.4
2014-06-30 19:56:23 -04:00
Bruce Momjian
0f7482733a pg_upgrade: remove pg_multixact files left by initdb
This fixes a bug that caused vacuum to fail when the '0000' files left
by initdb were accessed as part of vacuum's cleanup of old pg_multixact
files.

Backpatch through 9.3
2014-06-24 16:11:06 -04:00
Noah Misch
686f362bee Fix contrib/pg_upgrade/test.sh for $PWD containing spaces.
Most of the necessary quoting was in place; this catches the exceptions.
2014-06-19 21:47:08 -04:00
Noah Misch
be76a6d39e Secure Unix-domain sockets of "make check" temporary clusters.
Any OS user able to access the socket can connect as the bootstrap
superuser and proceed to execute arbitrary code as the OS user running
the test.  Protect against that by placing the socket in a temporary,
mode-0700 subdirectory of /tmp.  The pg_regress-based test suites and
the pg_upgrade test suite were vulnerable; the $(prove_check)-based test
suites were already secure.  Back-patch to 8.4 (all supported versions).
The hazard remains wherever the temporary cluster accepts TCP
connections, notably on Windows.

As a convenient side effect, this lets testing proceed smoothly in
builds that override DEFAULT_PGSOCKET_DIR.  Popular non-default values
like /var/run/postgresql are often unwritable to the build user.

Security: CVE-2014-0067
2014-06-14 09:41:13 -04:00
Peter Eisentraut
2f8e68bd7e Fix whitespace 2014-05-15 13:23:31 -04:00
Bruce Momjian
2e6602dfc2 pg_update: slight major version adjustment 2014-05-14 21:13:59 -04:00
Tom Lane
4456763c78 Fix version check for pg_upgrade line type crosscheck.
Per buildfarm.
2014-05-14 20:23:32 -04:00
Bruce Momjian
bb797b6404 pg_upgrade: error out on 'line' data type usage
The data type internal format changed in 9.4.  Also mention this in the
9.4 release notes.
2014-05-14 16:26:13 -04:00
Bruce Momjian
0a78320057 pgindent run for 9.4
This includes removing tabs after periods in C comments, which was
applied to back branches, so this change should not effect backpatching.
2014-05-06 12:12:18 -04:00
Heikki Linnakangas
a692ee5870 Replace SYSTEMQUOTEs with Windows-specific wrapper functions.
It's easy to forget using SYSTEMQUOTEs when constructing command strings
for system() or popen(). Even if we fix all the places missing it now, it is
bound to be forgotten again in the future. Introduce wrapper functions that
do the the extra quoting for you, and get rid of SYSTEMQUOTEs in all the
callers.

We previosly used SYSTEMQUOTEs in all the hard-coded command strings, and
this doesn't change the behavior of those. But user-supplied commands, like
archive_command, restore_command, COPY TO/FROM PROGRAM calls, as well as
pgbench's \shell, will now gain an extra pair of quotes. That is desirable,
but if you have existing scripts or config files that include an extra
pair of quotes, those might need to be adjusted.

Reviewed by Amit Kapila and Tom Lane
2014-05-05 16:07:40 +03:00
Heikki Linnakangas
503de54621 Add missing SYSTEMQUOTEs
Some popen() calls were missing SYSTEMQUOTEs, which caused initdb and
pg_upgrade to fail on Windows, if the installation path contained both
spaces and @ signs.

Patch by Nikhil Deshpande. Backpatch to all supported versions.
2014-04-30 10:35:52 +03:00
Bruce Momjian
c1275cf741 pg_upgrade: throw an error for non-existent tablespace directories
Non-existent tablespace directory references can occur if user
tablespaces are created inside data directories and the data directory
is renamed in preparation for running pg_upgrade, and the symbolic links
are not updated.

Backpatch to 9.3.
2014-04-17 11:42:21 -04:00
Bruce Momjian
95cb917297 pg_upgrade: remove redundant include files
The files were already included by pg_upgrade.h.
2014-04-16 13:27:07 -04:00
Peter Eisentraut
c92c3d50d7 vacuumdb: Add option --analyze-in-stages
Add vacuumdb option --analyze-in-stages which runs ANALYZE three times
with different configuration settings, adopting the logic from the
analyze_new_cluster.sh script that pg_upgrade generates.  That way,
users of pg_dump/pg_restore can also use that functionality.

Change pg_upgrade to create the script so that it calls vacuumdb instead
of implementing the logic itself.
2014-04-15 00:44:25 -04:00
Noah Misch
8f5578d0f9 Revert "Secure Unix-domain sockets of "make check" temporary clusters."
About half of the buildfarm members use too-long directory names,
strongly suggesting that this approach is a dead end.
2014-03-29 03:12:00 -04:00
Noah Misch
31c6e54ec9 Secure Unix-domain sockets of "make check" temporary clusters.
Any OS user able to access the socket can connect as the bootstrap
superuser and in turn execute arbitrary code as the OS user running the
test.  Protect against that by placing the socket in the temporary data
directory, which has mode 0700 thanks to initdb.  Back-patch to 8.4 (all
supported versions).  The hazard remains wherever the temporary cluster
accepts TCP connections, notably on Windows.

Attempts to run "make check" from a directory with a long name will now
fail.  An alternative not sharing that problem was to place the socket
in a subdirectory of /tmp, but that is only secure if /tmp is sticky.
The PG_REGRESS_SOCK_DIR environment variable is available as a
workaround when testing from long directory paths.

As a convenient side effect, this lets testing proceed smoothly in
builds that override DEFAULT_PGSOCKET_DIR.  Popular non-default values
like /var/run/postgresql are often unwritable to the build user.

Security: CVE-2014-0067
2014-03-29 00:52:56 -04:00
Bruce Momjian
ba08155bd5 pg_upgrade: use pg_usleep(); remove Windows workaround 2014-03-26 10:22:39 -04:00
Tom Lane
0268d21e5d Fix unportable shell-script syntax in pg_upgrade's test.sh.
I discovered the hard way that on some old shells, the locution
    FOO=""   unset FOO
does not behave the same as
    FOO="";  unset FOO
and in fact leaves FOO set to an empty string.  test.sh was inconsistently
spelling it different ways on adjacent lines.

This got broken relatively recently, in commit c737a2e56, so the lack of
field reports to date doesn't represent a lot of evidence that the problem
is rare.
2014-03-16 21:55:27 -04:00
Bruce Momjian
65b9671411 pg_upgrade: improve C comment about what old/new oids match 2014-03-05 10:43:45 -05:00
Tom Lane
60ff2fdd99 Centralize getopt-related declarations in a new header file pg_getopt.h.
We used to have externs for getopt() and its API variables scattered
all over the place.  Now that we find we're going to need to tweak the
variable declarations for Cygwin, it seems like a good idea to have
just one place to tweak.

In this commit, the variables are declared "#ifndef HAVE_GETOPT_H".
That may or may not work everywhere, but we'll soon find out.

Andres Freund
2014-02-15 14:31:30 -05:00