Commit Graph

86 Commits

Author SHA1 Message Date
Bruce Momjian
65b9671411 pg_upgrade: improve C comment about what old/new oids match 2014-03-05 10:43:45 -05:00
Bruce Momjian
7e04792a1c Update copyright for 2014
Update all files in head, and files COPYRIGHT and legal.sgml in all back
branches.
2014-01-07 16:05:30 -05:00
Peter Eisentraut
264aa14a2f pg_upgrade: Split off pg_fatal() from pg_log()
This allows decorating pg_fatal() with noreturn compiler hints, leading
to better diagnostics.

Reviewed-by: Marko Tiikkaja <marko@joh.to>
2013-10-09 22:48:41 -04:00
Stephen Frost
551938ae22 Post-pgindent cleanup
Make slightly better decisions about indentation than what pgindent
is capable of.  Mostly breaking out long function calls into one
line per argument, with a few other minor adjustments.

No functional changes- all whitespace.
pgindent ran cleanly (didn't change anything) after.
Passes all regressions.
2013-06-01 09:38:15 -04:00
Bruce Momjian
9af4159fce pgindent run for release 9.3
This is the first run of the Perl-based pgindent script.  Also update
pgindent instructions.
2013-05-29 16:58:43 -04:00
Kevin Grittner
3bf3ab8c56 Add a materialized view relations.
A materialized view has a rule just like a view and a heap and
other physical properties like a table.  The rule is only used to
populate the table, references in queries refer to the
materialized data.

This is a minimal implementation, but should still be useful in
many cases.  Currently data is only populated "on demand" by the
CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW statements.
It is expected that future releases will add incremental updates
with various timings, and that a more refined concept of defining
what is "fresh" data will be developed.  At some point it may even
be possible to have queries use a materialized in place of
references to underlying tables, but that requires the other
above-mentioned features to be working first.

Much of the documentation work by Robert Haas.
Review by Noah Misch, Thom Brown, Robert Haas, Marko Tiikkaja
Security review by KaiGai Kohei, with a decision on how best to
implement sepgsql still pending.
2013-03-03 18:23:31 -06:00
Alvaro Herrera
cd7d00ad65 Improve pg_upgrade commentary on multixact change
Per gripe from Andres Freund
2013-02-28 16:29:17 -03:00
Bruce Momjian
74205266d4 Fix pg_upgrade log file cleanup code
Recent pg_upgrade parallel improvements introduced a bug that prevented
cleanup of per-database log files.
2013-02-14 00:04:15 -05:00
Alvaro Herrera
8396447cdb Create libpgcommon, and move pg_malloc et al to it
libpgcommon is a new static library to allow sharing code among the
various frontend programs and backend; this lets us eliminate duplicate
implementations of common routines.  We avoid libpgport, because that's
intended as a place for porting issues; per discussion, it seems better
to keep them separate.

The first use case, and the only implemented by this patch, is pg_malloc
and friends, which many frontend programs were already using.

At the same time, we can use this to provide palloc emulation functions
for the frontend; this way, some palloc-using files in the backend can
also be used by the frontend cleanly.  To do this, we change palloc() in
the backend to be a function instead of a macro on top of
MemoryContextAlloc().  This was previously believed to cause loss of
performance, but this implementation has been tweaked by Tom and Andres
so that on modern compilers it provides a slight improvement over the
previous one.

This lets us clean up some places that were already with
localized hacks.

Most of the pg_malloc/palloc changes in this patch were authored by
Andres Freund. Zoltán Böszörményi also independently provided a form of
that.  libpgcommon infrastructure was authored by Álvaro.
2013-02-12 11:21:05 -03:00
Bruce Momjian
a9ceaa53be pg_upgrade: detect stale postmaster.pid lock files
If the postmaster.pid lock file exists, try starting/stopping the
cluster to check if the lock file is valid.

Per request from Tom.
2013-01-24 15:20:40 -05:00
Alvaro Herrera
0ac5ad5134 Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE".  These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE".  UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.

Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.

The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid.  Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates.  This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed.  pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.

Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header.  This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.

Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)

With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.

As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.

Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane.  There's probably room for several more tests.

There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it.  Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.

This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
	AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
	1290721684-sup-3951@alvh.no-ip.org
	1294953201-sup-2099@alvh.no-ip.org
	1320343602-sup-2290@alvh.no-ip.org
	1339690386-sup-8927@alvh.no-ip.org
	4FE5FF020200002500048A3D@gw.wicourts.gov
	4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 12:04:59 -03:00
Bruce Momjian
861ad67bd9 pg_upgrade: remove --single-transaction usage
With AtEOXact applied, --single-transaction makes pg_restore slower, and
has the potential to require lock table configuration, so remove the
argument.

Per suggestion from Tom.
2013-01-22 22:27:16 -05:00
Bruce Momjian
a89c46f9bc Allow parallel copy/link in pg_upgrade
This patch implements parallel copying/linking of files by tablespace
using the --jobs option in pg_upgrade.
2013-01-09 08:57:47 -05:00
Bruce Momjian
bd61a623ac Update copyrights for 2013
Fully update git head, and update back branches in ./COPYRIGHT and
legal.sgml files.
2013-01-01 17:15:01 -05:00
Bruce Momjian
6f1b9e4efd Add pg_upgrade --jobs parameter
Add pg_upgrade --jobs, which allows parallel dump/restore of databases,
which improves performance.
2012-12-26 19:26:30 -05:00
Bruce Momjian
6dd9584507 Improve pg_upgrade's status display
Pg_upgrade displays file names during copy and database names during
dump/restore.  Andrew Dunstan identified three bugs:

*  long file names were being truncated to 60 _leading_ characters, which
   often do not change for long file names

*  file names were truncated to 60 characters in log files

*  carriage returns were being output to log files

This commit fixes these --- it prints 60 _trailing_ characters to the
status display, and full path names without carriage returns to log
files.  It also suppresses status output to the log file unless verbose
mode is used.
2012-12-07 12:26:13 -05:00
Bruce Momjian
630cd14426 Add initdb --sync-only option to sync the data directory to durable
storage.

Have pg_upgrade use it, and enable server options fsync=off and
full_page_writes=off.

Document that users turning fsync from off to on should run initdb
--sync-only.

[ Previous commit was incorrectly applied as a git merge. ]
2012-12-03 22:47:59 -05:00
Bruce Momjian
25d1ed04a2 Revert initdb --sync-only patch that had incorrect commit messages. 2012-12-03 22:46:51 -05:00
Bruce Momjian
cd7569a546 dummy commit 2012-12-03 22:45:02 -05:00
Bruce Momjian
1c59e37665 Revert:
In pg_upgrade, remove pg_restore's --single-transaction option,
    as it throws errors in certain cases.
2012-12-01 10:21:45 -05:00
Bruce Momjian
209772350b Remove pg_restore's --single-transaction option, as it throws errors in
certain cases.
2012-12-01 09:58:00 -05:00
Bruce Momjian
5eeab9c85c In pg_upgrade, improve status wording now that we have per-database
status output for dump/restore.
2012-11-30 22:32:25 -05:00
Bruce Momjian
12ee6ec71f In pg_upgrade, dump each database separately and use
--single-transaction to restore each database schema.  This yields
performance improvements for databases with many tables.  Also, remove
split_old_dump() as it is no longer needed.
2012-11-30 16:30:13 -05:00
Bruce Momjian
a80b8037cf In pg_upgrade, pull the port number from postmaster.pid, like we do for
socket location.  Also, prevent putting the socket in the current
directory for pre-9.1 servers in live check and non-live check mode,
because pre-9.1 pg_ctl -w can't handle it.

Backpatch to 9.2.
2012-09-03 22:15:09 -04:00
Tom Lane
f763b77193 Fix pg_upgrade to cope with non-default unix_socket_directory scenarios.
When starting either an old or new postmaster, force it to place its Unix
socket in the current directory.  This makes it even harder for accidental
connections to occur during pg_upgrade, and also works around some
scenarios where the default socket location isn't usable.  (For example,
if the default location is something other than "/tmp", it might not exist
during "make check".)

When checking an already-running old postmaster, find out its actual socket
directory location from postmaster.pid, if possible.  This dodges problems
with an old postmaster having a configured location different from the
default built into pg_upgrade's libpq.  We can't find that out if the old
postmaster is pre-9.1, so also document how to cope with such scenarios
manually.

In support of this, centralize handling of the connection-related command
line options passed to pg_upgrade's subsidiary programs, such as pg_dump.
This should make future changes easier.

Bruce Momjian and Tom Lane
2012-09-03 13:52:40 -04:00
Alvaro Herrera
088c065ce8 pg_upgrade: Fix exec_prog API to be less flaky
The previous signature made it very easy to pass something other than
the printf-format specifier in the corresponding position, without any
warning from the compiler.

While at it, move some of the escaping, redirecting and quoting
responsibilities from the callers into exec_prog() itself.  This makes
the callsites cleaner.
2012-08-27 14:29:14 -04:00
Bruce Momjian
fbcfa90bb8 Fix pg_upgrade file share violation on Windows created by the commit
4741e9afb9.  This was done by adding an
optional second log file parameter to exec_prog(), and closing and
reopening the log file between system() calls.

Backpatch to 9.2.
2012-08-07 13:10:44 -04:00
Alvaro Herrera
666d494d19 pg_upgrade: abstract out copying of files from old cluster to new
Currently only pg_clog is copied, but some other directories could need
the same treatment as well, so create a subroutine to do it.

Extracted from my (somewhat larger) FOR KEY SHARE patch.
2012-07-05 11:38:42 -04:00
Heikki Linnakangas
038f3a0509 Fix pg_upgrade, broken by the xlogid/segno -> 64-bit int refactoring.
The xlogid + segno representation of a particular WAL segment doesn't make
much sense in pg_resetxlog anymore, now that we don't use that anywhere
else. Use the WAL filename instead, since that's a convenient way to name a
particular WAL segment.

I did this partially for pg_resetxlog in the original xlogid/segno -> uint64
patch, but I neglected pg_upgrade and the docs. This should now be more
complete.
2012-06-26 07:49:02 +03:00
Bruce Momjian
b9212e379c In pg_upgrade, verify that the install user has the same oid on both
clusters, and make sure the new cluster has no additional users.

Backpatch to 9.1.
2012-06-13 12:19:18 -04:00
Bruce Momjian
927d61eeff Run pgindent on 9.2 source tree in preparation for first 9.3
commit-fest.
2012-06-10 15:20:04 -04:00
Bruce Momjian
45ca31d6a7 Have pg_upgrade only use one extra log file for Win32, not two. 2012-05-25 09:10:07 -04:00
Bruce Momjian
d226e236c6 On Windows, have pg_upgrade use different two files to log pg_ctl
start/stop output, to fix file share error reported by Edmund Horner
2012-05-24 20:30:39 -04:00
Bruce Momjian
476291bef0 Adjust pg_upgrade to output a separate log file for pg_ctl output on
Windows, to avoid opening a file by multiple processes.
2012-05-23 20:19:34 -04:00
Robert Haas
5d4b60f2f2 Lots of doc corrections.
Josh Kupershmidt
2012-04-23 22:43:09 -04:00
Bruce Momjian
faec2815c8 In pg_upgrade, create a script to incrementally generate more accurate
optimizer statistics so the cluster can be made available sooner.
2012-03-16 18:54:23 -04:00
Bruce Momjian
7c8cfed78a Uppercase pg_upgrade status output title. 2012-03-14 19:22:11 -04:00
Bruce Momjian
717f6d6085 In pg_upgrade, add various logging improvements:
add ability to control permissions of created files
	have psql echo its queries for easier debugging
	output four separate log files, and delete them on success
	add -r/--retain option to keep log files after success
	make logs file append-only
	remove -g/-G/-l logging options
	sugggest tailing appropriate log file on failure
	enhance -v/--verbose behavior
2012-03-12 19:48:03 -04:00
Bruce Momjian
2127aac6ef In pg_upgrade, only lock the old cluster if link mode is used, and do it
right after we restore the schema (a common failure point), and right
before we do the link operation.

Per suggesgtions from Robert and ^!C^!^@lvaro
2012-03-05 21:20:06 -05:00
Bruce Momjian
e126958c2e Update copyright notices for year 2012. 2012-01-01 18:01:58 -05:00
Bruce Momjian
51eba98cf4 Remove pg_upgrade dependency on the 'postgres' database existing in the
new cluster.   vacuumdb, used by pg_upgrade, still has this dependency.
2011-10-28 21:19:21 -04:00
Bruce Momjian
caa1054df8 In pg_upgrade, use the new postmaster -C option to get the real data
directory, for config-only directory installs.  Only works for PG 9.2+
servers.
2011-10-07 14:40:23 -04:00
Bruce Momjian
f1312b5ed3 Add postgres.h to *.c files for pg_upgrade, ltree, and btree_gist, and
remove from local *.h files.

Per suggestion from Alvaro.
2011-08-26 21:16:24 -04:00
Bruce Momjian
17a16eeb7c In pg_upgrade, fix the -l/log option to work on Windows.
Also, double-quote the log file name in all places, to allow (on all
platforms) log file names with spaces.

Back patch to 9.0 and 9.1.
2011-07-20 18:31:03 -04:00
Peter Eisentraut
912bc4f038 Make pg_upgrade output more consistent with project style
Add errno-based output to error messages where appropriate, reformat
blocks to about 72 characters per line, use spaces instead of tabs for
indentation, and other style adjustments.
2011-07-12 07:13:51 +03:00
Bruce Momjian
bda27e502f Add pg_upgrade check to make sure the user has full access permission in
the current directory;  if not, throw an error.
2011-05-16 11:01:29 -04:00
Bruce Momjian
6c19bd96bc Allow pg_upgrade to honor libpq environment variables. Add 'local'
checks for PGHOST and PGHOSTADDR.
2011-05-16 10:48:40 -04:00
Bruce Momjian
5c5f83507c Adjust pg_upgrade FATAL error messages to have consistent newlines.
Also adjust some error message capitalization for consistency.
2011-05-06 21:47:42 -04:00
Bruce Momjian
44091442db In pg_upgrade, avoid one start/stop of the postmaster; use the -w
(wait) flag for pg_ctl start/stop;  remove the unused "quiet" flag in
the functions for starting/stopping the postmaster.
2011-04-25 20:18:23 -04:00
Bruce Momjian
bf50caf105 pgindent run before PG 9.1 beta 1. 2011-04-10 11:42:00 -04:00