Commit Graph

2726 Commits

Author SHA1 Message Date
Noah Misch 1ed8e771ad Remove dead-since-introduction pgcrypto code.
Marko Tiikkaja
2014-11-02 21:43:39 -05:00
Peter Eisentraut 83dc5908c2 pg_test_fsync: Update output format
Apparently, computers are now a bit faster than when this was first
added, so we need to make room for a digit or two in the ops/sec format.

While we're at it, adjust some of the other output for a more consistent
line length.
2014-10-20 15:36:51 -04:00
Tom Lane 488a7c9ccf Fix file-identification comment in contrib/pgcrypto/pgcrypto--1.2.sql.
Cosmetic oversight in commit 32984d8fc3.

Marko Tiikkaja
2014-10-20 10:53:57 -04:00
Tom Lane b2cbced9ee Support timezone abbreviations that sometimes change.
Up to now, PG has assumed that any given timezone abbreviation (such as
"EDT") represents a constant GMT offset in the usage of any particular
region; we had a way to configure what that offset was, but not for it
to be changeable over time.  But, as with most things horological, this
view of the world is too simplistic: there are numerous regions that have
at one time or another switched to a different GMT offset but kept using
the same timezone abbreviation.  Almost the entire Russian Federation did
that a few years ago, and later this month they're going to do it again.
And there are similar examples all over the world.

To cope with this, invent the notion of a "dynamic timezone abbreviation",
which is one that is referenced to a particular underlying timezone
(as defined in the IANA timezone database) and means whatever it currently
means in that zone.  For zones that use or have used daylight-savings time,
the standard and DST abbreviations continue to have the property that you
can specify standard or DST time and get that time offset whether or not
DST was theoretically in effect at the time.  However, the abbreviations
mean what they meant at the time in question (or most recently before that
time) rather than being absolutely fixed.

The standard abbreviation-list files have been changed to use this behavior
for abbreviations that have actually varied in meaning since 1970.  The
old simple-numeric definitions are kept for abbreviations that have not
changed, since they are a bit faster to resolve.

While this is clearly a new feature, it seems necessary to back-patch it
into all active branches, because otherwise use of Russian zone
abbreviations is going to become even more problematic than it already was.
This change supersedes the changes in commit 513d06ded et al to modify the
fixed meanings of the Russian abbreviations; since we've not shipped that
yet, this will avoid an undesirably incompatible (not to mention incorrect)
change in behavior for timestamps between 2011 and 2014.

This patch makes some cosmetic changes in ecpglib to keep its usage of
datetime lookup tables as similar as possible to the backend code, but
doesn't do anything about the increasingly obsolete set of timezone
abbreviation definitions that are hard-wired into ecpglib.  Whatever we
do about that will likely not be appropriate material for back-patching.
Also, a potential free() of a garbage pointer after an out-of-memory
failure in ecpglib has been fixed.

This patch also fixes pre-existing bugs in DetermineTimeZoneOffset() that
caused it to produce unexpected results near a timezone transition, if
both the "before" and "after" states are marked as standard time.  We'd
only ever thought about or tested transitions between standard and DST
time, but that's not what's happening when a zone simply redefines their
base GMT offset.

In passing, update the SGML documentation to refer to the Olson/zoneinfo/
zic timezone database as the "IANA" database, since it's now being
maintained under the auspices of IANA.
2014-10-16 15:22:10 -04:00
Tom Lane 90063a7612 Print planning time only in EXPLAIN ANALYZE, not plain EXPLAIN.
We've gotten enough push-back on that change to make it clear that it
wasn't an especially good idea to do it like that.  Revert plain EXPLAIN
to its previous behavior, but keep the extra output in EXPLAIN ANALYZE.
Per discussion.

Internally, I set this up as a separate flag ExplainState.summary that
controls printing of planning time and execution time.  For now it's
just copied from the ANALYZE option, but we could consider exposing it
to users.
2014-10-15 18:50:13 -04:00
Heikki Linnakangas 98aed6c721 Add --latency-limit option to pgbench.
This allows transactions that take longer than specified limit to be counted
separately. With --rate, transactions that are already late by the time we
get to execute them are skipped altogether. Using --latency-limit with
--rate allows you to "catch up" more quickly, if there's a hickup in the
server causing a lot of transactions to stall momentarily.

Fabien COELHO, reviewed by Rukh Meski and heavily refactored by me.
2014-10-13 20:50:24 +03:00
Bruce Momjian dc9c612767 pg_upgrade: prefix Unix shell script name output with "./"
This more clearly suggests the current directory.  While this also works
on Windows, it might be confusing.

Report by Christoph Berg
2014-10-11 18:38:41 -04:00
Heikki Linnakangas 733be2a5cd Remove unnecessary initialization of local variables.
Oops, forgot these in the prveious commit.
2014-10-10 13:00:53 +03:00
Heikki Linnakangas 33755e8edf Change the way encoding and locale checks are done in pg_upgrade.
Lc_collate and lc_ctype have been per-database settings since server version
8.4, but pg_upgrade was still treating them as cluster-wide options. It
fetched the values for the template0 databases in old and new cluster, and
compared them. That's backwards; the encoding and locale of the template0
database doesn't matter, as template0 is guaranteed to contain only ASCII
characters. But if there are any other databases that exist on both clusters
(in particular template1 and postgres databases), their encodings and
locales must be compatible.

Also, make the locale comparison more lenient. If the locale names are not
equal, try to canonicalize both of them by passing them to setlocale(). We
used to do that only when upgrading from 9.1 or below, but it seems like a
good idea even with newer versions. If we change the canonical form of a
locale, this allows pg_upgrade to still work. I'm about to do just that to
fix bug #11431, by mapping a locale name that contains non-ASCII characters
to a pure-ASCII alias of the same locale.

No backpatching, because earlier versions of pg_upgrade still support
upgrading from 8.3 servers. That would be more complicated, so it doesn't
seem worth it, given that we haven't received any complaints about this
from users.
2014-10-10 10:39:32 +03:00
Heikki Linnakangas 86f809088c Fix typo in error message. 2014-10-02 15:51:31 +03:00
Heikki Linnakangas 84f0ea3f68 Refactor pgbench log-writing code to a separate function.
The doCustom function was incredibly long, this makes it a little bit more
readable.
2014-10-02 14:01:19 +03:00
Heikki Linnakangas 32984d8fc3 Add functions for dealing with PGP armor header lines to pgcrypto.
This add a new pgp_armor_headers function to extract armor headers from an
ASCII-armored blob, and a new overloaded variant of the armor function, for
constructing an ASCII-armor with extra headers.

Marko Tiikkaja and me.
2014-10-01 16:03:39 +03:00
Andres Freund 0ef3c29a4b Improve documentation about binary/textual output mode for output plugins.
Also improve related error message as it contributed to the confusion.

Discussion: CAB7nPqQrqFzjqCjxu4GZzTrD9kpj6HMn9G5aOOMwt1WZ8NfqeA@mail.gmail.com,
    CAB7nPqQXc_+g95zWnqaa=mVQ4d3BVRs6T41frcEYi2ocUrR3+A@mail.gmail.com

Per discussion between Michael Paquier, Robert Haas and Andres Freund

Backpatch to 9.4 where logical decoding was introduced.
2014-10-01 13:22:17 +02:00
Bruce Momjian 35419aeb83 pg_upgrade: have pg_upgrade fail for old 9.4 JSONB format
Backpatch through 9.4
2014-09-29 20:19:59 -04:00
Andres Freund 9b6bb9b471 Define META_FREE in a way that doesn't cause -Wempty-body warnings.
That get rids of the only -Wempty-body warning when compiling postgres
with gcc 4.8/9. As 6550b901f shows, it's useful to be able to use that
option routinely.

Without asserts there's many more warnings, but that's food for
another commit.
2014-09-26 02:55:38 +02:00
Heikki Linnakangas 1dcfb8da09 Refactor space allocation for base64 encoding/decoding in pgcrypto.
Instead of trying to accurately calculate the space needed, use a StringInfo
that's enlarged as needed. This is just moving things around currently - the
old code was not wrong - but this is in preparation for a patch that adds
support for extra armor headers, and would make the space calculation more
complicated.

Marko Tiikkaja
2014-09-25 16:36:58 +03:00
Andres Freund 604f7956b9 Improve code around the recently added rm_identify rmgr callback.
There are four weaknesses in728f152e07f998d2cb4fe5f24ec8da2c3bda98f2:

* append_init() in heapdesc.c was ugly and required that rm_identify
  return values are only valid till the next call. Instead just add a
  couple more switch() cases for the INIT_PAGE cases. Now the returned
  value will always be valid.
* a couple rm_identify() callbacks missed masking xl_info with
  ~XLR_INFO_MASK.
* pg_xlogdump didn't map a NULL rm_identify to UNKNOWN or a similar
  string.
* append_init() was called when id=NULL - which should never actually
  happen. But it's better to be careful.
2014-09-22 17:49:34 +02:00
Tom Lane 898f8a96ef Fix failure of contrib/auto_explain to print per-node timing information.
This has been broken since commit af7914c662,
which added the EXPLAIN (TIMING) option.  Although that commit included
updates to auto_explain, they evidently weren't tested very carefully,
because the code failed to print node timings even when it should, due to
failure to set es.timing in the ExplainState struct.  Reported off-list by
Neelakanth Nadgir of Salesforce.

In passing, clean up the documentation for auto_explain's options a
little bit, including re-ordering them into what seems to me a more
logical order.
2014-09-19 13:19:27 -04:00
Andres Freund bdd5726c34 Add the capability to display summary statistics to pg_xlogdump.
The new --stats/--stats=record options to pg_xlogdump display per
rmgr/per record statistics about the parsed WAL. This is useful to
understand what the WAL primarily consists of, to allow targeted
optimizations on application, configuration, and core code level.

It is likely that we will want to fine tune the statistics further,
but the feature already is quite helpful.

Author: Abhijit Menon-Sen, slightly editorialized by me
Reviewed-By: Andres Freund, Dilip Kumar and Furuya Osamu
Discussion: 20140604104716.GA3989@toroid.org
2014-09-19 16:33:16 +02:00
Andres Freund 728f152e07 Add rmgr callback to name xlog record types for display purposes.
This is primarily useful for the upcoming pg_xlogdump --stats feature,
but also allows to remove some duplicated code in the rmgr_desc
routines.

Due to the separation and harmonization, the output of dipsplayed
records changes somewhat. But since this isn't enduser oriented
content that's ok.

It's potentially desirable to further change pg_xlogdump's display of
records. It previously wasn't possible to show the record type
separately from the description forcing it to be in the last
column. But that's better done in a separate commit.

Author: Abhijit Menon-Sen, slightly editorialized by me
Reviewed-By: Álvaro Herrera, Andres Freund, and Heikki Linnakangas
Discussion: 20140604104716.GA3989@toroid.org
2014-09-19 16:20:29 +02:00
Bruce Momjian c3c75fcd7a pg_upgrade: adjust C comments 2014-09-11 18:44:00 -04:00
Heikki Linnakangas 01a2bfd172 Fix Windows build.
I renamed a variable, but missed an #ifdef WIN32 block.
2014-09-11 15:15:40 +03:00
Heikki Linnakangas 54a2d5b37b Simplify calculation of Poisson distributed delays in pgbench --rate mode.
The previous coding first generated a uniform random value between 0.0 and
1.0, then converted that to an integer between 1 and 10000, and divided that
again by 10000. Those conversions are unnecessary; we can use the double
value that pg_erand48() returns directly. While we're at it, put the logic
into a helper function, getPoissonRand().

The largest delay generated by the old coding was about 9.2 times the
average, because of the way the uniformly distributed value used for the
calculation was truncated to 1/10000 granularity. The new coding doesn't
have such clamping. With my laptop's DBL_MIN value, the maximum delay with
the new coding is about 700x the average. That seems acceptable - any
reasonable pgbench session should last long enough to average that out.

Backpatch to 9.4.
2014-09-11 13:00:48 +03:00
Heikki Linnakangas 02e3bcc661 Change the way latency is calculated with pgbench --rate option.
The reported latency values now include the "schedule lag" time, that is,
the time between the transaction's scheduled start time and the time it
actually started. This relates better to a model where requests arrive at a
certain rate, and we are interested in the response time to the end user or
application, rather than the response time of the database itself.

Also, when --rate is used, include the schedule lag time in the log output.

The --rate option is new in 9.4, so backpatch to 9.4. It seems better to
make this change in 9.4, while we're still in the beta period, than ship a
9.4 version that calculates the values differently than 9.5.
2014-09-11 12:57:32 +03:00
Bruce Momjian acc8e41681 pg_upgrade: compare control version, not catalog version
Also modify test for the possibility the large object value might not
exist in the old cluster.

Fix for commit e1598a15f4
2014-09-10 20:22:10 -04:00
Bruce Momjian e1598a15f4 pg_upgrade: check for large object size compatibility 2014-09-10 19:23:36 -04:00
Peter Eisentraut 220bb39dee doc: Reflect renaming of Mac OS X to OS X
bug #10528
2014-09-09 13:56:29 -04:00
Bruce Momjian a74a4aa23b pg_upgrade: preserve the timestamp epoch
This is useful for replication tools like Slony and Skytools.

Report by Sergey Konoplev
2014-09-05 19:19:41 -04:00
Peter Eisentraut 303f4d1012 Assorted message fixes and improvements 2014-09-05 01:25:27 -04:00
Andres Freund d6fa44fce7 Add skip-empty-xacts option to test_decoding for use in the regression tests.
The regression tests for contrib/test_decoding regularly failed on
postgres instances that were very slow. Either because the hardware
itself was slow or because very expensive debugging options like
CLOBBER_CACHE_ALWAYS were used.

The reason they failed was just that some additional transactions were
decoded. Analyze and vacuum, triggered by autovac.

To fix just add a option to test_decoding to only display transactions
in which a change was actually displayed. That's not pretty because it
removes information from the tests; but better than constantly failing
tests in very likely harmless ways.

Backpatch to 9.4 where logical decoding was introduced.

Discussion: 20140629142511.GA26930@awork2.anarazel.de
2014-09-01 15:59:44 +02:00
Andres Freund 4b4b680c3d Make backend local tracking of buffer pins memory efficient.
Since the dawn of time (aka Postgres95) multiple pins of the same
buffer by one backend have been optimized not to modify the shared
refcount more than once. This optimization has always used a NBuffer
sized array in each backend keeping track of a backend's pins.

That array (PrivateRefCount) was one of the biggest per-backend memory
allocations, depending on the shared_buffers setting. Besides the
waste of memory it also has proven to be a performance bottleneck when
assertions are enabled as we make sure that there's no remaining pins
left at the end of transactions. Also, on servers with lots of memory
and a correspondingly high shared_buffers setting the amount of random
memory accesses can also lead to poor cpu cache efficiency.

Because of these reasons a backend's buffers pins are now kept track
of in a small statically sized array that overflows into a hash table
when necessary. Benchmarks have shown neutral to positive performance
results with considerably lower memory usage.

Patch by me, review by Robert Haas.

Discussion: 20140321182231.GA17111@alap3.anarazel.de
2014-08-30 14:03:21 +02:00
Tom Lane 7f7eec89b6 Fix citext upgrade script for disallowance of oidvector element assignment.
In commit 45e02e3232, we intentionally
disallowed updates on individual elements of oidvector columns.  While that
still seems like a sane idea in the abstract, we (I) forgot that citext's
"upgrade from unpackaged" script did in fact perform exactly such updates,
in order to fix the problem that citext indexes should have a collation
but would not in databases dumped or upgraded from pre-9.1 installations.

Even if we wanted to add casts to allow such updates, there's no practical
way to do so in the back branches, so the only real alternative is to make
citext's kluge even klugier.  In this patch, I cast the oidvector to text,
fix its contents with regexp_replace, and cast back to oidvector.  (Ugh!)

Since the aforementioned commit went into all active branches, we have to
fix this in all branches that contain the now-broken update script.

Per report from Eric Malm.
2014-08-28 18:21:05 -04:00
Peter Eisentraut 2d759341d9 Fix whitespace 2014-08-26 17:26:45 -04:00
Andres Freund 57ca1d4f01 Specify the port in dblink and postgres_fdw tests.
That allows to run those tests against a postmaster listening on a
nonstandard port without requiring to export PGPORT in postmaster's
environment.

This still doesn't support connecting to a nondefault host without
configuring it in postmaster's environment. That's harder and less
frequently used though. So this is a useful step.
2014-08-26 12:28:08 +02:00
Andres Freund ddc2504dbc Don't hardcode contrib_regression dbname in postgres_fdw and dblink tests.
That allows parallel installcheck to succeed inside contrib/. The
output is not particularly pretty unless make's -O option to
synchronize the output is used.

There's other tests, outside contrib, that use a hardcoded,
non-unique, database name. Those prohibit paralell installcheck to be
used across more directories; but that's something for a separate
patch.
2014-08-26 12:27:26 +02:00
Bruce Momjian a7ae1dcf49 pg_upgrade: prevent automatic oid assignment
Prevent automatic oid assignment when in binary upgrade mode.  Also
throw an error when contrib/pg_upgrade_support functions are called when
not in binary upgrade mode.

This prevent automatically-assigned oids from conflicting with later
pre-assigned oids coming from the old cluster.  It also makes sure oids
are preserved in call important cases.
2014-08-25 22:19:05 -04:00
Bruce Momjian 5d16332e96 pg_upgrade: use CTE query rather than temp table
Now that 8.3 is not supported, we can use a CTE and not temp tables.
This allows for auto-oid assignment protection in a future patch.
2014-08-25 21:22:08 -04:00
Bruce Momjian 6cb74a67e2 revert "Throw error for ALTER TABLE RESET of an invalid option"
Reverts commits 73d78e11a0 and
b0488e5c4f.  Also reverts pg_upgrade
changes.
2014-08-25 20:07:37 -04:00
Bruce Momjian 2209b3923a pg_upgrade: remove support for 8.3 old clusters
This trims down the code, and is in preparation for hardening pg_upgrade
against auto-oid assignment.
2014-08-25 20:05:07 -04:00
Bruce Momjian b0488e5c4f pg_upgrade: add SET log_min_error_statement = warning
Add to commit 73d78e11a0
2014-08-25 17:24:33 -04:00
Bruce Momjian 73d78e11a0 Throw error for ALTER TABLE RESET of an invalid option
Also adjust pg_upgrade to not use this method for optional TOAST table
creation.

Patch by Fabrízio de Royes Mello
2014-08-25 17:06:40 -04:00
Bruce Momjian ebe30ad59b pg_ctl, pg_upgrade: allow multiple -o/-O options, append them
Report by Pavel Raiskup
2014-08-25 16:30:26 -04:00
Andres Freund d153b80161 Fix typos in some error messages thrown by extension scripts when fed to psql.
Some of the many error messages introduced in 458857cc missed 'FROM
unpackaged'. Also e016b724 and 45ffeb7e forgot to quote extension
version numbers.

Backpatch to 9.1, just like 458857cc which introduced the messages. Do
so because the error messages thrown when the wrong command is copy &
pasted aren't easy to understand.
2014-08-25 18:30:37 +02:00
Heikki Linnakangas 3cd934f635 Don't track DEALLOCATE in pg_stat_statements.
We also don't track PREPARE, nor do we track planning time in general, so
let's ignore DEALLOCATE as well for consistency.

Backpatch to 9.4, but not further than that. Although it seems unlikely that
anyone is relying on the current behavior, this is a behavioral change.

Fabien Coelho
2014-08-25 19:13:24 +03:00
Andres Freund ebf20f65d9 Fix newly introduced misspelling of existence in pg_buffercache.
Peter Geoghegan
2014-08-22 17:49:47 +02:00
Andres Freund f57791985a Add pinning_backends column to the pg_buffercache extension.
The new column shows how many backends have a buffer pinned. That can
be useful during development or to diagnose production issues
e.g. caused by vacuum waiting for cleanup locks.

To handle upgrades transparently - the extension might be used in
views - deal with callers expecting the old number of columns.

Reviewed by Fujii Masao and Rajeev rastogi.
2014-08-22 00:28:37 +02:00
Bruce Momjian 7567d94910 pg_upgrade: adjust logging to use QUERY_ALLOC lengths
Allows the logging to print the entire text of failed queries, rather
than a truncated version.
2014-08-20 13:04:45 -04:00
Noah Misch fb2aece8ae Replace a few strncmp() calls with strlcpy().
strncmp() is a specialized API unsuited for routine copying into
fixed-size buffers.  On a system where the length of a single filename
can exceed MAXPGPATH, the pg_archivecleanup change prevents a simple
crash in the subsequent strlen().  Few filesystems support names that
long, and calling pg_archivecleanup with untrusted input is still not a
credible use case.  Therefore, no back-patch.

David Rowley
2014-08-18 22:59:31 -04:00
Peter Eisentraut 9ee9ca545b Fix whitespace 2014-08-15 14:52:06 -04:00
Bruce Momjian ea9b8abd6f pg_upgrade: fix define adjusted during testing
from commit e4c3c99ac3
2014-08-15 12:28:18 -04:00
Bruce Momjian e4c3c99ac3 pg_upgrade: error if run from top of new PGDATA on Windows
Print a clear error message in this case, rather than wait for initdb
--sync-only to generate a "Permission denied" error.
2014-08-15 12:04:03 -04:00
Peter Eisentraut 1d678bf7bc Add some noreturn attributes based on compiler recommendations 2014-08-13 22:40:48 -04:00
Tatsuo Ishii 51222a1766 Enhance pgbench's option checking.
Now benchmarking options such as -c cannot be used if initializing
option (-i) is specified.  Also initializing options such as -F cannot
be used if initializing option is not specified.

Tatsuo Ishii and Fabien COELHO.
2014-08-12 17:27:08 +09:00
Bruce Momjian 4c6780fd17 pg_upgrade: prevent oid conflicts with new-cluster TOAST tables
Previously, TOAST tables only required in the new cluster could cause
oid conflicts if they were auto-numbered and a later conflicting oid had
to be assigned.

Backpatch through 9.3
2014-08-07 14:56:13 -04:00
Bruce Momjian ee646df59a pg_upgrade: assume user is install user
The user specified to the upgrade was effectively the install user, but
that was not clearly stated in the comments, documentation, or error
messages.
2014-08-04 11:56:16 -04:00
Bruce Momjian b4bd6f6b05 pg_upgrade: remove reference to autovacuum_multixact_freeze_max_age
autovacuum_multixact_freeze_max_age was added as a pg_ctl start
parameter in 9.3.X to prevent autovacuum from running.  However, only
some 9.3.X releases have autovacuum_multixact_freeze_max_age as it was
added in a minor PG 9.3 release.  It also isn't needed because -b turns
off autovacuum in 9.1+.

Without this fix, trying to upgrade from an early 9.3 release to 9.4
would fail.

Report by EDB

Backpatch through 9.3
2014-08-04 11:46:05 -04:00
Heikki Linnakangas cd74a151b5 Windows doesn't have M_PI; define it ourselves when needed.
This should fix the Windows build, broken by commit ed802e7d.
2014-08-04 18:06:34 +03:00
Robert Haas ed802e7dc3 pgbench: Allow \setrandom to generate Gaussian/exponential distributions.
Mitsumasa KONDO and Fabien COELHO, with further wordsmithing by me.
2014-07-30 13:22:08 -04:00
Bruce Momjian d826d8ecce pg_upgrade: improve C comment wording 2014-07-29 22:09:03 -04:00
Alvaro Herrera 0531549801 Avoid uselessly looking up old LOCK_ONLY multixacts
Commit 0ac5ad5134 removed an optimization in multixact.c that skipped
fetching members of MultiXactId that were older than our
OldestVisibleMXactId value.  The reason this was removed is that it is
possible for multixacts that contain updates to be older than that
value.  However, if the caller is certain that the multi does not
contain an update (because the infomask bits say so), it can pass this
info down to GetMultiXactIdMembers, enabling it to use the old
optimization.

Pointed out by Andres Freund in 20131121200517.GM7240@alap2.anarazel.de
2014-07-29 15:41:06 -04:00
Tom Lane 27cef0a561 Check block number against the correct fork in get_raw_page().
get_raw_page tried to validate the supplied block number against
RelationGetNumberOfBlocks(), which of course is only right when
accessing the main fork.  In most cases, the main fork is longer
than the others, so that the check was too weak (allowing a
lower-level error to be reported, but no real harm to be done).
However, very small tables could have an FSM larger than their heap,
in which case the mistake prevented access to some FSM pages.
Per report from Torsten Foertsch.

In passing, make the bad-block-number error into an ereport not elog
(since it's certainly not an internal error); and fix sloppily
maintained comment for RelationGetNumberOfBlocksInFork.

This has been wrong since we invented relation forks, so back-patch
to all supported branches.
2014-07-22 11:46:29 -04:00
Noah Misch d7cdf6ee36 Diagnose incompatible OpenLDAP versions during build and test.
With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, PostgreSQL
backends can crash at exit.  Raise a warning during "configure" based on
the compile-time OpenLDAP version number, and test the crash scenario in
the dblink test suite.  Back-patch to 9.0 (all supported versions).
2014-07-22 11:01:03 -04:00
Peter Eisentraut e0da6ec618 pg_upgrade: Fix spacing in help output 2014-07-15 15:08:16 -04:00
Magnus Hagander a16bac36ec Remove dependency on wsock32.lib in favor of ws2_32
ws2_32 is the new version of the library that should be used, as
it contains the require functionality from wsock32 as well as some
more (which is why some binaries were already using ws2_32).

Michael Paquier, reviewed by MauMau
2014-07-15 14:18:39 +02:00
Noah Misch 0ffc201a51 Add file version information to most installed Windows binaries.
Prominent binaries already had this metadata.  A handful of minor
binaries, such as pg_regress.exe, still lack it; efforts to eliminate
such exceptions are welcome.

Michael Paquier, reviewed by MauMau.
2014-07-14 14:07:52 -04:00
Noah Misch c0e8fb6148 contrib/test_decoding/Makefile sets MODULES, so omit OBJS.
Michael Paquier
2014-07-14 14:07:45 -04:00
Fujii Masao ab77485945 Remove incorrect comment from postgres_fdw.c.
Etsuro Fujita
2014-07-14 19:28:26 +09:00
Andres Freund 626bfad6cc Fix decoding of consecutive MULTI_INSERTs emitted by one heap_multi_insert().
Commit 1b86c81d2d fixed the decoding of toasted columns for the rows
contained in one xl_heap_multi_insert record. But that's not actually
enough, because heap_multi_insert() will actually first toast all
passed in rows and then emit several *_multi_insert records; one for
each page it fills with tuples.

Add a XLOG_HEAP_LAST_MULTI_INSERT flag which is set in
xl_heap_multi_insert->flag denoting that this multi_insert record is
the last emitted by one heap_multi_insert() call. Then use that flag
in decode.c to only set clear_toast_afterwards in the right situation.

Expand the number of rows inserted via COPY in the corresponding
regression test to make sure that more than one heap page is filled
with tuples by one heap_multi_insert() call.

Backpatch to 9.4 like the previous commit.
2014-07-12 14:28:19 +02:00
Tom Lane 59efda3e50 Implement IMPORT FOREIGN SCHEMA.
This command provides an automated way to create foreign table definitions
that match remote tables, thereby reducing tedium and chances for error.
In this patch, we provide the necessary core-server infrastructure and
implement the feature fully in the postgres_fdw foreign-data wrapper.
Other wrappers will throw a "feature not supported" error until/unless
they are updated.

Ronan Dunklau and Michael Paquier, additional work by me
2014-07-10 15:01:43 -04:00
Bruce Momjian 6a605cd6bd Adjust blank lines around PG_MODULE_MAGIC defines, for consistency
Report by Robert Haas
2014-07-10 14:02:08 -04:00
Bruce Momjian 3088cc3704 pg_upgrade: allow upgrades for new-only TOAST tables
Previously, when calculations on the need for toast tables changed,
pg_upgrade could not handle cases where the new cluster needed a TOAST
table and the old cluster did not.  (It already handled the opposite
case.)  This fixes the "OID mismatch" error typically generated in this
case.

Backpatch through 9.2
2014-07-07 13:24:08 -04:00
Andres Freund 1b86c81d2d Fix decoding of MULTI_INSERTs when rows other than the last are toasted.
When decoding the results of a HEAP2_MULTI_INSERT (currently only
generated by COPY FROM) toast columns for all but the last tuple
weren't replaced by their actual contents before being handed to the
output plugin. The reassembled toast datums where disregarded after
every REORDER_BUFFER_CHANGE_(INSERT|UPDATE|DELETE) which is correct
for plain inserts, updates, deletes, but not multi inserts - there we
generate several REORDER_BUFFER_CHANGE_INSERTs for a single
xl_heap_multi_insert record.

To solve the problem add a clear_toast_afterwards boolean to
ReorderBufferChange's union member that's used by modifications. All
row changes but multi_inserts always set that to true, but
multi_insert sets it only for the last change generated.

Add a regression test covering decoding of multi_inserts - there was
none at all before.

Backpatch to 9.4 where logical decoding was introduced.

Bug found by Petr Jelinek.
2014-07-06 15:58:01 +02:00
Noah Misch 333b7db8b3 Consistently pass an "unsigned char" to ctype.h functions.
The isxdigit() calls relied on undefined behavior.  The isascii() call
was well-defined, but our prevailing style is to include the cast.
Back-patch to 9.4, where the isxdigit() calls were introduced.
2014-07-06 00:29:51 -04:00
Bruce Momjian a61daa14d5 pg_upgrade: preserve database and relation minmxid values
Also set these values for pre-9.3 old clusters that don't have values to
preserve.

Analysis by Alvaro

Backpatch through 9.3
2014-07-02 15:29:38 -04:00
Bruce Momjian 89cf2d5203 pg_upgrade: no need to remove "members" files for pre-9.3 upgrades
Per analysis by Alvaro

Backpatch through 9.3
2014-07-02 13:11:05 -04:00
Tom Lane fbb1d7d73f Allow CREATE/ALTER DATABASE to manipulate datistemplate and datallowconn.
Historically these database properties could be manipulated only by
manually updating pg_database, which is error-prone and only possible for
superusers.  But there seems no good reason not to allow database owners to
set them for their databases, so invent CREATE/ALTER DATABASE options to do
that.  Adjust a couple of places that were doing it the hard way to use the
commands instead.

Vik Fearing, reviewed by Pavel Stehule
2014-07-01 20:10:38 -04:00
Tom Lane f23425fa95 Improve handling of OOM score adjustment in sample Linux start script.
Per a suggestion from Christoph Berg.
2014-07-01 17:23:16 -04:00
Tom Lane 5a421a47eb Fix inadequately-sized output buffer in contrib/unaccent.
The output buffer size in unaccent_lexize() was calculated as input string
length times pg_database_encoding_max_length(), which effectively assumes
that replacement strings aren't more than one character.  While that was
all that we previously documented it to support, the code actually has
always allowed replacement strings of arbitrary length; so if you tried
to make use of longer strings, you were at risk of buffer overrun.  To fix,
use an expansible StringInfo buffer instead of trying to determine the
maximum space needed a-priori.

This would be a security issue if unaccent rules files could be installed
by unprivileged users; but fortunately they can't, so in the back branches
the problem can be labeled as improper configuration by a superuser.
Nonetheless, a memory stomp isn't a nice way of reacting to improper
configuration, so let's back-patch the fix.
2014-07-01 11:23:21 -04:00
Tom Lane 03a25cec8d Issue a WARNING about invalid rule file format in contrib/unaccent.
We were already issuing a WARNING, albeit only elog not ereport, for
duplicate source strings; so warning rather than just being stoically
silent seems like the best thing to do here.  Arguably both of these
complaints should be upgraded to ERRORs, but that might be more
behavioral change than people want.

Note: the faulty line is already printed via an errcontext hook,
so there's no need for more information than these messages provide.
2014-06-30 22:03:37 -04:00
Tom Lane 1b2488731c Allow multi-character source strings in contrib/unaccent.
This could be useful in languages where diacritic signs are represented as
separate characters; more generally it supports using unaccent dictionaries
for substring substitutions beyond narrowly conceived "diacritic removal".
In any case, since the rule-file parser doesn't complain about
multi-character source strings, it behooves us to do something unsurprising
with them.
2014-06-30 21:46:29 -04:00
Tom Lane 97c40ce614 Allow empty replacement strings in contrib/unaccent.
This is useful in languages where diacritic signs are represented as
separate characters; it's also one step towards letting unaccent be used
for arbitrary substring substitutions.

In passing, improve the user documentation for unaccent, which was sadly
vague about some important details.

Mohammad Alhashash, reviewed by Abhijit Menon-Sen
2014-06-30 20:51:30 -04:00
Bruce Momjian 55863274d9 pg_upgrade: update C comments about pg_dumpall
There were some C comments that hadn't been updated from the switch of
using only pg_dumpall to using pg_dump and pg_dumpall, so update them.
Also, don't bother using --schema-only for pg_dumpall --globals-only.

Backpatch through 9.4
2014-06-30 19:56:23 -04:00
Noah Misch 9d0826c595 Don't prematurely free the BufferAccessStrategy in pgstat_heap().
This function continued to use it after heap_endscan() freed it.  In
passing, don't explicit create a strategy here.  Instead, use the one
created by heap_beginscan_strat(), if any.  Back-patch to 9.2, where use
of a BufferAccessStrategy here was introduced.
2014-06-30 16:59:19 -04:00
Bruce Momjian 0f7482733a pg_upgrade: remove pg_multixact files left by initdb
This fixes a bug that caused vacuum to fail when the '0000' files left
by initdb were accessed as part of vacuum's cleanup of old pg_multixact
files.

Backpatch through 9.3
2014-06-24 16:11:06 -04:00
Joe Conway 1dde5782e3 Clean up data conversion short-lived memory context.
dblink uses a short-lived data conversion memory context. However it
was not deleted when no longer needed, leading to a noticeable memory
leak under some circumstances. Plug the hole, along with minor
refactoring. Backpatch to 9.2 where the leak was introduced.

Report and initial patch by MauMau. Reviewed/modified slightly by
Tom Lane and me.
2014-06-20 12:24:59 -07:00
Noah Misch 686f362bee Fix contrib/pg_upgrade/test.sh for $PWD containing spaces.
Most of the necessary quoting was in place; this catches the exceptions.
2014-06-19 21:47:08 -04:00
Noah Misch c82725edfa Let installcheck-world pass against a server requiring a password.
Give passwords to each user created in support of an ECPG connection
test case.  Use SET SESSION AUTHORIZATION, not a fresh connection, to
reduce privileges during a dblink test case.

To test against such a server, both the "make installcheck-world"
environment and the postmaster environment must provide the default
user's password; $PGPASSFILE is the principal way to do so.  (The
postmaster environment needs it for dblink and postgres_fdw tests.)
2014-06-19 21:41:26 -04:00
Tom Lane df8b7bc9ff Improve our mechanism for controlling the Linux out-of-memory killer.
Arrange for postmaster child processes to respond to two environment
variables, PG_OOM_ADJUST_FILE and PG_OOM_ADJUST_VALUE, to determine whether
they reset their OOM score adjustments and if so to what.  This is superior
to the previous design involving #ifdef's in several ways.  The behavior is
now available in a default build, and both ends of the adjustment --- the
original adjustment of the postmaster's level and the subsequent
readjustment by child processes --- can now be controlled in one place,
namely the postmaster launch script.  So it's no longer necessary for the
launch script to act on faith that the server was compiled with the
appropriate options.  In addition, if someone wants to use an OOM score
other than zero for the child processes, that doesn't take a recompile
anymore; and we no longer have to cater separately to the two different
historical kernel APIs for this adjustment.

Gurjeet Singh, somewhat revised by me
2014-06-18 20:12:51 -04:00
Tom Lane 8f889b1083 Implement UPDATE tab SET (col1,col2,...) = (SELECT ...), ...
This SQL-standard feature allows a sub-SELECT yielding multiple columns
(but only one row) to be used to compute the new values of several columns
to be updated.  While the same results can be had with an independent
sub-SELECT per column, such a workaround can require a great deal of
duplicated computation.

The standard actually says that the source for a multi-column assignment
could be any row-valued expression.  The implementation used here is
tightly tied to our existing sub-SELECT support and can't handle other
cases; the Bison grammar would have some issues with them too.  However,
I don't feel too bad about this since other cases can be converted into
sub-SELECTs.  For instance, "SET (a,b,c) = row_valued_function(x)" could
be written "SET (a,b,c) = (SELECT * FROM row_valued_function(x))".
2014-06-18 13:22:34 -04:00
Tom Lane 2146f13408 Avoid recursion when processing simple lists of AND'ed or OR'ed clauses.
Since most of the system thinks AND and OR are N-argument expressions
anyway, let's have the grammar generate a representation of that form when
dealing with input like "x AND y AND z AND ...", rather than generating
a deeply-nested binary tree that just has to be flattened later by the
planner.  This avoids stack overflow in parse analysis when dealing with
queries having more than a few thousand such clauses; and in any case it
removes some rather unsightly inconsistencies, since some parts of parse
analysis were generating N-argument ANDs/ORs already.

It's still possible to get a stack overflow with weirdly parenthesized
input, such as "x AND (y AND (z AND ( ... )))", but such cases are not
mainstream usage.  The maximum depth of parenthesization is already
limited by Bison's stack in such cases, anyway, so that the limit is
probably fairly platform-independent.

Patch originally by Gurjeet Singh, heavily revised by me
2014-06-16 15:55:30 -04:00
Noah Misch be76a6d39e Secure Unix-domain sockets of "make check" temporary clusters.
Any OS user able to access the socket can connect as the bootstrap
superuser and proceed to execute arbitrary code as the OS user running
the test.  Protect against that by placing the socket in a temporary,
mode-0700 subdirectory of /tmp.  The pg_regress-based test suites and
the pg_upgrade test suite were vulnerable; the $(prove_check)-based test
suites were already secure.  Back-patch to 8.4 (all supported versions).
The hazard remains wherever the temporary cluster accepts TCP
connections, notably on Windows.

As a convenient side effect, this lets testing proceed smoothly in
builds that override DEFAULT_PGSOCKET_DIR.  Popular non-default values
like /var/run/postgresql are often unwritable to the build user.

Security: CVE-2014-0067
2014-06-14 09:41:13 -04:00
Heikki Linnakangas 0ef0b6784c Change the signature of rm_desc so that it's passed a XLogRecord.
Just feels more natural, and is more consistent with rm_redo.
2014-06-14 10:46:48 +03:00
Fujii Masao 654e8e4447 Save pg_stat_statements statistics file into $PGDATA/pg_stat directory at shutdown.
187492b6c2 changed pgstat.c so that
the stats files were saved into $PGDATA/pg_stat directory when the server
was shutdowned. But it accidentally forgot to change the location of
pg_stat_statements permanent stats file. This commit fixes pg_stat_statements
so that its stats file is also saved into $PGDATA/pg_stat at shutdown.

Since this fix changes the file layout, we don't back-patch it to 9.3
where this oversight was introduced.
2014-06-04 12:09:45 +09:00
Tom Lane c941aed96b When using the OSSP UUID library, cache its uuid_t state object.
The original coding in contrib/uuid-ossp created and destroyed a uuid_t
object (or, in some cases, even two of them) each time it was called.
This is not the intended usage: you're supposed to keep the uuid_t object
around so that the library can cache its state across uses.  (Other UUID
libraries seem to keep equivalent state behind-the-scenes in static
variables, but OSSP chose differently.)  Aside from being quite inefficient,
creating a new uuid_t loses knowledge of the previously generated UUID,
which in theory could result in duplicate V1-style UUIDs being created
on sufficiently fast machines.

On at least some platforms, creating a new uuid_t also draws some entropy
from /dev/urandom, leaving less for the rest of the system.  This seems
sufficiently unpleasant to justify back-patching this change.
2014-05-29 13:51:02 -04:00
Tom Lane 25dd07e0f6 Fix uuid-ossp regression tests based on buildfarm feedback.
The previous version of these tests expected uuid_generate_v1() to always
emit MAC addresses with the local-admin and multicast address bits zero.
However, several of the buildfarm critters are reporting values with the
local-admin bit set.  (Perhaps they're running inside VMs or jails.)
And a couple are reporting values with the multicast bit set, probably
meaning that the UUID library couldn't read the system MAC address.

Also, it emerges that if OSSP UUID can't read the system MAC address, it
falls back to V1MC behavior wherein the whole node field gets randomized
each time, breaking the test that expected the node field to remain stable
in V1 output.  (It looks like e2fs doesn't behave that way, though.)

It's not entirely clear why we can't get a system MAC address, since the
buildfarm scripts would not work without internet access.  Nonetheless,
the regression tests had better cope with the case, so adjust the tests
to expect these behaviors.
2014-05-28 23:15:51 -04:00
Tom Lane 71ed8b3ca7 Revert "Fix bogus %name-prefix option syntax in all our Bison files."
This reverts commit 45b7abe59e.

It turns out that the %name-prefix syntax without "=" does not work
at all in pre-2.4 Bison.  We are not prepared to make such a large
jump in minimum required Bison version just to suppress a warning
message in a version hardly any developers are using yet.
When 3.0 gets more popular, we'll figure out a way to deal with this.
In the meantime, BISONFLAGS=-Wno-deprecated is recommendable for
anyone using 3.0 who doesn't want to see the warning.
2014-05-28 19:21:01 -04:00
Tom Lane 45b7abe59e Fix bogus %name-prefix option syntax in all our Bison files.
%name-prefix doesn't use an "=" sign according to the Bison docs, but it
silently accepted one anyway, until Bison 3.0.  This was originally a
typo of mine in commit 012abebab1, and we
seem to have slavishly copied the error into all the other grammar files.

Per report from Vik Fearing; analysis by Peter Eisentraut.

Back-patch to all active branches, since somebody might try to build
a back branch with up-to-date tools.
2014-05-28 15:41:53 -04:00
Tom Lane c0f27628d2 Improve regression tests for uuid-ossp.
On reflection, the timestamp-advances test might fail if we're unlucky
enough for the time_mid field to change between two calls, since uuid_cmp
is just bytewise comparison and the field ordering has more significant
fields later.  Build some field extraction functions so we can do a more
honest test of that.  Also check that the version and reserved fields
contain what they should.
2014-05-28 14:21:17 -04:00
Tom Lane 2103218dd4 Fix stack clobber in new uuid-ossp code.
The V5 (SHA1 hashing) code wrote 20 bytes into a 16-byte local variable.
This had accidentally failed to fail in my testing and Matteo's, but
buildfarm results exposed the problem.
2014-05-28 11:50:41 -04:00
Tom Lane b8cc8f9473 Support BSD and e2fsprogs UUID libraries alongside OSSP UUID library.
Allow the contrib/uuid-ossp extension to be built atop any one of these
three popular UUID libraries.  (The extension's name is now arguably a
misnomer, but we'll keep it the same so as not to cause unnecessary
compatibility issues for users.)

We would not normally consider a change like this post-beta1, but the issue
has been forced by our upgrade to autoconf 2.69, whose more rigorous header
checks are causing OSSP's header files to be rejected on some platforms.
It's been foreseen for some time that we'd have to move away from depending
on OSSP UUID due to lack of upstream maintenance, so this is a down payment
on that problem.

While at it, add some simple regression tests, in hopes of catching any
major incompatibilities between the three implementations.

Matteo Beccati, with some further hacking by me
2014-05-27 19:42:08 -04:00