Commit Graph

2706 Commits

Author SHA1 Message Date
Bruce Momjian c3c75fcd7a pg_upgrade: adjust C comments 2014-09-11 18:44:00 -04:00
Heikki Linnakangas 01a2bfd172 Fix Windows build.
I renamed a variable, but missed an #ifdef WIN32 block.
2014-09-11 15:15:40 +03:00
Heikki Linnakangas 54a2d5b37b Simplify calculation of Poisson distributed delays in pgbench --rate mode.
The previous coding first generated a uniform random value between 0.0 and
1.0, then converted that to an integer between 1 and 10000, and divided that
again by 10000. Those conversions are unnecessary; we can use the double
value that pg_erand48() returns directly. While we're at it, put the logic
into a helper function, getPoissonRand().

The largest delay generated by the old coding was about 9.2 times the
average, because of the way the uniformly distributed value used for the
calculation was truncated to 1/10000 granularity. The new coding doesn't
have such clamping. With my laptop's DBL_MIN value, the maximum delay with
the new coding is about 700x the average. That seems acceptable - any
reasonable pgbench session should last long enough to average that out.

Backpatch to 9.4.
2014-09-11 13:00:48 +03:00
Heikki Linnakangas 02e3bcc661 Change the way latency is calculated with pgbench --rate option.
The reported latency values now include the "schedule lag" time, that is,
the time between the transaction's scheduled start time and the time it
actually started. This relates better to a model where requests arrive at a
certain rate, and we are interested in the response time to the end user or
application, rather than the response time of the database itself.

Also, when --rate is used, include the schedule lag time in the log output.

The --rate option is new in 9.4, so backpatch to 9.4. It seems better to
make this change in 9.4, while we're still in the beta period, than ship a
9.4 version that calculates the values differently than 9.5.
2014-09-11 12:57:32 +03:00
Bruce Momjian acc8e41681 pg_upgrade: compare control version, not catalog version
Also modify test for the possibility the large object value might not
exist in the old cluster.

Fix for commit e1598a15f4
2014-09-10 20:22:10 -04:00
Bruce Momjian e1598a15f4 pg_upgrade: check for large object size compatibility 2014-09-10 19:23:36 -04:00
Peter Eisentraut 220bb39dee doc: Reflect renaming of Mac OS X to OS X
bug #10528
2014-09-09 13:56:29 -04:00
Bruce Momjian a74a4aa23b pg_upgrade: preserve the timestamp epoch
This is useful for replication tools like Slony and Skytools.

Report by Sergey Konoplev
2014-09-05 19:19:41 -04:00
Peter Eisentraut 303f4d1012 Assorted message fixes and improvements 2014-09-05 01:25:27 -04:00
Andres Freund d6fa44fce7 Add skip-empty-xacts option to test_decoding for use in the regression tests.
The regression tests for contrib/test_decoding regularly failed on
postgres instances that were very slow. Either because the hardware
itself was slow or because very expensive debugging options like
CLOBBER_CACHE_ALWAYS were used.

The reason they failed was just that some additional transactions were
decoded. Analyze and vacuum, triggered by autovac.

To fix just add a option to test_decoding to only display transactions
in which a change was actually displayed. That's not pretty because it
removes information from the tests; but better than constantly failing
tests in very likely harmless ways.

Backpatch to 9.4 where logical decoding was introduced.

Discussion: 20140629142511.GA26930@awork2.anarazel.de
2014-09-01 15:59:44 +02:00
Andres Freund 4b4b680c3d Make backend local tracking of buffer pins memory efficient.
Since the dawn of time (aka Postgres95) multiple pins of the same
buffer by one backend have been optimized not to modify the shared
refcount more than once. This optimization has always used a NBuffer
sized array in each backend keeping track of a backend's pins.

That array (PrivateRefCount) was one of the biggest per-backend memory
allocations, depending on the shared_buffers setting. Besides the
waste of memory it also has proven to be a performance bottleneck when
assertions are enabled as we make sure that there's no remaining pins
left at the end of transactions. Also, on servers with lots of memory
and a correspondingly high shared_buffers setting the amount of random
memory accesses can also lead to poor cpu cache efficiency.

Because of these reasons a backend's buffers pins are now kept track
of in a small statically sized array that overflows into a hash table
when necessary. Benchmarks have shown neutral to positive performance
results with considerably lower memory usage.

Patch by me, review by Robert Haas.

Discussion: 20140321182231.GA17111@alap3.anarazel.de
2014-08-30 14:03:21 +02:00
Tom Lane 7f7eec89b6 Fix citext upgrade script for disallowance of oidvector element assignment.
In commit 45e02e3232, we intentionally
disallowed updates on individual elements of oidvector columns.  While that
still seems like a sane idea in the abstract, we (I) forgot that citext's
"upgrade from unpackaged" script did in fact perform exactly such updates,
in order to fix the problem that citext indexes should have a collation
but would not in databases dumped or upgraded from pre-9.1 installations.

Even if we wanted to add casts to allow such updates, there's no practical
way to do so in the back branches, so the only real alternative is to make
citext's kluge even klugier.  In this patch, I cast the oidvector to text,
fix its contents with regexp_replace, and cast back to oidvector.  (Ugh!)

Since the aforementioned commit went into all active branches, we have to
fix this in all branches that contain the now-broken update script.

Per report from Eric Malm.
2014-08-28 18:21:05 -04:00
Peter Eisentraut 2d759341d9 Fix whitespace 2014-08-26 17:26:45 -04:00
Andres Freund 57ca1d4f01 Specify the port in dblink and postgres_fdw tests.
That allows to run those tests against a postmaster listening on a
nonstandard port without requiring to export PGPORT in postmaster's
environment.

This still doesn't support connecting to a nondefault host without
configuring it in postmaster's environment. That's harder and less
frequently used though. So this is a useful step.
2014-08-26 12:28:08 +02:00
Andres Freund ddc2504dbc Don't hardcode contrib_regression dbname in postgres_fdw and dblink tests.
That allows parallel installcheck to succeed inside contrib/. The
output is not particularly pretty unless make's -O option to
synchronize the output is used.

There's other tests, outside contrib, that use a hardcoded,
non-unique, database name. Those prohibit paralell installcheck to be
used across more directories; but that's something for a separate
patch.
2014-08-26 12:27:26 +02:00
Bruce Momjian a7ae1dcf49 pg_upgrade: prevent automatic oid assignment
Prevent automatic oid assignment when in binary upgrade mode.  Also
throw an error when contrib/pg_upgrade_support functions are called when
not in binary upgrade mode.

This prevent automatically-assigned oids from conflicting with later
pre-assigned oids coming from the old cluster.  It also makes sure oids
are preserved in call important cases.
2014-08-25 22:19:05 -04:00
Bruce Momjian 5d16332e96 pg_upgrade: use CTE query rather than temp table
Now that 8.3 is not supported, we can use a CTE and not temp tables.
This allows for auto-oid assignment protection in a future patch.
2014-08-25 21:22:08 -04:00
Bruce Momjian 6cb74a67e2 revert "Throw error for ALTER TABLE RESET of an invalid option"
Reverts commits 73d78e11a0 and
b0488e5c4f.  Also reverts pg_upgrade
changes.
2014-08-25 20:07:37 -04:00
Bruce Momjian 2209b3923a pg_upgrade: remove support for 8.3 old clusters
This trims down the code, and is in preparation for hardening pg_upgrade
against auto-oid assignment.
2014-08-25 20:05:07 -04:00
Bruce Momjian b0488e5c4f pg_upgrade: add SET log_min_error_statement = warning
Add to commit 73d78e11a0
2014-08-25 17:24:33 -04:00
Bruce Momjian 73d78e11a0 Throw error for ALTER TABLE RESET of an invalid option
Also adjust pg_upgrade to not use this method for optional TOAST table
creation.

Patch by Fabrízio de Royes Mello
2014-08-25 17:06:40 -04:00
Bruce Momjian ebe30ad59b pg_ctl, pg_upgrade: allow multiple -o/-O options, append them
Report by Pavel Raiskup
2014-08-25 16:30:26 -04:00
Andres Freund d153b80161 Fix typos in some error messages thrown by extension scripts when fed to psql.
Some of the many error messages introduced in 458857cc missed 'FROM
unpackaged'. Also e016b724 and 45ffeb7e forgot to quote extension
version numbers.

Backpatch to 9.1, just like 458857cc which introduced the messages. Do
so because the error messages thrown when the wrong command is copy &
pasted aren't easy to understand.
2014-08-25 18:30:37 +02:00
Heikki Linnakangas 3cd934f635 Don't track DEALLOCATE in pg_stat_statements.
We also don't track PREPARE, nor do we track planning time in general, so
let's ignore DEALLOCATE as well for consistency.

Backpatch to 9.4, but not further than that. Although it seems unlikely that
anyone is relying on the current behavior, this is a behavioral change.

Fabien Coelho
2014-08-25 19:13:24 +03:00
Andres Freund ebf20f65d9 Fix newly introduced misspelling of existence in pg_buffercache.
Peter Geoghegan
2014-08-22 17:49:47 +02:00
Andres Freund f57791985a Add pinning_backends column to the pg_buffercache extension.
The new column shows how many backends have a buffer pinned. That can
be useful during development or to diagnose production issues
e.g. caused by vacuum waiting for cleanup locks.

To handle upgrades transparently - the extension might be used in
views - deal with callers expecting the old number of columns.

Reviewed by Fujii Masao and Rajeev rastogi.
2014-08-22 00:28:37 +02:00
Bruce Momjian 7567d94910 pg_upgrade: adjust logging to use QUERY_ALLOC lengths
Allows the logging to print the entire text of failed queries, rather
than a truncated version.
2014-08-20 13:04:45 -04:00
Noah Misch fb2aece8ae Replace a few strncmp() calls with strlcpy().
strncmp() is a specialized API unsuited for routine copying into
fixed-size buffers.  On a system where the length of a single filename
can exceed MAXPGPATH, the pg_archivecleanup change prevents a simple
crash in the subsequent strlen().  Few filesystems support names that
long, and calling pg_archivecleanup with untrusted input is still not a
credible use case.  Therefore, no back-patch.

David Rowley
2014-08-18 22:59:31 -04:00
Peter Eisentraut 9ee9ca545b Fix whitespace 2014-08-15 14:52:06 -04:00
Bruce Momjian ea9b8abd6f pg_upgrade: fix define adjusted during testing
from commit e4c3c99ac3
2014-08-15 12:28:18 -04:00
Bruce Momjian e4c3c99ac3 pg_upgrade: error if run from top of new PGDATA on Windows
Print a clear error message in this case, rather than wait for initdb
--sync-only to generate a "Permission denied" error.
2014-08-15 12:04:03 -04:00
Peter Eisentraut 1d678bf7bc Add some noreturn attributes based on compiler recommendations 2014-08-13 22:40:48 -04:00
Tatsuo Ishii 51222a1766 Enhance pgbench's option checking.
Now benchmarking options such as -c cannot be used if initializing
option (-i) is specified.  Also initializing options such as -F cannot
be used if initializing option is not specified.

Tatsuo Ishii and Fabien COELHO.
2014-08-12 17:27:08 +09:00
Bruce Momjian 4c6780fd17 pg_upgrade: prevent oid conflicts with new-cluster TOAST tables
Previously, TOAST tables only required in the new cluster could cause
oid conflicts if they were auto-numbered and a later conflicting oid had
to be assigned.

Backpatch through 9.3
2014-08-07 14:56:13 -04:00
Bruce Momjian ee646df59a pg_upgrade: assume user is install user
The user specified to the upgrade was effectively the install user, but
that was not clearly stated in the comments, documentation, or error
messages.
2014-08-04 11:56:16 -04:00
Bruce Momjian b4bd6f6b05 pg_upgrade: remove reference to autovacuum_multixact_freeze_max_age
autovacuum_multixact_freeze_max_age was added as a pg_ctl start
parameter in 9.3.X to prevent autovacuum from running.  However, only
some 9.3.X releases have autovacuum_multixact_freeze_max_age as it was
added in a minor PG 9.3 release.  It also isn't needed because -b turns
off autovacuum in 9.1+.

Without this fix, trying to upgrade from an early 9.3 release to 9.4
would fail.

Report by EDB

Backpatch through 9.3
2014-08-04 11:46:05 -04:00
Heikki Linnakangas cd74a151b5 Windows doesn't have M_PI; define it ourselves when needed.
This should fix the Windows build, broken by commit ed802e7d.
2014-08-04 18:06:34 +03:00
Robert Haas ed802e7dc3 pgbench: Allow \setrandom to generate Gaussian/exponential distributions.
Mitsumasa KONDO and Fabien COELHO, with further wordsmithing by me.
2014-07-30 13:22:08 -04:00
Bruce Momjian d826d8ecce pg_upgrade: improve C comment wording 2014-07-29 22:09:03 -04:00
Alvaro Herrera 0531549801 Avoid uselessly looking up old LOCK_ONLY multixacts
Commit 0ac5ad5134 removed an optimization in multixact.c that skipped
fetching members of MultiXactId that were older than our
OldestVisibleMXactId value.  The reason this was removed is that it is
possible for multixacts that contain updates to be older than that
value.  However, if the caller is certain that the multi does not
contain an update (because the infomask bits say so), it can pass this
info down to GetMultiXactIdMembers, enabling it to use the old
optimization.

Pointed out by Andres Freund in 20131121200517.GM7240@alap2.anarazel.de
2014-07-29 15:41:06 -04:00
Tom Lane 27cef0a561 Check block number against the correct fork in get_raw_page().
get_raw_page tried to validate the supplied block number against
RelationGetNumberOfBlocks(), which of course is only right when
accessing the main fork.  In most cases, the main fork is longer
than the others, so that the check was too weak (allowing a
lower-level error to be reported, but no real harm to be done).
However, very small tables could have an FSM larger than their heap,
in which case the mistake prevented access to some FSM pages.
Per report from Torsten Foertsch.

In passing, make the bad-block-number error into an ereport not elog
(since it's certainly not an internal error); and fix sloppily
maintained comment for RelationGetNumberOfBlocksInFork.

This has been wrong since we invented relation forks, so back-patch
to all supported branches.
2014-07-22 11:46:29 -04:00
Noah Misch d7cdf6ee36 Diagnose incompatible OpenLDAP versions during build and test.
With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, PostgreSQL
backends can crash at exit.  Raise a warning during "configure" based on
the compile-time OpenLDAP version number, and test the crash scenario in
the dblink test suite.  Back-patch to 9.0 (all supported versions).
2014-07-22 11:01:03 -04:00
Peter Eisentraut e0da6ec618 pg_upgrade: Fix spacing in help output 2014-07-15 15:08:16 -04:00
Magnus Hagander a16bac36ec Remove dependency on wsock32.lib in favor of ws2_32
ws2_32 is the new version of the library that should be used, as
it contains the require functionality from wsock32 as well as some
more (which is why some binaries were already using ws2_32).

Michael Paquier, reviewed by MauMau
2014-07-15 14:18:39 +02:00
Noah Misch 0ffc201a51 Add file version information to most installed Windows binaries.
Prominent binaries already had this metadata.  A handful of minor
binaries, such as pg_regress.exe, still lack it; efforts to eliminate
such exceptions are welcome.

Michael Paquier, reviewed by MauMau.
2014-07-14 14:07:52 -04:00
Noah Misch c0e8fb6148 contrib/test_decoding/Makefile sets MODULES, so omit OBJS.
Michael Paquier
2014-07-14 14:07:45 -04:00
Fujii Masao ab77485945 Remove incorrect comment from postgres_fdw.c.
Etsuro Fujita
2014-07-14 19:28:26 +09:00
Andres Freund 626bfad6cc Fix decoding of consecutive MULTI_INSERTs emitted by one heap_multi_insert().
Commit 1b86c81d2d fixed the decoding of toasted columns for the rows
contained in one xl_heap_multi_insert record. But that's not actually
enough, because heap_multi_insert() will actually first toast all
passed in rows and then emit several *_multi_insert records; one for
each page it fills with tuples.

Add a XLOG_HEAP_LAST_MULTI_INSERT flag which is set in
xl_heap_multi_insert->flag denoting that this multi_insert record is
the last emitted by one heap_multi_insert() call. Then use that flag
in decode.c to only set clear_toast_afterwards in the right situation.

Expand the number of rows inserted via COPY in the corresponding
regression test to make sure that more than one heap page is filled
with tuples by one heap_multi_insert() call.

Backpatch to 9.4 like the previous commit.
2014-07-12 14:28:19 +02:00
Tom Lane 59efda3e50 Implement IMPORT FOREIGN SCHEMA.
This command provides an automated way to create foreign table definitions
that match remote tables, thereby reducing tedium and chances for error.
In this patch, we provide the necessary core-server infrastructure and
implement the feature fully in the postgres_fdw foreign-data wrapper.
Other wrappers will throw a "feature not supported" error until/unless
they are updated.

Ronan Dunklau and Michael Paquier, additional work by me
2014-07-10 15:01:43 -04:00
Bruce Momjian 6a605cd6bd Adjust blank lines around PG_MODULE_MAGIC defines, for consistency
Report by Robert Haas
2014-07-10 14:02:08 -04:00
Bruce Momjian 3088cc3704 pg_upgrade: allow upgrades for new-only TOAST tables
Previously, when calculations on the need for toast tables changed,
pg_upgrade could not handle cases where the new cluster needed a TOAST
table and the old cluster did not.  (It already handled the opposite
case.)  This fixes the "OID mismatch" error typically generated in this
case.

Backpatch through 9.2
2014-07-07 13:24:08 -04:00
Andres Freund 1b86c81d2d Fix decoding of MULTI_INSERTs when rows other than the last are toasted.
When decoding the results of a HEAP2_MULTI_INSERT (currently only
generated by COPY FROM) toast columns for all but the last tuple
weren't replaced by their actual contents before being handed to the
output plugin. The reassembled toast datums where disregarded after
every REORDER_BUFFER_CHANGE_(INSERT|UPDATE|DELETE) which is correct
for plain inserts, updates, deletes, but not multi inserts - there we
generate several REORDER_BUFFER_CHANGE_INSERTs for a single
xl_heap_multi_insert record.

To solve the problem add a clear_toast_afterwards boolean to
ReorderBufferChange's union member that's used by modifications. All
row changes but multi_inserts always set that to true, but
multi_insert sets it only for the last change generated.

Add a regression test covering decoding of multi_inserts - there was
none at all before.

Backpatch to 9.4 where logical decoding was introduced.

Bug found by Petr Jelinek.
2014-07-06 15:58:01 +02:00
Noah Misch 333b7db8b3 Consistently pass an "unsigned char" to ctype.h functions.
The isxdigit() calls relied on undefined behavior.  The isascii() call
was well-defined, but our prevailing style is to include the cast.
Back-patch to 9.4, where the isxdigit() calls were introduced.
2014-07-06 00:29:51 -04:00
Bruce Momjian a61daa14d5 pg_upgrade: preserve database and relation minmxid values
Also set these values for pre-9.3 old clusters that don't have values to
preserve.

Analysis by Alvaro

Backpatch through 9.3
2014-07-02 15:29:38 -04:00
Bruce Momjian 89cf2d5203 pg_upgrade: no need to remove "members" files for pre-9.3 upgrades
Per analysis by Alvaro

Backpatch through 9.3
2014-07-02 13:11:05 -04:00
Tom Lane fbb1d7d73f Allow CREATE/ALTER DATABASE to manipulate datistemplate and datallowconn.
Historically these database properties could be manipulated only by
manually updating pg_database, which is error-prone and only possible for
superusers.  But there seems no good reason not to allow database owners to
set them for their databases, so invent CREATE/ALTER DATABASE options to do
that.  Adjust a couple of places that were doing it the hard way to use the
commands instead.

Vik Fearing, reviewed by Pavel Stehule
2014-07-01 20:10:38 -04:00
Tom Lane f23425fa95 Improve handling of OOM score adjustment in sample Linux start script.
Per a suggestion from Christoph Berg.
2014-07-01 17:23:16 -04:00
Tom Lane 5a421a47eb Fix inadequately-sized output buffer in contrib/unaccent.
The output buffer size in unaccent_lexize() was calculated as input string
length times pg_database_encoding_max_length(), which effectively assumes
that replacement strings aren't more than one character.  While that was
all that we previously documented it to support, the code actually has
always allowed replacement strings of arbitrary length; so if you tried
to make use of longer strings, you were at risk of buffer overrun.  To fix,
use an expansible StringInfo buffer instead of trying to determine the
maximum space needed a-priori.

This would be a security issue if unaccent rules files could be installed
by unprivileged users; but fortunately they can't, so in the back branches
the problem can be labeled as improper configuration by a superuser.
Nonetheless, a memory stomp isn't a nice way of reacting to improper
configuration, so let's back-patch the fix.
2014-07-01 11:23:21 -04:00
Tom Lane 03a25cec8d Issue a WARNING about invalid rule file format in contrib/unaccent.
We were already issuing a WARNING, albeit only elog not ereport, for
duplicate source strings; so warning rather than just being stoically
silent seems like the best thing to do here.  Arguably both of these
complaints should be upgraded to ERRORs, but that might be more
behavioral change than people want.

Note: the faulty line is already printed via an errcontext hook,
so there's no need for more information than these messages provide.
2014-06-30 22:03:37 -04:00
Tom Lane 1b2488731c Allow multi-character source strings in contrib/unaccent.
This could be useful in languages where diacritic signs are represented as
separate characters; more generally it supports using unaccent dictionaries
for substring substitutions beyond narrowly conceived "diacritic removal".
In any case, since the rule-file parser doesn't complain about
multi-character source strings, it behooves us to do something unsurprising
with them.
2014-06-30 21:46:29 -04:00
Tom Lane 97c40ce614 Allow empty replacement strings in contrib/unaccent.
This is useful in languages where diacritic signs are represented as
separate characters; it's also one step towards letting unaccent be used
for arbitrary substring substitutions.

In passing, improve the user documentation for unaccent, which was sadly
vague about some important details.

Mohammad Alhashash, reviewed by Abhijit Menon-Sen
2014-06-30 20:51:30 -04:00
Bruce Momjian 55863274d9 pg_upgrade: update C comments about pg_dumpall
There were some C comments that hadn't been updated from the switch of
using only pg_dumpall to using pg_dump and pg_dumpall, so update them.
Also, don't bother using --schema-only for pg_dumpall --globals-only.

Backpatch through 9.4
2014-06-30 19:56:23 -04:00
Noah Misch 9d0826c595 Don't prematurely free the BufferAccessStrategy in pgstat_heap().
This function continued to use it after heap_endscan() freed it.  In
passing, don't explicit create a strategy here.  Instead, use the one
created by heap_beginscan_strat(), if any.  Back-patch to 9.2, where use
of a BufferAccessStrategy here was introduced.
2014-06-30 16:59:19 -04:00
Bruce Momjian 0f7482733a pg_upgrade: remove pg_multixact files left by initdb
This fixes a bug that caused vacuum to fail when the '0000' files left
by initdb were accessed as part of vacuum's cleanup of old pg_multixact
files.

Backpatch through 9.3
2014-06-24 16:11:06 -04:00
Joe Conway 1dde5782e3 Clean up data conversion short-lived memory context.
dblink uses a short-lived data conversion memory context. However it
was not deleted when no longer needed, leading to a noticeable memory
leak under some circumstances. Plug the hole, along with minor
refactoring. Backpatch to 9.2 where the leak was introduced.

Report and initial patch by MauMau. Reviewed/modified slightly by
Tom Lane and me.
2014-06-20 12:24:59 -07:00
Noah Misch 686f362bee Fix contrib/pg_upgrade/test.sh for $PWD containing spaces.
Most of the necessary quoting was in place; this catches the exceptions.
2014-06-19 21:47:08 -04:00
Noah Misch c82725edfa Let installcheck-world pass against a server requiring a password.
Give passwords to each user created in support of an ECPG connection
test case.  Use SET SESSION AUTHORIZATION, not a fresh connection, to
reduce privileges during a dblink test case.

To test against such a server, both the "make installcheck-world"
environment and the postmaster environment must provide the default
user's password; $PGPASSFILE is the principal way to do so.  (The
postmaster environment needs it for dblink and postgres_fdw tests.)
2014-06-19 21:41:26 -04:00
Tom Lane df8b7bc9ff Improve our mechanism for controlling the Linux out-of-memory killer.
Arrange for postmaster child processes to respond to two environment
variables, PG_OOM_ADJUST_FILE and PG_OOM_ADJUST_VALUE, to determine whether
they reset their OOM score adjustments and if so to what.  This is superior
to the previous design involving #ifdef's in several ways.  The behavior is
now available in a default build, and both ends of the adjustment --- the
original adjustment of the postmaster's level and the subsequent
readjustment by child processes --- can now be controlled in one place,
namely the postmaster launch script.  So it's no longer necessary for the
launch script to act on faith that the server was compiled with the
appropriate options.  In addition, if someone wants to use an OOM score
other than zero for the child processes, that doesn't take a recompile
anymore; and we no longer have to cater separately to the two different
historical kernel APIs for this adjustment.

Gurjeet Singh, somewhat revised by me
2014-06-18 20:12:51 -04:00
Tom Lane 8f889b1083 Implement UPDATE tab SET (col1,col2,...) = (SELECT ...), ...
This SQL-standard feature allows a sub-SELECT yielding multiple columns
(but only one row) to be used to compute the new values of several columns
to be updated.  While the same results can be had with an independent
sub-SELECT per column, such a workaround can require a great deal of
duplicated computation.

The standard actually says that the source for a multi-column assignment
could be any row-valued expression.  The implementation used here is
tightly tied to our existing sub-SELECT support and can't handle other
cases; the Bison grammar would have some issues with them too.  However,
I don't feel too bad about this since other cases can be converted into
sub-SELECTs.  For instance, "SET (a,b,c) = row_valued_function(x)" could
be written "SET (a,b,c) = (SELECT * FROM row_valued_function(x))".
2014-06-18 13:22:34 -04:00
Tom Lane 2146f13408 Avoid recursion when processing simple lists of AND'ed or OR'ed clauses.
Since most of the system thinks AND and OR are N-argument expressions
anyway, let's have the grammar generate a representation of that form when
dealing with input like "x AND y AND z AND ...", rather than generating
a deeply-nested binary tree that just has to be flattened later by the
planner.  This avoids stack overflow in parse analysis when dealing with
queries having more than a few thousand such clauses; and in any case it
removes some rather unsightly inconsistencies, since some parts of parse
analysis were generating N-argument ANDs/ORs already.

It's still possible to get a stack overflow with weirdly parenthesized
input, such as "x AND (y AND (z AND ( ... )))", but such cases are not
mainstream usage.  The maximum depth of parenthesization is already
limited by Bison's stack in such cases, anyway, so that the limit is
probably fairly platform-independent.

Patch originally by Gurjeet Singh, heavily revised by me
2014-06-16 15:55:30 -04:00
Noah Misch be76a6d39e Secure Unix-domain sockets of "make check" temporary clusters.
Any OS user able to access the socket can connect as the bootstrap
superuser and proceed to execute arbitrary code as the OS user running
the test.  Protect against that by placing the socket in a temporary,
mode-0700 subdirectory of /tmp.  The pg_regress-based test suites and
the pg_upgrade test suite were vulnerable; the $(prove_check)-based test
suites were already secure.  Back-patch to 8.4 (all supported versions).
The hazard remains wherever the temporary cluster accepts TCP
connections, notably on Windows.

As a convenient side effect, this lets testing proceed smoothly in
builds that override DEFAULT_PGSOCKET_DIR.  Popular non-default values
like /var/run/postgresql are often unwritable to the build user.

Security: CVE-2014-0067
2014-06-14 09:41:13 -04:00
Heikki Linnakangas 0ef0b6784c Change the signature of rm_desc so that it's passed a XLogRecord.
Just feels more natural, and is more consistent with rm_redo.
2014-06-14 10:46:48 +03:00
Fujii Masao 654e8e4447 Save pg_stat_statements statistics file into $PGDATA/pg_stat directory at shutdown.
187492b6c2 changed pgstat.c so that
the stats files were saved into $PGDATA/pg_stat directory when the server
was shutdowned. But it accidentally forgot to change the location of
pg_stat_statements permanent stats file. This commit fixes pg_stat_statements
so that its stats file is also saved into $PGDATA/pg_stat at shutdown.

Since this fix changes the file layout, we don't back-patch it to 9.3
where this oversight was introduced.
2014-06-04 12:09:45 +09:00
Tom Lane c941aed96b When using the OSSP UUID library, cache its uuid_t state object.
The original coding in contrib/uuid-ossp created and destroyed a uuid_t
object (or, in some cases, even two of them) each time it was called.
This is not the intended usage: you're supposed to keep the uuid_t object
around so that the library can cache its state across uses.  (Other UUID
libraries seem to keep equivalent state behind-the-scenes in static
variables, but OSSP chose differently.)  Aside from being quite inefficient,
creating a new uuid_t loses knowledge of the previously generated UUID,
which in theory could result in duplicate V1-style UUIDs being created
on sufficiently fast machines.

On at least some platforms, creating a new uuid_t also draws some entropy
from /dev/urandom, leaving less for the rest of the system.  This seems
sufficiently unpleasant to justify back-patching this change.
2014-05-29 13:51:02 -04:00
Tom Lane 25dd07e0f6 Fix uuid-ossp regression tests based on buildfarm feedback.
The previous version of these tests expected uuid_generate_v1() to always
emit MAC addresses with the local-admin and multicast address bits zero.
However, several of the buildfarm critters are reporting values with the
local-admin bit set.  (Perhaps they're running inside VMs or jails.)
And a couple are reporting values with the multicast bit set, probably
meaning that the UUID library couldn't read the system MAC address.

Also, it emerges that if OSSP UUID can't read the system MAC address, it
falls back to V1MC behavior wherein the whole node field gets randomized
each time, breaking the test that expected the node field to remain stable
in V1 output.  (It looks like e2fs doesn't behave that way, though.)

It's not entirely clear why we can't get a system MAC address, since the
buildfarm scripts would not work without internet access.  Nonetheless,
the regression tests had better cope with the case, so adjust the tests
to expect these behaviors.
2014-05-28 23:15:51 -04:00
Tom Lane 71ed8b3ca7 Revert "Fix bogus %name-prefix option syntax in all our Bison files."
This reverts commit 45b7abe59e.

It turns out that the %name-prefix syntax without "=" does not work
at all in pre-2.4 Bison.  We are not prepared to make such a large
jump in minimum required Bison version just to suppress a warning
message in a version hardly any developers are using yet.
When 3.0 gets more popular, we'll figure out a way to deal with this.
In the meantime, BISONFLAGS=-Wno-deprecated is recommendable for
anyone using 3.0 who doesn't want to see the warning.
2014-05-28 19:21:01 -04:00
Tom Lane 45b7abe59e Fix bogus %name-prefix option syntax in all our Bison files.
%name-prefix doesn't use an "=" sign according to the Bison docs, but it
silently accepted one anyway, until Bison 3.0.  This was originally a
typo of mine in commit 012abebab1, and we
seem to have slavishly copied the error into all the other grammar files.

Per report from Vik Fearing; analysis by Peter Eisentraut.

Back-patch to all active branches, since somebody might try to build
a back branch with up-to-date tools.
2014-05-28 15:41:53 -04:00
Tom Lane c0f27628d2 Improve regression tests for uuid-ossp.
On reflection, the timestamp-advances test might fail if we're unlucky
enough for the time_mid field to change between two calls, since uuid_cmp
is just bytewise comparison and the field ordering has more significant
fields later.  Build some field extraction functions so we can do a more
honest test of that.  Also check that the version and reserved fields
contain what they should.
2014-05-28 14:21:17 -04:00
Tom Lane 2103218dd4 Fix stack clobber in new uuid-ossp code.
The V5 (SHA1 hashing) code wrote 20 bytes into a 16-byte local variable.
This had accidentally failed to fail in my testing and Matteo's, but
buildfarm results exposed the problem.
2014-05-28 11:50:41 -04:00
Tom Lane b8cc8f9473 Support BSD and e2fsprogs UUID libraries alongside OSSP UUID library.
Allow the contrib/uuid-ossp extension to be built atop any one of these
three popular UUID libraries.  (The extension's name is now arguably a
misnomer, but we'll keep it the same so as not to cause unnecessary
compatibility issues for users.)

We would not normally consider a change like this post-beta1, but the issue
has been forced by our upgrade to autoconf 2.69, whose more rigorous header
checks are causing OSSP's header files to be rejected on some platforms.
It's been foreseen for some time that we'd have to move away from depending
on OSSP UUID due to lack of upstream maintenance, so this is a down payment
on that problem.

While at it, add some simple regression tests, in hopes of catching any
major incompatibilities between the three implementations.

Matteo Beccati, with some further hacking by me
2014-05-27 19:42:08 -04:00
Robert Haas 616afee14d worker_spi: Initialize bgw_notify_pid in all cases.
Commit 090d0f2050 added new code showing
how it can be useful to set bgw_notify_pid to a non-zero value, but it
failed to make sure that the existing call to RegisterBackgroundWorker
initialized the new field at all.

Report and patch by Shigeru Hanada.
2014-05-26 22:56:28 -04:00
Tom Lane 9d7ded0f42 Avoid unportable usage of sscanf(UINT64_FORMAT).
On Mingw, it seems that scanf() doesn't necessarily accept the same format
codes that printf() does, and in particular it may fail to recognize %llu
even though printf() does.  Since configure only probes printf() behavior
while setting up the INT64_FORMAT macros, this means it's unsafe to use
those macros with scanf().  We had only one instance of such a coding
pattern, in contrib/pg_stat_statements, so change that code to avoid
the problem.

Per buildfarm warnings.  Back-patch to 9.0 where the troublesome code
was introduced.

Michael Paquier
2014-05-26 22:23:29 -04:00
Tom Lane 9a65fb3507 Allow total number of transactions in pgbench to exceed INT_MAX.
Change the total-transactions counters from int32 to int64 to accommodate
cases where we do more than 2^31 transactions during a run.  This patch
does not change the INT_MAX limit on explicit "-t" parameters, but it
does allow the product of the -t and -c parameters to exceed INT_MAX, or
allow a -T limit that is large enough that more than 2^31 transactions
can be completed.  While pgbench did not actually fail in such cases,
it did print an incorrect total-transactions count, and some of the
derived numbers such as TPS would have been wrong as well.

Tomas Vondra
2014-05-25 16:33:29 -04:00
Tom Lane e41cc470bb Fix non-C89-compatible coding in pgbench.
C89 says that compound initializers may only contain constant expressions;
a restriction violated by commit 89d00cbe.  While we've had no actual field
complaints about this, C89 is still the project standard, and it's not
saving all that much code to break compatibility here.  So let's adhere to
the old restriction.

In passing, replace a bunch of hardwired constants "256" with
sizeof(target-variable), just because the latter is more readable and
less breakable.  And const-ify where possible.

Back-patch to 9.3 where the nonportable code was added.

Andres Freund and Tom Lane
2014-05-19 00:06:49 -04:00
Tom Lane af215d8190 Suppress some more valgrind whining about btree_gist.
A couple of functions didn't bother to zero out pad bytes in datums that
would ultimately go to disk.  Harmless, but valgrind doesn't know that.
2014-05-16 15:29:02 -04:00
Tom Lane 39586bc1e9 Fix a second cause of undersized pallocs for btree_gist indexes on macaddr.
gbt_macad_union also allocated 12-byte structs where we really need 16.

Per report from Andres Freund.  No back-patch since there's no current
risk of a real problem.
2014-05-16 15:18:09 -04:00
Tom Lane 82bbb60c30 Fix valgrind warning for btree_gist indexes on macaddr.
The macaddr opclass stores two macaddr structs (each of size 6) in an
index column that's declared as being of type gbtreekey16, ie 16 bytes.
In the original coding this led to passing a palloc'd value of size 12
to the index insertion code, so that data would be fetched past the
end of the allocated value during index tuple construction.  This makes
valgrind unhappy.  In principle it could result in a SIGSEGV, though
with the current implementation of palloc there's no risk since
the 12-byte request size would be rounded up to 16 bytes anyway.

To fix, add a field to struct gbtree_ninfo showing the declared size of
the index datums, and use that in the palloc requests; and use palloc0
to be sure that any wasted bytes are cleanly initialized.

Per report from Andres Freund.  No back-patch since there's no current
risk of a real problem.
2014-05-16 15:11:51 -04:00
Heikki Linnakangas d900e192a3 Add test case for logical decoding of prepared transactions.
Andres Freund
2014-05-16 11:34:50 +03:00
Heikki Linnakangas afd0fcbc5d Fix test_decoding test case's check that slot has been dropped.
pg_stat_replication shows connected replication clients. The ddl test case
never has any replication clients connected, so querying pg_stat_replication
is pointless. To check that a slot has been dropped correctly, query
pg_replication_slots instead.

Andres Freund
2014-05-16 11:34:46 +03:00
Peter Eisentraut 2f8e68bd7e Fix whitespace 2014-05-15 13:23:31 -04:00
Bruce Momjian 2e6602dfc2 pg_update: slight major version adjustment 2014-05-14 21:13:59 -04:00
Tom Lane 4456763c78 Fix version check for pg_upgrade line type crosscheck.
Per buildfarm.
2014-05-14 20:23:32 -04:00
Bruce Momjian bb797b6404 pg_upgrade: error out on 'line' data type usage
The data type internal format changed in 9.4.  Also mention this in the
9.4 release notes.
2014-05-14 16:26:13 -04:00
Heikki Linnakangas 540ac7cea9 Initialize padding bytes in btree_gist varbit support.
The code expands a varbit gist leaf key to a node key by copying the bit
data twice in a varlen datum, as both the lower and upper key. The lower key
was expanded to INTALIGN size, but the padding bytes were not initialized.
That's a problem because when the lower/upper keys are compared, the padding
bytes are used compared too, when the values are otherwise equal. That could
lead to incorrect query results.

REINDEX is advised for any btree_gist indexes on bit or bit varying data
type, to fix any garbage padding bytes on disk.

Per Valgrind, reported by Andres Freund. Backpatch to all supported
versions.
2014-05-13 15:15:13 +03:00
Heikki Linnakangas d3c72e23df Avoid some pnstrdup()s when constructing jsonb
This speeds up text to jsonb parsing and hstore to jsonb conversions
somewhat.
2014-05-09 12:46:21 +03:00
Heikki Linnakangas 7572b77359 Fix build after removing JsonbValue.estSize field.
Oops, I didn't realize that contrib/hstore refers to jsonb stuff.
2014-05-08 00:06:28 +03:00
Bruce Momjian 0a78320057 pgindent run for 9.4
This includes removing tabs after periods in C comments, which was
applied to back branches, so this change should not effect backpatching.
2014-05-06 12:12:18 -04:00
Heikki Linnakangas a692ee5870 Replace SYSTEMQUOTEs with Windows-specific wrapper functions.
It's easy to forget using SYSTEMQUOTEs when constructing command strings
for system() or popen(). Even if we fix all the places missing it now, it is
bound to be forgotten again in the future. Introduce wrapper functions that
do the the extra quoting for you, and get rid of SYSTEMQUOTEs in all the
callers.

We previosly used SYSTEMQUOTEs in all the hard-coded command strings, and
this doesn't change the behavior of those. But user-supplied commands, like
archive_command, restore_command, COPY TO/FROM PROGRAM calls, as well as
pgbench's \shell, will now gain an extra pair of quotes. That is desirable,
but if you have existing scripts or config files that include an extra
pair of quotes, those might need to be adjusted.

Reviewed by Amit Kapila and Tom Lane
2014-05-05 16:07:40 +03:00
Tom Lane 2d00190495 Rationalize common/relpath.[hc].
Commit a730183926 created rather a mess by
putting dependencies on backend-only include files into include/common.
We really shouldn't do that.  To clean it up:

* Move TABLESPACE_VERSION_DIRECTORY back to its longtime home in
catalog/catalog.h.  We won't consider this symbol part of the FE/BE API.

* Push enum ForkNumber from relfilenode.h into relpath.h.  We'll consider
relpath.h as the source of truth for fork numbers, since relpath.c was
already partially serving that function, and anyway relfilenode.h was
kind of a random place for that enum.

* So, relfilenode.h now includes relpath.h rather than vice-versa.  This
direction of dependency is fine.  (That allows most, but not quite all,
of the existing explicit #includes of relpath.h to go away again.)

* Push forkname_to_number from catalog.c to relpath.c, just to centralize
fork number stuff a bit better.

* Push GetDatabasePath from catalog.c to relpath.c; it was rather odd
that the previous commit didn't keep this together with relpath().

* To avoid needing relfilenode.h in common/, redefine the underlying
function (now called GetRelationPath) as taking separate OID arguments,
and make the APIs using RelFileNode or RelFileNodeBackend into macro
wrappers.  (The macros have a potential multiple-eval risk, but none of
the existing call sites have an issue with that; one of them had such a
risk already anyway.)

* Fix failure to follow the directions when "init" fork type was added;
specifically, the errhint in forkname_to_number wasn't updated, and neither
was the SGML documentation for pg_relation_size().

* Fix tablespace-path-too-long check in CreateTableSpace() to account for
fork-name component of maximum-length pathnames.  This requires putting
FORKNAMECHARS into a header file, but it was rather useless (and
actually unreferenced) where it was.

The last couple of items are potentially back-patchable bug fixes,
if anyone is sufficiently excited about them; but personally I'm not.

Per a gripe from Christoph Berg about how include/common wasn't
self-contained.
2014-04-30 17:30:50 -04:00
Heikki Linnakangas 503de54621 Add missing SYSTEMQUOTEs
Some popen() calls were missing SYSTEMQUOTEs, which caused initdb and
pg_upgrade to fail on Windows, if the installation path contained both
spaces and @ signs.

Patch by Nikhil Deshpande. Backpatch to all supported versions.
2014-04-30 10:35:52 +03:00