The originally committed patch for modifying CTEs didn't interact well
with EXPLAIN, as noted by myself, and also had corner-case problems with
triggers, as noted by Dean Rasheed. Those problems show it is really not
practical for ExecutorEnd to call any user-defined code; so split the
cleanup duties out into a new function ExecutorFinish, which must be called
between the last ExecutorRun call and ExecutorEnd. Some Asserts have been
added to these functions to help verify correct usage.
It is no longer necessary for callers of the executor to call
AfterTriggerBeginQuery/AfterTriggerEndQuery for themselves, as this is now
done by ExecutorStart/ExecutorFinish respectively. If you really need to
suppress that and do it for yourself, pass EXEC_FLAG_SKIP_TRIGGERS to
ExecutorStart.
Also, refactor portal commit processing to allow for the possibility that
PortalDrop will invoke user-defined code. I think this is not actually
necessary just yet, since the portal-execution-strategy logic forces any
non-pure-SELECT query to be run to completion before we will consider
committing. But it seems like good future-proofing.
HeapTupleHeader's t_infomask and t_infomask2 are defined as 16-bit
unsigned integers, so when the 16th bit was set, heap_page_item was
returning them as negative values, which was ugly.
The change to pageinspect--unpackaged--1.0.sql allows a module upgraded
from 9.0 to be cleanly updated from the previous definition.
File encodings can be specified separately from client encoding.
If not specified, client encoding is used for backward compatibility.
Cases when the encoding doesn't match client encoding are slower
than matched cases because we don't have conversion procs for other
encodings. Performance improvement would be be a future work.
Original patch by Hitoshi Harada, and modified by me.
This is both very useful in its own right, and an important test case
for the core FDW support.
This commit includes a small refactoring of copy.c to expose its option
checking code as a separately callable function. The original patch
submission duplicated hundreds of lines of that code, which seemed pretty
unmaintainable.
Shigeru Hanada, reviewed by Itagaki Takahiro and Tom Lane
intarray and tsearch2 both reference core support functions in their GIN
opclasses, and the signatures of those functions changed for 9.1. We added
backwards-compatible pg_proc entries for the functions in order to allow
9.0 dump files to be restored at all, but that hack leaves the opclasses
pointing at pg_proc entries different from what they'd point to if the
contrib modules were installed fresh in 9.1. To forestall any possibility
of future problems, fix the opclasses to match fresh installs via the
expedient of direct UPDATEs on pg_amproc in the update-from-unpackaged
scripts. (Yech ... but the alternatives are worse, or require far more
effort than seems justified right now.)
Note: updating pg_amproc is sufficient because there will be no pg_depend
entries corresponding to these dependencies, since the referenced functions
are all pinned.
The initial version of the update-from-unpackaged script neglected to
include the <> operators that were added to the opclasses during 9.1.
To make this script produce the same final state as the regular install
script, use the same ALTER OPERATOR FAMILY trick as in pg_trgm.
Take care of some loose ends in the update-from-unpackaged script, and
apply some ugly hacks to ensure that it produces the same catalog state
as the fresh-install script. Per discussion, this seems like a safer
plan than having two different catalog states that both call themselves
"pg_trgm 1.0", even if it's not immediately clear that the subtle
differences would ever matter.
Also, fix the stub function gin_extract_trgm() so that it works instead
of just bleating. Needed because this function will get called during a
regular dump and reload, if there are any indexes using its opclass.
The user won't have an opportunity to update the extension till later,
so telling him to do so is unhelpful.
These are needed to support reloading dumps of 9.0 installations containing
contrib/intarray or contrib/tsearch2. Since not only regular dump/reload
but binary upgrade would fail, it seems worth the trouble to carry these
stubs for awhile. Note that the contrib opclasses referencing these
functions will still work fine, since GIN doesn't actually pay any
attention to the declared signature of a support function.
Initially it was called int_aggregate after the old SQL file, but since
the documentation just says "intagg" and that's also the directory name,
let's conform to that instead.
From first pass of testing. Notably, there seems to be no need for
adminpack--unpackaged--1.0.sql because none of the objects that the
old module creates would ever be dumped by pg_dump anyway (they are
all in pg_catalog).
It was never terribly consistent to use OR REPLACE (because of the lack of
comparable functionality for data types, operators, etc), and
experimentation shows that it's now positively pernicious in the extension
world. We really want a failure to occur if there are any conflicts, else
it's unclear what the extension-ownership state of the conflicted object
ought to be. Most of the time, CREATE EXTENSION will fail anyway because
of conflicts on other object types, but an extension defining only
functions can succeed, with bad results.
This isn't fully tested as yet, in particular I'm not sure that the
"foo--unpackaged--1.0.sql" scripts are OK. But it's time to get some
buildfarm cycles on it.
sepgsql is not converted to an extension, mainly because it seems to
require a very nonstandard installation process.
Dimitri Fontaine and Tom Lane
relative, by creating a function path_is_relative_and_below_cwd() to
check for specific requirements. It is unclear if this fixes a security
problem or not but the new code is more robust.
This follows recent discussions, so it's quite a bit different from
Dimitri's original. There will probably be more changes once we get a bit
of experience with it, but let's get it in and start playing with it.
This is still just core code. I'll start converting contrib modules
shortly.
Dimitri Fontaine and Tom Lane
This follows my proposal of yesterday, namely that we try to recreate the
previous state of the extension exactly, instead of allowing CREATE
EXTENSION to run a SQL script that might create some entirely-incompatible
on-disk state. In --binary-upgrade mode, pg_dump won't issue CREATE
EXTENSION at all, but instead uses a kluge function provided by
pg_upgrade_support to recreate the pg_extension row (and extension-level
pg_depend entries) without creating any member objects. The member objects
are then restored in the same way as if they weren't members, in particular
using pg_upgrade's normal hacks to preserve OIDs that need to be preserved.
Then, for each member object, ALTER EXTENSION ADD is issued to recreate the
pg_depend entry that marks it as an extension member.
In passing, fix breakage in pg_upgrade's enum-type support: somebody didn't
fix it when the noise word VALUE got added to ALTER TYPE ADD. Also,
rationalize parsetree representation of COMMENT ON DOMAIN and fix
get_object_address() to allow OBJECT_DOMAIN.
This adds collation support for columns and domains, a COLLATE clause
to override it per expression, and B-tree index support.
Peter Eisentraut
reviewed by Pavel Stehule, Itagaki Takahiro, Robert Haas, Noah Misch
Unlike Btree-based LIKE optimization, this works for non-left-anchored
search patterns. The effectiveness of the search depends on how many
trigrams can be extracted from the pattern. (The worst case, with no
trigrams, degrades to a full-table scan, so this isn't a panacea. But
it can be very useful.)
Alexander Korotkov, reviewed by Jan Urbanski
contrib/intarray's gettoken() uses a fixed-size buffer to collect an
integer's digits, and did not guard against overrunning the buffer.
This is at least a backend crash risk, and in principle might allow
arbitrary code execution. The code didn't check for overflow of the
integer value either, which while not presenting a crash risk was still
bad.
Thanks to Apple Inc's security team for reporting this issue and supplying
the fix.
Security: CVE-2010-4015
This is still pretty rough - among other things, the documentation
needs work, and the messages need a visit from the style police -
but this gets the basic framework in place.
KaiGai Kohei
Reduce #includes to minimum actually needed; in particular include
postgres_fe.h not postgres.h, so as to stop build failures on some
platforms.
Use get_progname() instead of hardwired program name; improve error
checking for command line syntax; bring error messages into line with
style guidelines; include strerror result in die() cases.
Un-break Windows build (I hope) by making the HAVE_FSYNC_WRITETHROUGH
code match the backend. Fix incorrect program help message. static-ize
all functions.
Actually rename the program, rather than just claiming we did. Hook it
into the build system. Get rid of useless dependency on libpq. Clean up
#include list and messy whitespace.
In particular, make hstore @> '' succeed for all hstores, likewise
hstore ?& '{}'. Previously the results were inconsistent and could
depend on whether you were using a GiST index, GIN index, or seqscan.
This applies the fix for bug #5784 to remaining places where we wish
to reject nulls in user-supplied arrays. In all these places, there's
no reason not to allow a null bitmap to be present, so long as none of
the current elements are actually null.
I did not change some other places where we are looking at system catalog
entries or aggregate transition values, as the presence of a null bitmap
in such an array would be suspicious.
The array containment operators now behave per mathematical expectation
for empty arrays (ie, an empty array is contained in anything).
Both these operators and the query_int operators now work as expected in
GiST and GIN index searches, rather than having corner cases where the
index searches gave different answers.
Also, fix unexpected failures where the operators would claim that an array
contained nulls, when in fact there was no longer any null present (similar
to bug #5784). The restriction to not have nulls is still there, as
removing it would take a lot of added code complexity and probably slow
things down significantly.
Also, remove the arbitrary restriction to 1-D arrays; unlike the other
restriction, this was buying us nothing performance-wise.
Assorted cosmetic improvements and marginal performance improvements, too.
up relations, but rather order old/new relations and use the same array
index value for both. This should speed up pg_upgrade for databases
with many relations.
which is stored in pg_largeobject_metadata.
No backpatch to 9.0 because you can't migrate from 9.0 to 9.0 with the
same catversion (because of tablespace conflict), and a pre-9.0
migration to 9.0 has not large object permissions to migrate.
Toast tables have identical pg_class.oid and pg_class.relfilenode, but
for clarity it is good to preserve the pg_class.oid.
Update comments regarding what is preserved, and do some
variable/function renaming for clarity.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
Foreign tables are a core component of SQL/MED. This commit does
not provide a working SQL/MED infrastructure, because foreign tables
cannot yet be queried. Support for foreign table scans will need to
be added in a future patch. However, this patch creates the necessary
system catalog structure, syntax support, and support for ancillary
operations such as COMMENT and SECURITY LABEL.
Shigeru Hanada, heavily revised by Robert Haas
pointers, which simplifies the code. This was not possible in 9.0 because
everything was in a single nested struct, but is possible now.
Per suggestion from Tom.
Don't insist on pg_dumpall and psql being present in the old cluster,
since they are not needed. Do insist on pg_resetxlog being present
(in both old and new), since we need it. Also check for pg_config,
but only in the new cluster. Remove the useless attempt to call
pg_config in the old cluster; we don't need to know the old value of
--pkglibdir. (In the case of a stripped-down migration installation
there might be nothing there to look at anyway, so any future change
that might reintroduce that need would have to be considered carefully.)
Per my attempts to build a minimal previous-version installation to support
pg_upgrade.
It appears that this will be faster for all but the shortest strings;
at least one some platforms, memcmp() can use word-at-a-time comparisons.
Noah Misch, somewhat pared down.
After parsing a parenthesized subexpression, we must pop all pending
ANDs and NOTs off the stack, just like the case for a simple operand.
Per bug #5793.
Also fix clones of this routine in contrib/intarray and contrib/ltree,
where input of types query_int and ltxtquery had the same problem.
Back-patch to all supported versions.
This patch replaces Guttman's generalized split method with a simple
sort-by-center-points algorithm. Since the data is only one-dimensional
we don't really need the slow and none-too-stable Guttman method.
This is in part a bug fix, since seg has the same size_alpha versus
size_beta typo that was recently fixed in contrib/cube. It seems
prudent to apply this rather aggressive fix only in HEAD, though.
Back branches will just get the typo fix.
Alexander Korotkov, reviewed by Yeb Havinga
1. Don't reimplement S_ISDIR() and S_ISREG() badly.
2. Don't reimplement access() badly.
This code appears to have been copied from ancient versions of the
corresponding backend routines, and not patched to incorporate subsequent
fixes (see my commits of 2008-03-31 and 2010-01-14 respectively).
It might be a good idea to change it to just *call* those routines,
but for now I'll just transpose these fixes over.
Most of the functions that execute XPath queries leaked the data structures
created by libxml2. This memory would not be recovered until end of
session, so it mounts up pretty quickly in any serious use of the feature.
Per report from Pavel Stehule, though this isn't his patch.
Back-patch to all supported branches.
This eliminates the need for inefficient implementions of this
functionality in both contrib/dblink and contrib/tablefunc, so remove
them. The upcoming patch implementing an in-core format() function
will also require this functionality.
In passing, add some regression tests.
Having this in src/include/port.h makes no sense, now that copydir.c lives
in src/backend/strorage rather than src/port. Along the way, remove an
obsolete comment from contrib/pg_upgrade that makes reference to the old
location.
Replace for loops in makefiles with proper dependencies. Parallel
make can now span across directories. Also, make -k and make -q work
properly.
GNU make 3.80 or newer is now required.
After much expenditure of effort, we've got this to the point where the
performance penalty is pretty minimal in typical cases.
Andrew Dunstan, reviewed by Brendan Jurd, Dean Rasheed, and Tom Lane
Various places were testing TRIGGER_FIRED_BEFORE() where what they really
meant was !TRIGGER_FIRED_AFTER(), or vice versa. This needs to be cleaned
up because there are about to be more than two possible states.
We might want to note this in the 9.1 release notes as something for
trigger authors to double-check.
For consistency's sake I also changed some places that assumed that
TRIGGER_FIRED_FOR_ROW and TRIGGER_FIRED_FOR_STATEMENT are necessarily
mutually exclusive; that's not in immediate danger of breaking, but
it's still sloppier than it should be.
Extracted from Dean Rasheed's patch for triggers on views. I'm committing
this separately since it's an identifiable separate issue, and is the
only reason for the patch to touch most of these particular files.
This is intended as infrastructure to support integration with label-based
mandatory access control systems such as SE-Linux. Further changes (mostly
hooks) will be needed, but this is a big chunk of it.
KaiGai Kohei and Robert Haas
There was an incorrect Assert in hstoreValidOldFormat(), which would cause
immediate core dumps when attempting to work with pre-9.0 hstore data,
but of course only in an assert-enabled build.
Also, ghstore_decompress() incorrectly applied DatumGetHStoreP() to a datum
that wasn't actually an hstore, but rather a ghstore (ie, a gist signature
bitstring). That used to be harmless, but could now result in misbehavior
if the hstore format conversion code happened to trigger. In reality,
since ghstore is not marked toastable (and doesn't need to be), this
function is useless anyway; we can lobotomize it down to returning the
passed-in pointer.
Both bugs found by Andrew Gierth, though this isn't exactly his proposed
patch.
functions to the core XML code. Per discussion, the former depends on
XMLOPTION while the others do not. These supersede a version previously
offered by contrib/xml2.
Mike Fowler, reviewed by Pavel Stehule
out immediately on any out-of-memory condition. It's rather pointless to
imagine that pgbench will be able to continue usefully after a malloc
failure, and in any case there were a number of unchecked mallocs.
pairs that can be handled by xslt_process().
There is much else to do here, but this patch seems useful in its own right
for as long as this code survives.
Pavel Stehule, reviewed by Mike Fowler
- Rename TSParserGetPrsid to get_ts_parser_oid.
- Rename TSDictionaryGetDictid to get_ts_dict_oid.
- Rename TSTemplateGetTmplid to get_ts_template_oid.
- Rename TSConfigGetCfgid to get_ts_config_oid.
- Rename FindConversionByName to get_conversion_oid.
- Rename GetConstraintName to get_constraint_oid.
- Add new functions get_opclass_oid, get_opfamily_oid, get_rewrite_oid,
get_rewrite_oid_without_relid, get_trigger_oid, and get_cast_oid.
The name of each function matches the corresponding catalog.
Thanks to KaiGai Kohei for the review.
Operating directly on the underlying varlena saves palloc and memcpy
overhead, which testing shows to be significant.
Extracted from a larger patch by Alexander Korotkov.
supposing that they should set SHLIB_LINK rather than LDFLAGS_SL. Since these
don't go through Makefile.shlib that was a no-op on most platforms. Also
regularize the few platform-specific Makefiles that did pay attention to
SHLIB_LINK: it seems that the real value of that is to pull in BE_DLLLIBS,
so do that instead. Per buildfarm failures on cygwin.
dblink_build_sql_insert() and related functions. Now the column numbers
are treated as logical not physical column numbers. This will provide saner
behavior in the presence of dropped columns; furthermore, if we ever get
around to allowing rearrangement of logical column ordering, the original
definition would become nearly untenable from a usability standpoint.
Per recent discussion of dblink's handling of dropped columns.
Not back-patched for fear of breaking existing applications.
columns correctly. In passing, get rid of some dead logic in the
underlying get_sql_insert() etc functions --- there is no caller that
will pass null value-arrays to them.
Per bug report from Robert Voinea.
dblink_build_sql_insert() and related functions. In particular, be sure to
reject references to dropped and out-of-range column numbers. The numbers
are still interpreted as physical column numbers, though, for backward
compatibility.
This patch replaces Joe's patch of 2010-02-03, which handled only some aspects
of the problem.
lock the target relation just once per SQL function call. The original coding
obtained and released lock several times per call. Aside from saving a
not-insignificant number of cycles, this eliminates possible race conditions
if someone tries to modify the relation's schema concurrently. Also
centralize locking and permission-checking logic.
Problem noted while investigating a trouble report from Robert Voinea --- his
problem is still to be fixed, though.
hardcoding a 'template0' check, per suggestion from Alvaro.
This might fix a problem where someone has allowed 'template0'
connections, but it is a cleaner approach even if doesn't fix the
bug.
* There is no chmod() on Windows.
* Must always use the 3-parameter version of open()
* There is no dynloader.h - but it also appears unnecessary on all platforms
* Don't include shlobj.h because it causes compile errors, and from what I can
see it's not actually used. This may need to be added back for mingw
and/or cygwin in the worst case.
cmp parameter for pg_scandir(). The code failed to support this anyway
for Sun/Windows, so pretending we could accept a parameter other than
NULL was just asking for trouble.
rather than returning NULL for some-but-not-all failures as they used to.
Remove now-redundant tests for NULL from call sites.
We had to do something about this because many call sites were failing to
check for NULL; and changing it like this seems a lot more useful and
mistake-proof than adding checks to the call sites without them.
unsatisfiable query, such as indexcol && empty_array. It should return -1
to tell GIN no scan is required; but silly typo disabled the logic for that,
resulting in unnecessary "GIN indexes do not support whole-index scans" error.
Per bug report from Jeff Trout.
Back-patch to 8.3 where the logic was introduced.
writes. The first worker still uses "pgbench_log.<pid>" for the name, but
additional workers use "pgbench_log.<pid>.<serial-number>" instead.
Reported by Greg Smith.
in versions >= 8.3). The core code is more robust and efficient than what
was there before, and this also reduces risks involved in swapping different
libxml error handler settings.
Before 8.3, there is still some risk of problems if add-on modules such as
Perl invoke libxml without setting their own error handler. Given the lack
of reports I'm not sure there's a risk in practice, so I didn't take the
step of actually duplicating the core code into older contrib/xml2 branches.
Instead I just tweaked the existing code to ensure it didn't leave a dangling
pointer to short-lived memory when throwing an error.
This involves modifying the module to have a stable ABI, that is, the
xslt_process() function still exists even without libxslt. It throws a
runtime error if called, but doesn't prevent executing the CREATE FUNCTION
call. This is a good thing anyway to simplify cross-version upgrades.
These are unnecessary and probably dangerous. I don't see any immediate
risk situations in the core XML support or contrib/xml2 itself, but there
could be issues with external uses of libxml2, and in any case it's an
accident waiting to happen.
Get rid of the code that attempted to funnel libxml2's memory allocations
into palloc. We already knew from experience with the core xml datatype
that trying to do this is simply not reliable. Unlike the core code, I
did not bother adding a lot of PG_TRY/PG_CATCH logic to try to ensure that
everything is cleaned up on error exit. Hence, we might leak some memory
if one of these functions fails partway through. Given the deprecated
status of this contrib module and the fact that errors partway through
the functions shouldn't be too common, it doesn't seem worth worrying about.
Also fix a separate bug in xpath_table, that it did the wrong things
if given a result tuple descriptor with less than 2 columns. While
such a case isn't very useful in practice, we shouldn't fail or stomp
memory when it occurs.
Add some simple regression tests based on all the reported crash cases
that I have on hand.
This should be back-patched, but let's see if the buildfarm likes it first.
The main motivation for changing this is bug #4921, in which it's pointed out
that it's no longer safe to apply ltree operations to the result of
ARRAY(SELECT ...) if the sub-select might return no rows. Before 8.3,
the ARRAY() construct would return NULL, which might or might not be helpful
but at least it wouldn't result in an error. Now it returns an empty array
which results in a failure for no good reason, since the ltree operations
are all perfectly capable of dealing with zero-element arrays.
As far as I can find, these ltree functions are the only places where zero
array dimensionality is rejected unnecessarily.
Back-patch to 8.3 to prevent behavioral regression of queries that worked
in older releases.
The purpose of this change is to eliminate the need for every caller
of SearchSysCache, SearchSysCacheCopy, SearchSysCacheExists,
GetSysCacheOid, and SearchSysCacheList to know the maximum number
of allowable keys for a syscache entry (currently 4). This will
make it far easier to increase the maximum number of keys in a
future release should we choose to do so, and it makes the code
shorter, too.
Design and review by Tom Lane.
being called as aggregates, and to get the aggregate transition state memory
context if needed. Use it instead of poking directly into AggState and
WindowAggState in places that shouldn't know so much.
We should have done this in 8.4, probably, but better late than never.
Revised version of a patch by Hitoshi Harada.
of shared or nailed system catalogs. This has two key benefits:
* The new CLUSTER-based VACUUM FULL can be applied safely to all catalogs.
* We no longer have to use an unsafe reindex-in-place approach for reindexing
shared catalogs.
CLUSTER on nailed catalogs now works too, although I left it disabled on
shared catalogs because the resulting pg_index.indisclustered update would
only be visible in one database.
Since reindexing shared system catalogs is now fully transactional and
crash-safe, the former special cases in REINDEX behavior have been removed;
shared catalogs are treated the same as non-shared.
This commit does not do anything about the recently-discussed problem of
deadlocks between VACUUM FULL/CLUSTER on a system catalog and other
concurrent queries; will address that in a separate patch. As a stopgap,
parallel_schedule has been tweaked to run vacuum.sql by itself, to avoid
such failures during the regression tests.
exceed the total number of non-dropped source table fields for
dblink_build_sql_*(). Addresses bug report from Rushabh Lathia.
Backpatch all the way to the 7.3 branch.
(SFRM_Materialize mode) to return tuples. Since we don't return from the
dblink function in tuplestore mode, release the PGresult with a PG_CATCH
block on error. Also rearrange to share the same code to materialize the
tuplestore. Patch by Takahiro Itagaki.
Tom Lane
Alvaro Herrera
Itagaki Takahiro
Robert Haas
Bruce Momjian
Peter Eisentraut
Heikki Linnakangas
Magnus Hagander
Simon Riggs
Teodor Sigaev
Andrew Dunstan
Joe Conway