query node, since that won't work unless the planner is upgraded.
Someday we should try to support at least some cases of this, but for
now just plug the hole in the dike. Per discussion with Dmitry Tkach.
Both plannable queries and utility commands are now always executed
within Portals, which have been revamped so that they can handle the
load (they used to be good only for single SELECT queries). Restructure
code to push command-completion-tag selection logic out of postgres.c,
so that it won't have to be duplicated between simple and extended queries.
initdb forced due to addition of a field to Query nodes.
that the types of untyped string-literal constants are deduced (ie,
when coerce_type is applied to 'em, that's what the type must be).
Remove the ancient hack of storing the input Param-types array as a
global variable, and put the info into ParseState instead. This touches
a lot of files because of adjustment of routine parameter lists, but
it's really not a large patch. Note: PREPARE statement still insists on
exact specification of parameter types, but that could easily be relaxed
now, if we wanted to do so.
recursion in RewriteQuery(); also, detect recursion in fireRIRrules(),
so as to catch self-referential views per example from Ryan VanderBijl.
Minor code restructuring to make it easier to catch recursive case.
simplify callers. It turns out the common case is that the caller
does want to recurse into sub-queries, so push support for that into
these subroutines.
so that all executable expression nodes inherit from a common supertype
Expr. This is somewhat of an exercise in code purity rather than any
real functional advance, but getting rid of the extra Oper or Func node
formerly used in each operator or function call should provide at least
a little space and speed improvement.
initdb forced by changes in stored-rules representation.
whose conditions might yield NULL. The negated qual to attach to the
original query is properly 'x IS NOT TRUE', not 'NOT x'. This fix
produces correct behavior, but we may be taking a performance hit because
the planner is much stupider about IS NOT TRUE than it is about NOT
clauses. Future TODO: teach prepqual, other parts of planner how to
cope with BooleanTest clauses more effectively.
rather than being reordered according to INSTEAD attribute for
implementation convenience.
Also, increase compiled-in recursion depth limit from 10 to 100 rewrite
cycles. 10 seems pretty marginal for situations where multiple rules
exist for the same query. There was a complaint about this recently,
so I'm going to bump it up. (Perhaps we should make the limit a GUC
parameter, but that's too close to being a new feature to do in beta.)
to be flexible about assignment casts without introducing ambiguity in
operator/function resolution. Introduce a well-defined promotion hierarchy
for numeric datatypes (int2->int4->int8->numeric->float4->float8).
Change make_const to initially label numeric literals as int4, int8, or
numeric (never float8 anymore).
Explicitly mark Func and RelabelType nodes to indicate whether they came
from a function call, explicit cast, or implicit cast; use this to do
reverse-listing more accurately and without so many heuristics.
Explicit casts to char, varchar, bit, varbit will truncate or pad without
raising an error (the pre-7.2 behavior), while assigning to a column without
any explicit cast will still raise an error for wrong-length data like 7.3.
This more nearly follows the SQL spec than 7.2 behavior (we should be
reporting a 'completion condition' in the explicit-cast cases, but we have
no mechanism for that, so just do silent truncation).
Fix some problems with enforcement of typmod for array elements;
it didn't work at all in 'UPDATE ... SET array[n] = foo', for example.
Provide a generalized array_length_coerce() function to replace the
specialized per-array-type functions that used to be needed (and were
missing for NUMERIC as well as all the datetime types).
Add missing conversions int8<->float4, text<->numeric, oid<->int8.
initdb forced.
that are explicitly JOINed are not considered dependencies unless they
are actually used in the query: mere presence in the joinaliasvars
list of a JOIN RTE doesn't count as being used. The patch touches
a number of files because I needed to generalize the API of
query_tree_walker to support an additional flag bit, but the changes
are otherwise quite small.
array header, and to compute sizing and alignment of array elements
the same way normal tuple access operations do --- viz, using the
tupmacs.h macros att_addlength and att_align. This makes the world
safe for arrays of cstrings or intervals, and should make it much
easier to write array-type-polymorphic functions; as examples see
the cleanups of array_out and contrib/array_iterator. By Joe Conway
and Tom Lane.
code review by Tom Lane. Remaining issues: functions that take or
return tuple types are likely to break if one drops (or adds!)
a column in the table defining the type. Need to think about what
to do here.
Along the way: some code review for recent COPY changes; mark system
columns attnotnull = true where appropriate, per discussion a month ago.
COPY x (a,d,c,b) from stdin;
COPY x (a,c) to stdout;
as well as the corresponding changes to pg_dump to use the new
functionality. This functionality is not available when using
the BINARY option. If a column is not specified in the COPY FROM
statement, its default values will be used.
In addition to this functionality, I tweaked a couple of the
error messages emitted by the new COPY <options> checks.
Brent Verner
entries, per pghackers discussion. This fixes aggregates to live in
namespaces, and also simplifies/speeds up lookup in parse_func.c.
Also, add a 'proimplicit' flag to pg_proc that controls whether a type
coercion function may be invoked implicitly, or only explicitly. The
current settings of these flags are more permissive than I would like,
but we will need to debate and refine the behavior; for now, I avoided
breaking regression tests as much as I could.
INSERT statements to the planner. Taking it out of the parser was right
(so that defaults don't get into stored rules), but it has to happen
before rewrite rule expansion, else references to NEW.field behave
incorrectly. Accordingly, add a step to the rewriter to insert defaults
just before rewrite-rule expansion.
addRangeTableEntry calls. Remove relname field from RTEs, since
it will no longer be a useful unique identifier of relations;
we want to encourage people to rely on the relation OID instead.
Further work on dumping qual expressions in EXPLAIN, too.
now has an RTE of its own, and references to its outputs now are Vars
referencing the JOIN RTE, rather than CASE-expressions. This allows
reverse-listing in ruleutils.c to use the correct alias easily, rather
than painfully reverse-engineering the alias namespace as it used to do.
Also, nested FULL JOINs work correctly, because the result of the inner
joins are simple Vars that the planner can cope with. This fixes a bug
reported a couple times now, notably by Tatsuo on 18-Nov-01. The alias
Vars are expanded into COALESCE expressions where needed at the very end
of planning, rather than during parsing.
Also, beginnings of support for showing plan qualifier expressions in
EXPLAIN. There are probably still cases that need work.
initdb forced due to change of stored-rule representation.
in cases of qualified rules as well as unqualified ones. Tweak rules
test to avoid cluttering output with dummy SELECT results. Update
documentation to match code.
manipulation of rtable/jointree by planner. Rewriter was generating
actions that shared rtable/jointree substructure, which caused havoc
when planner got to the later actions that it'd already mucked up.
or view that's been dropped and then recreated with the same name (but,
perhaps, different columns). Eventually we'd like to support this but
for now all we can do is fail cleanly, rather than possibly coredumping
if we proceed using the obsolete rule.
original table ('OLD' table) in its join tree if OLD is referenced by
either the rule action, the rule qual, or the original query qual that
will be added to the rule action. However, we only want one instance
of the original table to be included; so beware of the possibility that
the rule action already has a jointree entry for OLD.
work where we can (given that the executor only handles it at top level)
and generate an error where we can't. Note that while the parser has
been allowing views to say SELECT FOR UPDATE for a few weeks now, that
hasn't actually worked until just now.
report from Joel Burton. Turns out that my simple idea of turning the
SELECT into a subquery does not interact well *at all* with the way the
rule rewriter works. Really what we need to make INSERT ... SELECT work
cleanly is to decouple targetlists from rangetables: an INSERT ... SELECT
wants to have two levels of targetlist but only one rangetable. No time
for that for 7.1, however, so I've inserted some ugly hacks to make the
rewriter know explicitly about the structure of INSERT ... SELECT queries.
Ugh :-(
(WAL logging for this is not done yet, however.) Clean up a number of really
crufty things that are no longer needed now that DROP behaves nicely. Make
temp table mapper do the right things when drop or rename affecting a temp
table is rolled back. Also, remove "relation modified while in use" error
check, in favor of locking tables at first reference and holding that lock
throughout the statement.
SQL92 semantics, including support for ALL option. All three can be used
in subqueries and views. DISTINCT and ORDER BY work now in views, too.
This rewrite fixes many problems with cross-datatype UNIONs and INSERT/SELECT
where the SELECT yields different datatypes than the INSERT needs. I did
that by making UNION subqueries and SELECT in INSERT be treated like
subselects-in-FROM, thereby allowing an extra level of targetlist where the
datatype conversions can be inserted safely.
INITDB NEEDED!
(Don't forget that an alias is required.) Views reimplemented as expanding
to subselect-in-FROM. Grouping, aggregates, DISTINCT in views actually
work now (he says optimistically). No UNION support in subselects/views
yet, but I have some ideas about that. Rule-related permissions checking
moved out of rewriter and into executor.
INITDB REQUIRED!
user is now defined in terms of the user id, the user name is only computed
upon request (for display purposes). This is kind of the opposite of the
previous state, which would maintain the user name and compute the user id
for permission checks.
Besides perhaps saving a few cycles (integer vs string), this now creates a
single point of attack for changing the user id during a connection, for
purposes of "setuid" functions, etc.
from Param nodes, per discussion a few days ago on pghackers. Add new
expression node type FieldSelect that implements the functionality where
it's actually needed. Clean up some other unused fields in Func nodes
as well.
NOTE: initdb forced due to change in stored expression trees for rules.
entry that has rules. This allows us to release the rule parsetrees
on relcache flush without needing a working freeObject() routine.
Formerly, the rule trees were leaked permanently at relcache flush.
Also, clean up handling of rule creation and deletion --- there was
not sufficient locking of the relation being modified, and there was
no reliable notification of other backends that a relcache reload
was needed. Also, clean up relcache.c code so that scans of system
tables needed to load a relcache entry are done in the caller's
memory context, not in CacheMemoryContext. This prevents any
un-pfreed memory from those scans from becoming a permanent memory
leak.
to give wrong results: it should be looking at inJoinSet not inFromCl.
Also, make 'modified' flag be local to ApplyRetrieveRule: we should
append a rule's quals to the query iff that particular rule applies,
not if we have fired any previously-considered rule for the query!
be an expression not just a simple Var, so long as only one table is
referenced (so that code isn't really any more difficult than before).
This whole thing is still fundamentally bogus, but at least we can accept
a few more cases than before.
mark query as having subselects if a subselect was added from a rule
WHERE condition (as opposed to a rule action). Also, fix adjustment
of varlevelsup so that it actually has some prospect of working when
inserting an expression containing a subselect into a subquery.
SELECT DISTINCT ON (expr [, expr ...]) targetlist ...
and there is a check to make sure that the user didn't specify an ORDER BY
that's incompatible with the DISTINCT operation.
Reimplement nodeUnique and nodeGroup to use the proper datatype-specific
equality function for each column being compared --- they used to do
bitwise comparisons or convert the data to text strings and strcmp().
(To add insult to injury, they'd look up the conversion functions once
for each tuple...) Parse/plan representation of DISTINCT is now a list
of SortClause nodes.
initdb forced by querytree change...
subselects can only appear on the righthand side of a binary operator.
That's still true for quantified predicates like x = ANY (SELECT ...),
but a subselect that delivers a single result can now appear anywhere
in an expression. This is implemented by changing EXPR_SUBLINK sublinks
to represent just the (SELECT ...) expression, without any 'left hand
side' or combining operator --- so they're now more like EXISTS_SUBLINK.
To handle the case of '(x, y, z) = (SELECT ...)', I added a new sublink
type MULTIEXPR_SUBLINK, which acts just like EXPR_SUBLINK used to.
But the grammar will only generate one for a multiple-left-hand-side
row expression.
mentioned in FROM but not elsewhere in the query: such tables should be
joined over anyway. Aside from being more standards-compliant, this allows
removal of some very ugly hacks for COUNT(*) processing. Also, allow
HAVING clause without aggregate functions, since SQL does. Clean up
CREATE RULE statement-list syntax the same way Bruce just fixed the
main stmtmulti production.
CAUTION: addition of a field to RangeTblEntry nodes breaks stored rules;
you will have to initdb if you have any rules.
modifyAggrefQual. This routine really, really needs to be retired, but
until we have subselects in FROM there's no chance of doing the job right.
In the meantime try to respond to unhandlable cases with elog rather than
coredump.
expression_tree_mutator rather than ad-hoc tree walking code. This shortens
the code materially and fixes a fair number of sins of omission. Also,
change modifyAggrefQual to *not* recurse into subselects, since its mission
is satisfied if it removes aggregate functions from the top level of a
WHERE clause. This cures problems with queries of the form SELECT ...
WHERE x IN (SELECT ... HAVING something-using-an-aggregate), which would
formerly get mucked up by modifyAggrefQual. The routine is still
fundamentally broken, of course, but I don't think there's any way to get
rid of it before we implement subselects in FROM ...
additional argument specifying the kind of lock to acquire/release (or
'NoLock' to do no lock processing). Ensure that all relations are locked
with some appropriate lock level before being examined --- this ensures
that relevant shared-inval messages have been processed and should prevent
problems caused by concurrent VACUUM. Fix several bugs having to do with
mismatched increment/decrement of relation ref count and mismatched
heap_open/close (which amounts to the same thing). A bogus ref count on
a relation doesn't matter much *unless* a SI Inval message happens to
arrive at the wrong time, which is probably why we got away with this
sloppiness for so long. Repair missing grab of AccessExclusiveLock in
DROP TABLE, ALTER/RENAME TABLE, etc, as noted by Hiroshi.
Recommend 'make clean all' after pulling this update; I modified the
Relation struct layout slightly.
Will post further discussion to pghackers list shortly.
documented intepretation of the lefthand and oper fields. Fix a number of
obscure problems while at it --- for example, the old code failed if the parser
decided to insert a type-coercion function just below the operator of a
SubLink.
CAUTION: this will break stored rules that contain subplans. You may
need to initdb.
with expression_tree_walker-based code. The former failed to cope with
expressions containing SubLinks, and the latter returned TRUE for both
SubLinks and Aggrefs (cut-and-paste bug?). There is a lot more scope for
using expression_tree_walker in this module, but I'll restrain myself
until the 6.6 split occurs from touching not-demonstrably-broken code.
not be marked inFromCl any longer. Otherwise the planner gets confused
and joins over them where in fact it does not have to.
Adjust hasSubLinks now with a recursive lookup - could be wrong in
multi action rules because parse state isn't reset correctly and all
actions in the rule are marked hasSubLinks if one of them has.
Jan
lists are now plain old garden-variety Lists, allocated with palloc,
rather than specialized expansible-array data allocated with malloc.
This substantially simplifies their handling and eliminates several
sources of memory leakage.
Several basic types of erroneous queries (syntax error, attempt to
insert a duplicate key into a unique index) now demonstrably leak
zero bytes per query.
Ok. I made patches replacing all of "#if FALSE" or "#if 0" to "#ifdef
NOT_USED" for current. I have tested these patches in that the
postgres binaries are identical.
INTERSECT and EXCEPT is available for postgresql-v6.4!
The patch against v6.4 is included at the end of the current text
(in uuencoded form!)
I also included the text of my Master's Thesis. (a postscript
version). I hope that you find something of it useful and would be
happy if parts of it find their way into the PostgreSQL documentation
project (If so, tell me, then I send the sources of the document!)
The contents of the document are:
-) The first chapter might be of less interest as it gives only an
overview on SQL.
-) The second chapter gives a description on much of PostgreSQL's
features (like user defined types etc. and how to use these features)
-) The third chapter starts with an overview of PostgreSQL's internal
structure with focus on the stages a query has to pass (i.e. parser,
planner/optimizer, executor). Then a detailed description of the
implementation of the Having clause and the Intersect/Except logic is
given.
Originally I worked on v6.3.2 but never found time enough to prepare
and post a patch. Now I applied the changes to v6.4 to get Intersect
and Except working with the new version. Chapter 3 of my documentation
deals with the changes against v6.3.2, so keep that in mind when
comparing the parts of the code printed there with the patched sources
of v6.4.
Here are some remarks on the patch. There are some things that have
still to be done but at the moment I don't have time to do them
myself. (I'm doing my military service at the moment) Sorry for that
:-(
-) I used a rewrite technique for the implementation of the Except/Intersect
logic which rewrites the query to a semantically equivalent query before
it is handed to the rewrite system (for views, rules etc.), planner,
executor etc.
-) In v6.3.2 the types of the attributes of two select statements
connected by the UNION keyword had to match 100%. In v6.4 the types
only need to be familiar (i.e. int and float can be mixed). Since this
feature did not exist when I worked on Intersect/Except it
does not work correctly for Except/Intersect queries WHEN USED IN
COMBINATION WITH UNIONS! (i.e. sometimes the wrong type is used for the
resulting table. This is because until now the types of the attributes of
the first select statement have been used for the resulting table.
When Intersects and/or Excepts are used in combination with Unions it
might happen, that the first select statement of the original query
appears at another position in the query which will be executed. The reason
for this is the technique used for the implementation of
Except/Intersect which does a query rewrite!)
NOTE: It is NOT broken for pure UNION queries and pure INTERSECT/EXCEPT
queries!!!
-) I had to add the field intersect_clause to some data structures
but did not find time to implement printfuncs for the new field.
This does NOT break the debug modes but when an Except/Intersect
is used the query debug output will be the already rewritten query.
-) Massive changes to the grammar rules for SELECT and INSERT statements
have been necessary (see comments in gram.y and documentation for
deatails) in order to be able to use mixed queries like
(SELECT ... UNION (SELECT ... EXCEPT SELECT)) INTERSECT SELECT...;
-) When using UNION/EXCEPT/INTERSECT you will get:
NOTICE: equal: "Don't know if nodes of type xxx are equal".
I did not have time to add comparsion support for all the needed nodes,
but the default behaviour of the function equal met my requirements.
I did not dare to supress this message!
That's the reason why the regression test for union will fail: These
messages are also included in the union.out file!
-) Somebody of you changed the union_planner() function for v6.4
(I copied the targetlist to new_tlist and that was removed and
replaced by a cleanup of the original targetlist). These chnages
violated some having queries executed against views so I changed
it back again. I did not have time to examine the differences between the
two versions but now it works :-)
If you want to find out, try the file queries/view_having.sql on
both versions and compare the results . Two queries won't produce a
correct result with your version.
regards
Stefan
Fixes a bug in the rule system that caused a crashing
backend when a join-view with calculated column is used
in subselect.
Modifies EXPLAIN to explain rewritten queries instead of
the plain SeqScan on a view. Rules can produce very deep
MORE
Jan.
columns of views at all (not only oid, cmin etc. too).
pgsql=> select cmin from pg_rules;
ERROR: system column cmin not available - pg_rules is a view
pgsql=> select * from pg_rules where pg_rules.oid = pg_class.oid;
ERROR: system column oid not available - pg_rules is a view
pgsql=>
Jan
for against a just updated CVS tree. It contains
Partial new rewrite system that handles subselects, view
aggregate columns, insert into select from view, updates
with set col = view-value and select rules restriction to
view definition.
Updates for rule/view backparsing utility functions to
handle subselects correct.
New system views pg_tables and pg_indexes (where you can
see the complete index definition in the latter one).
Enabling array references on query parameters.
Bugfix for functional index.
Little changes to system views pg_rules and pg_views.
The rule system isn't a release-stopper any longer.
But another stopper is that I don't know if the latest
changes to PL/pgSQL (not already in CVS) made it compile on
AIX. Still wait for some response from Dave.
Jan
patch is applied:
Rewrite rules on relation level work fine now.
Event qualifications on insert/update/delete rules work
fine now.
I added the new keyword OLD to reference the CURRENT
tuple. CURRENT will be removed in 6.5.
Update rules can reference NEW and OLD in the rule
qualification and the actions.
Insert/update/delete rules on views can be established to
let them behave like real tables.
For insert/update/delete rules multiple actions are
supported now. The actions can also be surrounded by
parantheses to make psql happy. Multiple actions are
required if update to a view requires updates to multiple
tables.
Regular users are permitted to create/drop rules on
tables they have RULE permissions for
(DefineQueryRewrite() is now able to get around the
access restrictions on pg_rewrite). This enables view
creation for regular users too. This required an extra
boolean parameter to pg_parse_and_plan() that tells to
set skipAcl on all rangetable entries of the resulting
queries. There is a new function
pg_exec_query_acl_override() that could be used by
backend utilities to use this facility.
All rule actions (not only views) inherit the permissions
of the event relations owner. Sample: User A creates
tables T1 and T2, creates rules that log
INSERT/UPDATE/DELETE on T1 in T2 (like in the regression
tests for rules I created) and grants ALL but RULE on T1
to user B. User B can now fully access T1 and the
logging happens in T2. But user B cannot access T2 at
all, only the rule actions can. And due to missing RULE
permissions on T1, user B cannot disable logging.
Rules on the attribute level are disabled (they don't
work properly and since regular users are now permitted
to create rules I decided to disable them).
Rules on select must have exactly one action that is a
select (so select rules must be a view definition).
UPDATE NEW/OLD rules are disabled (still broken, but
triggers can do it).
There are two new system views (pg_rule and pg_view) that
show the definition of the rules or views so the db admin
can see what the users do. They use two new functions
pg_get_ruledef() and pg_get_viewdef() that are builtins.
The functions pg_get_ruledef() and pg_get_viewdef() could
be used to implement rule and view support in pg_dump.
PostgreSQL is now the only database system I know, that
has rewrite rules on the query level. All others (where I
found a rule statement at all) use stored database
procedures or the like (triggers as we call them) for
active rules (as some call them).
Future of the rule system:
The now disabled parts of the rule system (attribute
level, multiple actions on select and update new stuff)
require a complete new rewrite handler from scratch. The
old one is too badly wired up.
After 6.4 I'll start to work on a new rewrite handler,
that fully supports the attribute level rules, multiple
actions on select and update new. This will be available
for 6.5 so we get full rewrite rule capabilities.
Jan
no longer returns buffer pointer, can be gotten from scan;
descriptor; bootstrap can create multi-key indexes;
pg_procname index now is multi-key index; oidint2, oidint4, oidname
are gone (must be removed from regression tests); use System Cache
rather than sequential scan in many places; heap_modifytuple no
longer takes buffer parameter; remove unused buffer parameter in
a few other functions; oid8 is not index-able; remove some use of
single-character variable names; cleanup Buffer variables usage
and scan descriptor looping; cleaned up allocation and freeing of
tuples; 18k lines of diff;
now. Here some tested features, (examples included in the patch):
1.1) Subselects in the having clause 1.2) Double nested subselects
1.3) Subselects used in the where clause and in the having clause
simultaneously 1.4) Union Selects using having 1.5) Indexes
on the base relations are used correctly 1.6) Unallowed Queries
are prevented (e.g. qualifications in the
having clause that belong to the where clause) 1.7) Insert
into as select
2) Queries using the having clause on view relations also work
but there are some restrictions:
2.1) Create View as Select ... Having ...; using base tables in
the select 2.1.1) The Query rewrite system:
2.1.2) Why are only simple queries allowed against a view from 2.1)
? 2.2) Select ... from testview1, testview2, ... having...; 3) Bug
in ExecMergeJoin ??
Regards Stefan
sequential scans! (I think it will also work with hash, index, etc
but I did not check it out! I made some High level changes which
should work for all access methods, but maybe I'm wrong. Please
let me know.)
Now it is possible to make queries like:
select s.sname, max(p.pid), min(p.pid) from part p, supplier s
where s.sid=p.sid group by s.sname having max(pid)=6 and min(pid)=1
or avg(pid)=4;
Having does not work yet for queries that contain a subselect
statement in the Having clause, I'll try to fix this in the next
days.
If there are some bugs, please let me know, I'll start to read the
mailinglists now!
Now here is the patch against the original 6.3 version (no snapshot!!):
Stefan
seems that my last post didn't make it through. That's good
since the diff itself didn't covered the renaming of
pg_user.h to pg_shadow.h and it's new content.
Here it's again. The complete regression test passwd with
only some float diffs. createuser and destroyuser work.
pg_shadow cannot be read by ordinary user.
below is the patch to have views to override the permission
checks for the accessed tables. Now we can do the following:
CREATE VIEW db_user AS SELECT
usename,
usesysid,
usecreatedb,
usetrace,
usecatupd,
'**********'::text as passwd,
valuntil
FROM pg_user;
REVOKE ALL ON pg_user FROM public;
REVOKE ALL ON db_user FROM public;
GRANT SELECT ON db_user TO public;
Tom Lane
Bruce Momjian
Jan Wieck
Peter Eisentraut
Marc G. Fournier
Vadim B. Mikheev
Thomas G. Lockhart
Bryan Henderson