key call sites are changed, but most called functions are still oldstyle.
An exception is that the PL managers are updated (so, for example, NULL
handling now behaves as expected in plperl and plpgsql functions).
NOTE initdb is forced due to added column in pg_proc.
think that both sides of indexqual look like index keys. An example is
create table inside (f1 float8 primary key);
create table outside (g1 float8, g2 float8);
select * from inside,outside where f1 = atan2(g1+1, g2);
ERROR: ExecInitIndexScan: both left and right ops are rel-vars
(note that failure is potentially platform-dependent). Solution is a
cleanup I had had in mind to make anyway: functional index keys should
be represented as Var nodes in the fixed indexqual, just like regular
index keys.
cases where joinclauses were present but some joins have to be made
by cartesian-product join anyway. An example is
SELECT * FROM a,b,c WHERE (a.f1 + b.f2 + c.f3) = 0;
Even though all the rels have joinclauses, we must join two of them
in cartesian style before we can use the join clause...
(SELECT FROM table*). Cause was reference to 'eref' field of an RTE,
which is null in an RTE loaded from a stored rule parsetree. There
wasn't any good reason to be touching the refname anyway...
table for an average of NTUP_PER_BUCKET tuples/bucket, but cost_hashjoin
was assuming a target load of one tuple/bucket. This was causing a
noticeable underestimate of hashjoin costs.
(LIKE and regexp matches). These are not yet referenced in pg_operator,
so by default the system will continue to use eqsel/neqsel.
Also, tweak convert_to_scalar() logic so that common prefixes of strings
are stripped off, allowing better accuracy when all strings in a table
share a common prefix.
contained a sub-SELECT nested within an AND/OR tree that cnfify()
thought it should rearrange. Same physical sub-SELECT node could
end up linked into multiple places in resulting expression tree.
This is harmless for most node types, but not for SubLink.
Repair bug by making physical copies of subexpressions that get
logically duplicated by cnfify(). Also, tweak the heuristic that
decides whether it's a good idea to do cnfify() --- we don't really
want that to happen when it would cause multiple copies of a subselect
to be generated, I think.
WHERE in a place where it can be part of a nestloop inner indexqual.
As the code stood, it put the same physical sub-Plan node into both
indxqual and indxqualorig of the IndexScan plan node. That confused
later processing in the optimizer (which expected that tracing the
subPlan list would visit each subplan node exactly once), and would
probably have blown up in the executor if the planner hadn't choked first.
Fix by making the 'fixed' indexqual be a complete deep copy of the
original indexqual, rather than trying to share nodes below the topmost
operator node. This had further ramifications though, because we were
making the aforesaid list of sub-Plan nodes during SS_process_sublinks
which is run before construction of the 'fixed' indexqual, meaning that
the copy of the sub-Plan didn't show up in that list. Fix by rearranging
logic so that the sub-Plan list is built by the final set_plan_references
pass, not in SS_process_sublinks. This may sound like a mess, but it's
actually a good deal cleaner now than it was before, because we are no
longer dependent on the assumption that planning will never make a copy
of a sub-Plan node.
to next integer. Previously, if selectivity was small, we could compute
very tiny scan cost on the basis of estimating that only 0.001 tuple
would be fetched, which is silly. This naturally led to some rather
silly plans...
to avoid undue sensitivity to roundoff error, believe that a zero
or slightly negative range estimate should represent a small
positive selectivity, rather than falling back on a generic default
estimate.
use a default value that's fairly small. We were generating a result
of about 0.1, but I think 0.01 is probably better --- want to encourage
use of an indexscan in this situation.
costs using the inner path's parent->rows count as the number of tuples
processed per inner scan iteration. This is wrong when we are using an
inner indexscan with indexquals based on join clauses, because the rows
count in a Relation node reflects the selectivity of the restriction
clauses for that rel only. Upshot was that if join clause was very
selective, we'd drastically overestimate the true cost of the join.
Fix is to calculate correct output-rows estimate for an inner indexscan
when the IndexPath node is created and save it in the path node.
Change of path node doesn't require initdb, since path nodes don't
appear in saved rules.
to simplify constant expressions and expand SubLink nodes into SubPlans
is done in a separate routine subquery_planner() that calls union_planner().
We formerly did most of this work in query_planner(), but that's the
wrong place because it may never see the real targetlist. Splitting
union_planner into two routines also allows us to avoid redundant work
when union_planner is invoked recursively for UNION and inheritance
cases. Upshot is that it is now possible to do something like
select float8(count(*)) / (select count(*) from int4_tbl) from int4_tbl
group by f1;
which has never worked before.
running gcc and HP's cc with warnings cranked way up. Signed vs unsigned
comparisons, routines declared static and then defined not-static,
that kind of thing. Tedious, but perhaps useful...
Implement TIME WITH TIME ZONE type (timetz internal type).
Remap length() for character strings to CHAR_LENGTH() for SQL92
and to remove the ambiguity with geometric length() functions.
Keep length() for character strings for backward compatibility.
Shrink stored views by removing internal column name list from visible rte.
Implement min(), max() for time and timetz data types.
Implement conversion of TIME to INTERVAL.
Implement abs(), mod(), fac() for the int8 data type.
Rename some math functions to generic names:
round(), sqrt(), cbrt(), pow(), etc.
Rename NUMERIC power() function to pow().
Fix int2 factorial to calculate result in int4.
Enhance the Oracle compatibility function translate() to work with string
arguments (from Edwin Ramirez).
Modify pg_proc system table to remove OID holes.
that the inputs to a given operator can be recursively simplified to
constants, it was evaluating the operator using the op's *original*
(unsimplified) arg list, so that any subexpressions had to be evaluated
again. A constant subexpression at depth N got evaluated N times.
Probably not very important in practical situations, but it made us look
real slow in MySQL's 'crashme' test...
subplan: do it if subplan has subplans itself, and always do it if the
subplan is an indexscan. (I originally set it to materialize an indexscan
only if the indexqual is fairly selective, but I dunno what I was
thinking ... an unselective indexscan is still expensive ...)
portion of the query result that will be retrieved. As far as I could
tell, the consensus was that we should let the planner do the best it
can with a LIMIT query, and require the user to add ORDER BY if he
wants consistent results from different LIMIT values.
represent the result of a binary-compatible type coercion. At runtime
it just evaluates its argument --- but during type resolution, exprType
will pick up the output type of the RelabelType node instead of the type
of the argument. This solves some longstanding problems with dropped
type coercions, an example being 'select now()::abstime::int4' which
used to produce date-formatted output, not an integer, because the
coercion to int4 was dropped on the floor.
selectivity functions and make the r-tree operators use them. The
estimation functions themselves are just stubs, unfortunately, but
perhaps someday someone will make them compute realistic estimates.
Change pg_am so that the optimizer can reliably tell the difference
between ordered and unordered indexes --- before it would think that
an r-tree index can be scanned in '<<' order, which is not right AFAIK.
Repair broken negator links for network_sup and related ops.
Initdb forced. This might be my last initdb force for 7.0 ... hope so
anyway ...
accesses versus sequential accesses, a (very crude) estimate of the
effects of caching on random page accesses, and cost to evaluate WHERE-
clause expressions. Export critical parameters for this model as SET
variables. Also, create SET variables for the planner's enable flags
(enable_seqscan, enable_indexscan, etc) so that these can be controlled
more conveniently than via PGOPTIONS.
Planner now estimates both startup cost (cost before retrieving
first tuple) and total cost of each path, so it can optimize queries
with LIMIT on a reasonable basis by interpolating between these costs.
Same facility is a win for EXISTS(...) subqueries and some other cases.
Redesign pathkey representation to achieve a major speedup in planning
(I saw as much as 5X on a 10-way join); also minor changes in planner
to reduce memory consumption by recycling discarded Path nodes and
not constructing unnecessary lists.
Minor cleanups to display more-plausible costs in some cases in
EXPLAIN output.
Initdb forced by change in interface to index cost estimation
functions.
SELECT a FROM t1 tx (a);
Allow join syntax, including queries like
SELECT * FROM t1 NATURAL JOIN t2;
Update RTE structure to hold column aliases in an Attr structure.
fields in JoinPaths --- turns out that we do need that after all :-(.
Also, rearrange planner so that only one RelOptInfo is created for a
particular set of joined base relations, no matter how many different
subsets of relations it can be created from. This saves memory and
processing time compared to the old method of making a bunch of RelOptInfos
and then removing the duplicates. Clean up the jointree iteration logic;
not sure if it's better, but I sure find it more readable and plausible
now, particularly for the case of 'bushy plans'.
nonoverlap_sets() and is_subset() to list.c, where they should have lived
to begin with, and rename to nonoverlap_setsi and is_subseti since they
only work on integer lists.
extracting from an AND subclause just those opclauses that are relevant
for a particular index. For example, we can now consider using an index
on x to process WHERE (x = 1 AND y = 2) OR (x = 2 AND y = 4) OR ...
2-Oct-98 or TODO.detail/cnfify) to decide whether we want to reduce
WHERE clause to CNF form, DNF form, or neither. This is a HUGE win.
The heuristic conditions could probably still use a little tweaking to
make sure we don't pick CNF when DNF would be better, or vice versa,
but the risk of exponential explosion in cnfify() is gone. I was able
to run ten-thousand-AND-subclause queries through the planner in a
reasonable amount of time.
SELECT DISTINCT ON (expr [, expr ...]) targetlist ...
and there is a check to make sure that the user didn't specify an ORDER BY
that's incompatible with the DISTINCT operation.
Reimplement nodeUnique and nodeGroup to use the proper datatype-specific
equality function for each column being compared --- they used to do
bitwise comparisons or convert the data to text strings and strcmp().
(To add insult to injury, they'd look up the conversion functions once
for each tuple...) Parse/plan representation of DISTINCT is now a list
of SortClause nodes.
initdb forced by querytree change...
(ie, WHERE x > lowbound AND x < highbound). It's not very bright yet
but it does something useful. Also, rename intltsel/intgtsel to
scalarltsel/scalargtsel to reflect usage better. Extend convert_to_scalar
to do something a little bit useful with string data types. Still need
to make it do something with date/time datatypes, but I'll wait for
Thomas's datetime unification dust to settle first. Eventually the
routine ought not have any type-specific knowledge at all; it ought to
be calling a type-dependent routine found via a pg_type column; but
that's a task for another day.
pghackers discussion of 5-Jan-2000. The amopselect and amopnpages
estimators are gone, and in their place is a per-AM amcostestimate
procedure (linked to from pg_am, not pg_amop).
Make all system indexes unique.
Make all cache loads use system indexes.
Rename *rel to *relid in inheritance tables.
Rename cache names to be clearer.
returns a list of RelOptInfos, eliminating the need for static state
in index_info. That static state was a direct cause of coredumps; if
anything decided to elog(ERROR) partway through an index_info search of
pg_index, the next query would try to close a scan pointer that was
pointing at no-longer-valid memory. Another example of the reasons to
avoid static state variables...
subselects can only appear on the righthand side of a binary operator.
That's still true for quantified predicates like x = ANY (SELECT ...),
but a subselect that delivers a single result can now appear anywhere
in an expression. This is implemented by changing EXPR_SUBLINK sublinks
to represent just the (SELECT ...) expression, without any 'left hand
side' or combining operator --- so they're now more like EXISTS_SUBLINK.
To handle the case of '(x, y, z) = (SELECT ...)', I added a new sublink
type MULTIEXPR_SUBLINK, which acts just like EXPR_SUBLINK used to.
But the grammar will only generate one for a multiple-left-hand-side
row expression.
In particular, don't bother to look up type information for attributes
where we're not actually going to use it, and avoid copying entire tlist
structure when it's not necessary.
mentioned in FROM but not elsewhere in the query: such tables should be
joined over anyway. Aside from being more standards-compliant, this allows
removal of some very ugly hacks for COUNT(*) processing. Also, allow
HAVING clause without aggregate functions, since SQL does. Clean up
CREATE RULE statement-list syntax the same way Bruce just fixed the
main stmtmulti production.
CAUTION: addition of a field to RangeTblEntry nodes breaks stored rules;
you will have to initdb if you have any rules.
in the Expr nodes they produce. This fixes a few cases of errors like
'typeidTypeRelid: Invalid type - oid = 0' caused by calling parser-related
routines on expression trees that have already been processed by planner-
related routines.
Frankpitt, plus some improvements from yours truly. The simplifier depends
on the proiscachable field of pg_proc to tell it whether a function is
safe to pre-evaluate --- things like nextval() are not, for example.
Update pg_proc.h to contain reasonable cacheability information; as of
6.5.* hardly any functions were marked cacheable. I may have erred too
far in the other direction; see recent mail to pghackers for more info.
This update does not force an initdb, exactly, but you won't see much
benefit from the simplifier until you do one.
additional argument specifying the kind of lock to acquire/release (or
'NoLock' to do no lock processing). Ensure that all relations are locked
with some appropriate lock level before being examined --- this ensures
that relevant shared-inval messages have been processed and should prevent
problems caused by concurrent VACUUM. Fix several bugs having to do with
mismatched increment/decrement of relation ref count and mismatched
heap_open/close (which amounts to the same thing). A bogus ref count on
a relation doesn't matter much *unless* a SI Inval message happens to
arrive at the wrong time, which is probably why we got away with this
sloppiness for so long. Repair missing grab of AccessExclusiveLock in
DROP TABLE, ALTER/RENAME TABLE, etc, as noted by Hiroshi.
Recommend 'make clean all' after pulling this update; I modified the
Relation struct layout slightly.
Will post further discussion to pghackers list shortly.
conditions. There are some pretty bogus heuristics in prepqual.c that
try to decide whether to output CNF or DNF format; they need to be replaced,
likely. Right now the code is probably too willing to choose DNF form,
which might hurt performance in some cases that used to work OK.
But at least we have a foundation to build on.
in or_normalize, remove detection of duplicate subexpressions (since it's
highly unlikely to be worth the amount of time it takes), and introduce
a dnfify() entry point so that unintelligible backwards logic in UNION
processing can be eliminated. This is just an intermediate step ---
next thing is to look at not forcing the qual into CNF form when it would
be better off in DNF form.
was rejecting negative attnums as bogus, which of course they are not.
Add code to get_attdisbursion to produce a useful value for OID attribute,
since VACUUM does not store stats for system attributes.
Also, repair bug that's been in eqjoinsel for a long time: it was taking
the max of the two columns' disbursions, whereas it should use the min.
space consumption in pull_args, and avoid doing the full CNF transform on
operands of operator clauses, where it's really not particularly helpful.
This answers the TODO item about large numbers of OR clauses, at least
partially. I was able to do a ten-thousand-OR-clause query with about
20Mb memory consumption ... it took an obscenely long time, but it worked...
Most parts of the planner should ignore, or indeed never even see, uplevel
Vars because they will be or have been replaced by Params. There were a
couple of places that got it wrong though, probably my fault from recent
changes...
documented intepretation of the lefthand and oper fields. Fix a number of
obscure problems while at it --- for example, the old code failed if the parser
decided to insert a type-coercion function just below the operator of a
SubLink.
CAUTION: this will break stored rules that contain subplans. You may
need to initdb.
and fix_opids processing to a single recursive pass over the plan tree
executed at the very tail end of planning, rather than haphazardly here
and there at different places. Now that tlist Vars do not get modified
until the very end, it's possible to get rid of the klugy var_equal and
match_varid partial-matching routines, and just use plain equal()
throughout the optimizer. This is a step towards allowing merge and
hash joins to be done on expressions instead of only Vars ...
sort order down into planner, instead of handling it only at the very top
level of the planner. This fixes many things. An explicit sort is now
avoided if there is a cheaper alternative (typically an indexscan) not
only for ORDER BY, but also for the internal sort of GROUP BY. It works
even when there is no other reason (such as a WHERE condition) to consider
the indexscan. It works for indexes on functions. It works for indexes
on functions, backwards. It's just so cool...
CAUTION: I have changed the representation of SortClause nodes, therefore
THIS UPDATE BREAKS STORED RULES. You will need to initdb.
above a Sort or Materialize node. As far as I can tell, the only place
that actually needed that was set_tlist_references, which was being lazy
about checking to see if it had a noname node to fix or not...
store all ordering information in pathkeys lists (which are now lists of
lists of PathKeyItem nodes, not just lists of lists of vars). This was
a big win --- the code is smaller and IMHO more understandable than it
was, even though it handles more cases. I believe the node changes will
not force an initdb for anyone; planner nodes don't show up in stored
rules.
commuted (ie, the index var appears on the right). These are now handled
the same way as merge and hash join quals that need to be commuted: the
actual reversing of the clause only happens if we actually choose the path
and generate a plan from it. Furthermore, the clause is only reversed in
the 'indexqual' field of the plan, not in the 'indxqualorig' field. This
allows the clause to still be recognized and removed from qpquals of upper
level join plans. Also, simplify and generalize match_clause_to_indexkey;
now it recognizes binary-compatible indexes for join as well as restriction
clauses.
work under a wider range of scenarios than it did --- it formerly did not
handle a multi-pass inner scan, nor cases in which the inner scan's
indxqualorig or non-index qual contained outer var references. I am not
sure that these limitations could be hit in the existing optimizer, but
they need to be fixed for future expansion.
> >
> > was implemented by Jan Wieck.
> > His work is for ascending order cases.
> >
> > Here is a patch to prevent sorting also in descending
> > order cases.
> > Because I had already changed _bt_first() to position
> > backward correctly before v6.5,this patch would work.
> >
Hiroshi Inoue
Inoue@tpf.co.jp
multi-scan indexscan plans; it tried to use the same table-to-index
attribute mapping for all the scans, even if they used different indexes.
It would klugily work as long as OR indexquals never used multikey indexes,
but that's not likely to hold up much longer...
to go along with expression_tree_walker. (_walker is not suitable for
routines that need to alter the tree structure significantly.) Other minor
cleanups in clauses.c.
hashjoinable clause, not one path for a randomly-chosen element of each
set of clauses with the same join operator. That is, if you wrote
SELECT ... WHERE t1.f1 = t2.f2 and t1.f3 = t2.f4,
and both '=' ops were the same opcode (say, all four fields are int4),
then the system would either consider hashing on f1=f2 or on f3=f4,
but it would *not* consider both possibilities. Boo hiss.
Also, revise estimation of hashjoin costs to include a penalty when the
inner join var has a high disbursion --- ie, the most common value is
pretty common. This tends to lead to badly skewed hash bucket occupancy
and way more comparisons than you'd expect on average.
I imagine that the cost calculation still needs tweaking, but at least
it generates a more reasonable plan than before on George Young's example.
so that Case works in WHERE join clauses. Temporary patch --- this routine
is one of many that ought to be changed to use centralized expression-tree-
walking logic.
rels that the inner path needs to join to, but it was only checking for
the first one. Failure could only have been observed with an OR-clause
that mentions 3 or more tables, and then only if the bogus path was
actually selected as cheapest ...
optimizer rather than parser. This has many advantages, such as not
getting fooled by chance uses of operator names ~ and ~~ (the operators
are identified by OID now), and not creating useless comparison operations
in contexts where the comparisons will not actually be used as indexquals.
The new code also recognizes exact-match LIKE and regex patterns, and
produces an = indexqual instead of >= and <=.
This change does NOT fix the problem with non-ASCII locales: the code
still doesn't know how to generate an upper bound indexqual for non-ASCII
collation order. But it's no worse than before, just the same deficiency
in a different place...
Also, dike out loc_restrictinfo fields in Plan nodes. These were doing
nothing useful in the absence of 'expensive functions' optimization,
and they took a considerable amount of processing to fill in.
The only place it was being used was as temporary storage in indxpath.c,
and the logic was wrong: the same restrictinfo node could get chosen to
carry the info for two different joins. Right fix is to return a second
list of unjoined-relids parallel to the list of clause groups.
identified by Hiroshi (incorrect cost attributed to OR clauses
after multiple passes through set_rest_selec()). I think the code
was trying to allow selectivities of OR subclauses to be passed in
from outside, but noplace was actually passing any useful data, and
set_rest_selec() was passing wrong data.
Restructure representation of "indexqual" in IndexPath nodes so that
it is the same as for indxqual in completed IndexScan nodes: namely,
a toplevel list with an entry for each pass of the index scan, having
sublists that are implicitly-ANDed index qual conditions for that pass.
You don't want to know what the old representation was :-(
Improve documentation of OR-clause indexscan functions.
Remove useless 'notclause' field from RestrictInfo nodes. (This might
force an initdb for anyone who has stored rules containing RestrictInfos,
but I do not think that RestrictInfo ever appears in completed plans.)
will gradually replace all of the boilerplate tree-walk-recursion code that
currently exists in O(N) slightly different forms in N subroutines.
I've had it with adding missing cases to these subroutines...
tlist and qual are NULL. It ought to handle these the same as the cases
where tlist contains only constant expressions, ie, be willing to generate
a Result-node plan. This did not use to matter, but it does now because
union_planner will flatten the tlist when aggregates are present. Thus,
'select count(1) from table' now causes query_planner to be given a null
tlist, and to duplicate 6.4's behavior we need it to give back a Result
plan rather than refusing the query. 6.4 was arguably doing the Wrong
Thing for this query, but I'm not going to open a semantics issue right
before 6.5 release ... can revisit that problem later.
returned NULL, which it will do in some cases where an elog(ERROR) would
probably be more appropriate. For the moment, generate a not-very-
informative error message rather than proceeding to certain coredump.
Probably ought to think about making query_planner elog instead of
returning NULL, but this is at least a safe change for now.
redundant) SearchSysCache searches per table column in an INSERT, which
accounted for a good percentage of the CPU time for INSERT ... VALUES().
Now it only does two searches in the typical case.
remove optimizer's arbitrary limit on how large a join it will use hashing
for. (The limit was too large to prevent the problems we'd been seeing,
anyway...)
lists are now plain old garden-variety Lists, allocated with palloc,
rather than specialized expansible-array data allocated with malloc.
This substantially simplifies their handling and eliminates several
sources of memory leakage.
Several basic types of erroneous queries (syntax error, attempt to
insert a duplicate key into a unique index) now demonstrably leak
zero bytes per query.
change functionality, but makes the code more ANSI C'ish.
My AIX xlc compiler barfs on all of these. Can someone please
review and apply to current.
<<port.patch>>
Thanks
Andreas
_copyResult didn't copy subPlan structure completely. _copyAgg is still
busted, apparently because of changes from EXCEPT/INTERSECT patch
(get_agg_tlist_references is no longer sufficient to find all aggregates).
No time to look at that tonight, however.
so remove them from MergeJoin node. Hack together a partial
solution for commuted mergejoin operators --- yesterday
a mergejoin int4 = int8 would crash if the planner decided to
commute it, today it works. The planner's representation of
mergejoins really needs a rewrite though.
Also, further testing of mergejoin ops in opr_sanity regress test.
Ok. I made patches replacing all of "#if FALSE" or "#if 0" to "#ifdef
NOT_USED" for current. I have tested these patches in that the
postgres binaries are identical.
INTERSECT and EXCEPT is available for postgresql-v6.4!
The patch against v6.4 is included at the end of the current text
(in uuencoded form!)
I also included the text of my Master's Thesis. (a postscript
version). I hope that you find something of it useful and would be
happy if parts of it find their way into the PostgreSQL documentation
project (If so, tell me, then I send the sources of the document!)
The contents of the document are:
-) The first chapter might be of less interest as it gives only an
overview on SQL.
-) The second chapter gives a description on much of PostgreSQL's
features (like user defined types etc. and how to use these features)
-) The third chapter starts with an overview of PostgreSQL's internal
structure with focus on the stages a query has to pass (i.e. parser,
planner/optimizer, executor). Then a detailed description of the
implementation of the Having clause and the Intersect/Except logic is
given.
Originally I worked on v6.3.2 but never found time enough to prepare
and post a patch. Now I applied the changes to v6.4 to get Intersect
and Except working with the new version. Chapter 3 of my documentation
deals with the changes against v6.3.2, so keep that in mind when
comparing the parts of the code printed there with the patched sources
of v6.4.
Here are some remarks on the patch. There are some things that have
still to be done but at the moment I don't have time to do them
myself. (I'm doing my military service at the moment) Sorry for that
:-(
-) I used a rewrite technique for the implementation of the Except/Intersect
logic which rewrites the query to a semantically equivalent query before
it is handed to the rewrite system (for views, rules etc.), planner,
executor etc.
-) In v6.3.2 the types of the attributes of two select statements
connected by the UNION keyword had to match 100%. In v6.4 the types
only need to be familiar (i.e. int and float can be mixed). Since this
feature did not exist when I worked on Intersect/Except it
does not work correctly for Except/Intersect queries WHEN USED IN
COMBINATION WITH UNIONS! (i.e. sometimes the wrong type is used for the
resulting table. This is because until now the types of the attributes of
the first select statement have been used for the resulting table.
When Intersects and/or Excepts are used in combination with Unions it
might happen, that the first select statement of the original query
appears at another position in the query which will be executed. The reason
for this is the technique used for the implementation of
Except/Intersect which does a query rewrite!)
NOTE: It is NOT broken for pure UNION queries and pure INTERSECT/EXCEPT
queries!!!
-) I had to add the field intersect_clause to some data structures
but did not find time to implement printfuncs for the new field.
This does NOT break the debug modes but when an Except/Intersect
is used the query debug output will be the already rewritten query.
-) Massive changes to the grammar rules for SELECT and INSERT statements
have been necessary (see comments in gram.y and documentation for
deatails) in order to be able to use mixed queries like
(SELECT ... UNION (SELECT ... EXCEPT SELECT)) INTERSECT SELECT...;
-) When using UNION/EXCEPT/INTERSECT you will get:
NOTICE: equal: "Don't know if nodes of type xxx are equal".
I did not have time to add comparsion support for all the needed nodes,
but the default behaviour of the function equal met my requirements.
I did not dare to supress this message!
That's the reason why the regression test for union will fail: These
messages are also included in the union.out file!
-) Somebody of you changed the union_planner() function for v6.4
(I copied the targetlist to new_tlist and that was removed and
replaced by a cleanup of the original targetlist). These chnages
violated some having queries executed against views so I changed
it back again. I did not have time to examine the differences between the
two versions but now it works :-)
If you want to find out, try the file queries/view_having.sql on
both versions and compare the results . Two queries won't produce a
correct result with your version.
regards
Stefan
I put some extra checks to make sure a query was a good candidate for
rewrite into a UNION. Besides the existing checks:
1. Make sure the AND/OR tree was rectangular. ( i.e. 3 X 4 or 10 X
3)
2. Only one table.
3. Must have an AND dimension.
4. At least 9 OP expressions total
Also cleaned up and commented.
[AC_MSG_RESULT(yes) AC_DEFINE(HAVE_LONG_INT_64)],
this line produces something like:
echo "$ac_t""yes" 1>&6 cat >> confdefs.h <<\EOF
and would append garbage "yes cat" to confdefs.h. Of course the
result confdefs.h is not syntactically correct therefore following
tests using confdefs.h would all fail. To avoid the problem, we
could switch the order of AC_MSG_RESULT and AC_DEFINE (see attached
patch). This happend on my LinuxPPC box.
Tatsuo Ishii t-ishii@sra.co.jp
patch is applied:
Rewrite rules on relation level work fine now.
Event qualifications on insert/update/delete rules work
fine now.
I added the new keyword OLD to reference the CURRENT
tuple. CURRENT will be removed in 6.5.
Update rules can reference NEW and OLD in the rule
qualification and the actions.
Insert/update/delete rules on views can be established to
let them behave like real tables.
For insert/update/delete rules multiple actions are
supported now. The actions can also be surrounded by
parantheses to make psql happy. Multiple actions are
required if update to a view requires updates to multiple
tables.
Regular users are permitted to create/drop rules on
tables they have RULE permissions for
(DefineQueryRewrite() is now able to get around the
access restrictions on pg_rewrite). This enables view
creation for regular users too. This required an extra
boolean parameter to pg_parse_and_plan() that tells to
set skipAcl on all rangetable entries of the resulting
queries. There is a new function
pg_exec_query_acl_override() that could be used by
backend utilities to use this facility.
All rule actions (not only views) inherit the permissions
of the event relations owner. Sample: User A creates
tables T1 and T2, creates rules that log
INSERT/UPDATE/DELETE on T1 in T2 (like in the regression
tests for rules I created) and grants ALL but RULE on T1
to user B. User B can now fully access T1 and the
logging happens in T2. But user B cannot access T2 at
all, only the rule actions can. And due to missing RULE
permissions on T1, user B cannot disable logging.
Rules on the attribute level are disabled (they don't
work properly and since regular users are now permitted
to create rules I decided to disable them).
Rules on select must have exactly one action that is a
select (so select rules must be a view definition).
UPDATE NEW/OLD rules are disabled (still broken, but
triggers can do it).
There are two new system views (pg_rule and pg_view) that
show the definition of the rules or views so the db admin
can see what the users do. They use two new functions
pg_get_ruledef() and pg_get_viewdef() that are builtins.
The functions pg_get_ruledef() and pg_get_viewdef() could
be used to implement rule and view support in pg_dump.
PostgreSQL is now the only database system I know, that
has rewrite rules on the query level. All others (where I
found a rule statement at all) use stored database
procedures or the like (triggers as we call them) for
active rules (as some call them).
Future of the rule system:
The now disabled parts of the rule system (attribute
level, multiple actions on select and update new stuff)
require a complete new rewrite handler from scratch. The
old one is too badly wired up.
After 6.4 I'll start to work on a new rewrite handler,
that fully supports the attribute level rules, multiple
actions on select and update new. This will be available
for 6.5 so we get full rewrite rule capabilities.
Jan
no longer returns buffer pointer, can be gotten from scan;
descriptor; bootstrap can create multi-key indexes;
pg_procname index now is multi-key index; oidint2, oidint4, oidname
are gone (must be removed from regression tests); use System Cache
rather than sequential scan in many places; heap_modifytuple no
longer takes buffer parameter; remove unused buffer parameter in
a few other functions; oid8 is not index-able; remove some use of
single-character variable names; cleanup Buffer variables usage
and scan descriptor looping; cleaned up allocation and freeing of
tuples; 18k lines of diff;
indices for restriction clauses containing a constant.
Note that if an index does not match directly (usually because the types
on both side of the clause don't match), and if a binary-compatible index
is identified, then the operator function will be replaced by a new
one. Should not be a problem, but be sure that if types are listed as
being binary compatible (in parse_coerce.h) then the comparison functions
are also binary-compatible, giving equivalent results.
Everyone using an [NOT] EXISTS subquery will have noticed that
already.
The bug is in "subselect.c" in the function "SS_process_sublinks()".
Here the whole function as it *SHOULD BE*:
Stephan
now. Here some tested features, (examples included in the patch):
1.1) Subselects in the having clause 1.2) Double nested subselects
1.3) Subselects used in the where clause and in the having clause
simultaneously 1.4) Union Selects using having 1.5) Indexes
on the base relations are used correctly 1.6) Unallowed Queries
are prevented (e.g. qualifications in the
having clause that belong to the where clause) 1.7) Insert
into as select
2) Queries using the having clause on view relations also work
but there are some restrictions:
2.1) Create View as Select ... Having ...; using base tables in
the select 2.1.1) The Query rewrite system:
2.1.2) Why are only simple queries allowed against a view from 2.1)
? 2.2) Select ... from testview1, testview2, ... having...; 3) Bug
in ExecMergeJoin ??
Regards Stefan
1. Removes the unnecessary "#define AbcRegProcedure 123"'s from
pg_proc.h.
2. Changes those #defines to use the names already defined in
fmgr.h.
3. Forces the make of fmgr.h in backend/Makefile instead of having
it
made as a dependency in access/common/Makefile *hack*hack*hack*
4. Rearranged the #includes to a less helter-skelter arrangement,
also
changing <file.h> to "file.h" to signify a non-system header.
5. Removed "pg_proc.h" from files where its only purpose was for
the
#defines removed in item #1.
6. Added "fmgr.h" to each file changed for completeness sake.
Turns out that #6 was not necessary for some files because fmgr.h
was being included in a roundabout way SIX levels deep by the first
include.
"access/genam.h"
->"access/relscan.h"
->"utils/rel.h"
->"access/strat.h"
->"access/skey.h"
->"fmgr.h"
So adding fmgr.h really didn't add anything to the compile, hopefully
just made it clearer to the programmer.
S Darren.
Attached you'll find a (big) patch that fixes make dep and make
depend in all Makefiles where I found it to be appropriate.
It also removes the dependency in Makefile.global for NAMEDATALEN
and OIDNAMELEN by making backend/catalog/genbki.sh and bin/initdb/initdb.sh
a little smarter.
This no longer requires initdb.sh that is turned into initdb with
a sed script when installing Postgres, hence initdb.sh should be
renamed to initdb (after the patch has been applied :-) )
This patch is against the 6.3 sources, as it took a while to
complete.
Please review and apply,
Cheers,
Jeroen van Vianen
yyerror ones from bison. It also includes a few 'enhancements' to
the C programming style (which are, of course, personal).
The other patch removes the compilation of backend/lib/qsort.c, as
qsort() is a standard function in stdlib.h and can be used any
where else (and it is). It was only used in
backend/optimizer/geqo/geqo_pool.c, backend/optimizer/path/predmig.c,
and backend/storage/page/bufpage.c
> > Some or all of these changes might not be appropriate for v6.3,
since we > > are in beta testing and since they do not affect the
current functionality. > > For those cases, how about submitting
patches based on the final v6.3 > > release?
There's more to come. Please review these patches. I ran the
regression tests and they only failed where this was expected
(random, geo, etc).
Cheers,
Jeroen
sequential scans! (I think it will also work with hash, index, etc
but I did not check it out! I made some High level changes which
should work for all access methods, but maybe I'm wrong. Please
let me know.)
Now it is possible to make queries like:
select s.sname, max(p.pid), min(p.pid) from part p, supplier s
where s.sid=p.sid group by s.sname having max(pid)=6 and min(pid)=1
or avg(pid)=4;
Having does not work yet for queries that contain a subselect
statement in the Having clause, I'll try to fix this in the next
days.
If there are some bugs, please let me know, I'll start to read the
mailinglists now!
Now here is the patch against the original 6.3 version (no snapshot!!):
Stefan
of some global variables to support subselects and calls union_planner().
Calls to SS_replace_correlation_vars() and SS_process_sublinks() in
query_planner() before planning.
Get rid of #ifdef INDEXSCAN_PATCH in createplan.c.
varchar length.
Cleans up code so attlen is always length.
Removed varchar() hack added earlier.
Will fix bug in selecting varchar() fields, and varchar() can be
variable length.
Patch by: wieck@sapserv.debis.de (Jan Wieck)
One of the design rules of PostgreSQL is extensibility. And
to follow this rule means (at least for me) that there should
not only be a builtin PL. Instead I would prefer a defined
interface for PL implemetations.
Pass List* of Aggregs into executor, and create needed array there.
No longer need to double-processs Aggregs with second copy in Query.
Fix crash when doing:
select sum(x+1) from test where 1 > 0;
==========================================
What follows is a set of diffs that cleans up the usage of BLCKSZ.
As a side effect, the person compiling the code can change the
value of BLCKSZ _at_their_own_risk_. By that, I mean that I've
tried it here at 4096 and 16384 with no ill-effects. A value
of 4096 _shouldn't_ affect much as far as the kernel/file system
goes, but making it bigger than 8192 can have severe consequences
if you don't know what you're doing. 16394 worked for me, _BUT_
when I went to 32768 and did an initdb, the SCSI driver broke and
the partition that I was running under went to hell in a hand
basket. Had to reboot and do a good bit of fsck'ing to fix things up.
The patch can be safely applied though. Just leave BLCKSZ = 8192
and everything is as before. It basically only cleans up all of the
references to BLCKSZ in the code.
If this patch is applied, a comment in the config.h file though above
the BLCKSZ define with warning about monkeying around with it would
be a good idea.
Darren darrenk@insightdist.com
(Also cleans up some of the #includes in files referencing BLCKSZ.)
==========================================
Makefile.global.
End result, if all goes well, should allow for much easier porting, since
there will no longer be a concept of a "port". Most, if not everything,
*should* be determined by configure, or by the compiler itself. Still
work to be done though :)
Subject: [PORTS] Patches for Irix 6.4
I have worked out how to compile PostgreSQL on Irix 6.4 using the -n32 compiler
mode and version 7.1 of the C compiler. (The n32 compiler use 32 bits
addressing,
but allows access to all the instructions in the MIPS4 instruction set.)
There were several problems:
1) The ld command is not referenced as a macro in all the Makefiles. On
this platform, you have to include -n32 on all the ld commands. Makefiles
were changed as needed.
3) Lots of warnings are generated from the compiler. Since the regression
tests worked OK, I didn't attempt to fix them. If anyone wants the compilation
log, please let me know, and I'll email it to you.
The version of postgresql was 970602. Here is Makefile.custom:
CUSTOM_COPT = -O2 -n32
MK_NO_LORDER = 1
LD = ld -n32
CC += -n32
Subject: [HACKERS] src.original/./backend/lib/fstack.c
Another change I suggested. I bracket an unused function and add a
return to quiet the compiler. In addition I added an internal
consistency check.
Subject: [HACKERS] backend/optimizer/geqo/geqo_erx.c
I sent these changes in with a bunch of others. Some were folded in but
others, like these, were not. I am not sure why so I am resending this
to the developers list by itself for discussion.
The readon why I suggest these changes is that the compiler can't tell
that minimum_count is initialized before it is used. The tests that I
add in here will cause an immediate error if it doesn't. As the comments
below suggest, if it is 100% guaranteed that the variable will always
be initialized then how this is so should be commented here. I don't
know how much strain the actual test puts on the performance but if it
isn't too much then maybe leave it in for absolute safety anyway.
There are also a few returns just to stop warnings.
for join-relations. Sizes already computed by
prune_rel_paths():compute_joinrel_size().
joinrels.c:
< if ( _use_right_sided_plans_ )
---
> if ( _use_right_sided_plans_ &&
> length (outer_rel->relids) > 1 )
- r_plans are useful when outer_rel is join-relation... It
decreases the size of search space...
nestloop's join clauses doesn't work in some cases:
* 1. fix_indxqual_references may change varattno-s in
* inner_indxqual;
* 2. clauses may be commuted
Subject: Re: [HACKERS] GEQO and views (rules)
Oke, this was caused by a classic bug :-/
I thougth, root->base_relation_list_ could be
represented as relid string 1-2-3-4- etc.
Instead, in case of views, the count of relids doesn't start with "1" but
maybe 4-5-6- etc . :-(
GEQO patch follows ... views are now all right.
2. PageWeights are variables now.
3. Fixed using ceil((double)selec*indextuples) as estimation
for expected heap pages: ceil((double)selec*relpages) now.
use sum(npages)/((nkeys == 1) ? 1 : nkeys + 1) as expected index page
estimation for multi-key quals - instead of sum(npages).
In old code npages for x > 10 and x < 20 is twice as for x > 10 - cool ?
FreeBSD
The Makefile(s) have all been cleaned up such that there is a single
LDFLAGS vs LD_ADD or LDADD or LDFLAGS or LDFLAGS_BE. The Makefile(s)
should be alot more straightforward then they were before...and
consistent
Subject: [HACKERS] Aggregate function patches
Here are the aggregate function patches I originally sent in last December.
They fix sum() and avg() behavior for ints and floats when NULL values are
involved.
I was waiting to resubmit these until I had a chance to write a v6.0->v6.1
database upgrade script to ensure that existing v6.0 databases which have
not been reloaded for v6.1 do no break with the new aggregate behavior.
These scripts are included below. It's OK with me if someone wants to do
something different with the upgrade strategy, but something like this
was discussed a few weeks ago.
Also, there were a couple of small items which cropped up in doing a clean
install of 970403 (actually 970402 + 970403 changes since the full 970403
tar file appears to be damaged or at least suspect). They are the first
two patches below and can be omitted if desired (although I think they
aren't dangerous :).
Subject: [HACKERS] locale patches !
Hi there,
here are little patches to get Postgres 6.1 works with locale stuff.
This is a patch against 970402.tar.gz, there are no problem to apply them
by hand to 6.0 release. Collate stuff tested about 1-2 months in real
working database but I'm sure there must be no problem. US hackers
could vote against locale implementation ( locale for sure will affect to
speed of postgres ), so I introduce variable USE_LOCALE which
controls locale stuff. Non-US users now could use ~* operator
for searching and <order by> for strings with nation alphabet.
Please, don't forget, as I did first time, to set environment variable
LC_CTYPE and LC_COLLATE because backend get locale information from them.
I start postmaster from a little script, assuming that shell is Bash shell
it looks like:
#!/bin/sh
export LC_CTYPE=koi8-r
export LC_COLLATE=koi8-r
postmaster -B 1024 -S -D/usr/local/pgsql/data/ -o '-Fe'
> Please apply them to the direcory "backend/optimizer/geqo".
> Two new files with different crossover techniques are included.
> Standard procedure is optimization by means of "geqo_erx.c"
> (Edge Recombination Crossover).
From: "Martin S. Utesch" <utesch@aut.tu-freiberg.de>
Subject: [HACKERS] linux/alpha patches
These patches lay the groundwork for a Linux/Alpha port. The port doesn't
actually work unless you tweak the linker to put all the pointers in the
first 32 bits of the address space, but it's at least a start. It
implements the test-and-set instruction in Alpha assembly, and also fixes
a lot of pointer-to-integer conversions, which is probably good anyway.
Subject: [HACKERS] linux/alpha patches
These patches lay the groundwork for a Linux/Alpha port. The port doesn't
actually work unless you tweak the linker to put all the pointers in the
first 32 bits of the address space, but it's at least a start. It
implements the test-and-set instruction in Alpha assembly, and also fixes
a lot of pointer-to-integer conversions, which is probably good anyway.
2. IndexScanableOperand now uses match_indexkey_operand
instead of equal_indexkey_var (if we have some index on attribute X
then we shouldn't use it for 'where some_func(X) OP CONST').
/usr/include/limits.h (which quiets the costsize.c warnings)...under
FreeBSD, /usr/include/limits.h *includes* machine/limits.h, while under
Solaris, there is no such things as /usr/include/machine...
Problem with Solaris pointed out by Mark Wahl
The problem is that the function arguments are not considered as possible key
candidates for index scan and so only a sequential scan is possible inside
the body of a function. I have therefore made some patches to the optimizer
so that indices are now used also by functions. I have also moved the plan
debug message from pg_eval to pg_plan so that it is printed also for plans
genereated for function execution. I had also to add an index rescan to the
executor because it ignored the parameters set in the execution state, they
were flagged as runtime variables in ExecInitIndexScan but then never used
by the executor so that the scan were always done with any key=1. Very odd.
This means that an index rescan is now done twice for each function execution
which uses an index, the first time when the index scan is initialized and
the second when the actual function arguments are finally available for the
execution. I don't know what is the cost of an double index scan but I
suppose it is anyway less than the cost of a full sequential scan, at leat
for large tables. This is my patch, you must also add -DINDEXSCAN_PATCH in
Makefile.global to enable the changes.
Submitted by: Massimo Dal Zotto <dz@cs.unitn.it>
Select queries with an isnull or notnull clause, like "select * where
somefield isnull", crash the backend if the table has at least one index.
If the indices are deleted the queries work again. Also the explain
command fail in the same way.
The is caused by a bug in subroutine of the optimizer which doesn't check
null values in the clauses.
Submitted by: Massimo Dal Zotto <dz@cs.unitn.it>