rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fix logical errors in tsquery selectivity estimation for prefix queries. 

I made multiple errors in commit 97532f7c29,
stemming mostly from failure to think about the available frequency data
as being element frequencies not value frequencies (so that occurrences of
different elements are not mutually exclusive).  This led to sillinesses
such as estimating that "word" would match more rows than "word:*".

The choice to clamp to a minimum estimate of DEFAULT_TS_MATCH_SEL also
seems pretty ill-considered in hindsight, as it would frequently result in
an estimate much larger than the available data suggests.  We do need some
sort of clamp, since a pattern not matching any of the MCELEMs probably
still needs a selectivity estimate of more than zero.  I chose instead to
clamp to at least what a non-MCELEM word would be estimated as, preserving
the property that "word:*" doesn't get an estimate less than plain "word",
whether or not the word appears in MCELEM.

Per investigation of a gripe from Bill Martin, though I suspect that his
example case actually isn't even reaching the erroneous code.

Back-patch to 9.1 where this code was introduced.
This commit is contained in:
Tom Lane 2012-09-11 21:23:20 -04:00
parent 4bc0d2e2cf
commit 1faf866ace
1 changed files with 28 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
src/backend/tsearch/ts_selfuncs.c
View File

@ -304,23 +304,29 @@ tsquery_opr_selec(QueryItem *item, char *operand,
{
/* Prefix match, ie the query item is lexeme:* */
Selectivity matched,
allmcvs;
int i;
allmces;
int i,
n_matched;
/*
* Our strategy is to scan through the MCV list and add up the
* frequencies of the ones that match the prefix, thereby assuming
* that the MCVs are representative of the whole lexeme population
* in this respect. Compare histogram_selectivity().
* Our strategy is to scan through the MCELEM list and combine the
* frequencies of the ones that match the prefix. We then
* extrapolate the fraction of matching MCELEMs to the remaining
* rows, assuming that the MCELEMs are representative of the whole
* lexeme population in this respect. (Compare
* histogram_selectivity().) Note that these are most common
* elements not most common values, so they're not mutually
* exclusive. We treat occurrences as independent events.
*
* This is only a good plan if we have a pretty fair number of
* MCVs available; we set the threshold at 100. If no stats or
* MCELEMs available; we set the threshold at 100. If no stats or
* insufficient stats, arbitrarily use DEFAULT_TS_MATCH_SEL*4.
*/
if (lookup == NULL || length < 100)
return (Selectivity) (DEFAULT_TS_MATCH_SEL * 4);
matched = allmcvs = 0;
matched = allmces = 0;
n_matched = 0;
for (i = 0; i < length; i++)
{
TextFreq *t = lookup + i;
@ -329,20 +335,26 @@ tsquery_opr_selec(QueryItem *item, char *operand,
if (tlen >= key.length &&
strncmp(key.lexeme, VARDATA_ANY(t->element),
key.length) == 0)
matched += t->frequency;
allmcvs += t->frequency;
{
matched += t->frequency - matched * t->frequency;
n_matched++;
}
allmces += t->frequency - allmces * t->frequency;
}
if (allmcvs > 0) /* paranoia about zero divide */
selec = matched / allmcvs;
else
selec = (Selectivity) (DEFAULT_TS_MATCH_SEL * 4);
/* Clamp to ensure sanity in the face of roundoff error */
CLAMP_PROBABILITY(matched);
CLAMP_PROBABILITY(allmces);
selec = matched + (1.0 - allmces) * ((double) n_matched / length);
/*
* In any case, never believe that a prefix match has selectivity
* less than DEFAULT_TS_MATCH_SEL.
* less than we would assign for a non-MCELEM lexeme. This
* preserves the property that "word:*" should be estimated to
* match at least as many rows as "word" would be.
*/
selec = Max(DEFAULT_TS_MATCH_SEL, selec);
selec = Max(Min(DEFAULT_TS_MATCH_SEL, minfreq / 2), selec);
}
else
{