2010-09-20 22:08:53 +02:00
|
|
|
<!-- doc/src/sgml/gin.sgml -->
|
2006-09-14 13:16:27 +02:00
|
|
|
|
|
|
|
<chapter id="GIN">
|
|
|
|
<title>GIN Indexes</title>
|
|
|
|
|
|
|
|
<indexterm>
|
|
|
|
<primary>index</primary>
|
|
|
|
<secondary>GIN</secondary>
|
|
|
|
</indexterm>
|
|
|
|
|
|
|
|
<sect1 id="gin-intro">
|
|
|
|
<title>Introduction</title>
|
|
|
|
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<acronym>GIN</acronym> stands for Generalized Inverted Index.
|
|
|
|
<acronym>GIN</acronym> is designed for handling cases where the items
|
|
|
|
to be indexed are composite values, and the queries to be handled by
|
|
|
|
the index need to search for element values that appear within
|
|
|
|
the composite items. For example, the items could be documents,
|
|
|
|
and the queries could be searches for documents containing specific words.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
We use the word <firstterm>item</> to refer to a composite value that
|
|
|
|
is to be indexed, and the word <firstterm>key</> to refer to an element
|
|
|
|
value. <acronym>GIN</acronym> always stores and searches for keys,
|
|
|
|
not item values per se.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
A <acronym>GIN</acronym> index stores a set of (key, posting list) pairs,
|
|
|
|
where a <firstterm>posting list</> is a set of row IDs in which the key
|
|
|
|
occurs. The same row ID can appear in multiple posting lists, since
|
|
|
|
an item can contain more than one key. Each key value is stored only
|
|
|
|
once, so a <acronym>GIN</acronym> index is very compact for cases
|
|
|
|
where the same key appears many times.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<acronym>GIN</acronym> is generalized in the sense that the
|
|
|
|
<acronym>GIN</acronym> access method code does not need to know the
|
|
|
|
specific operations that it accelerates.
|
|
|
|
Instead, it uses custom strategies defined for particular data types.
|
|
|
|
The strategy defines how keys are extracted from indexed items and
|
|
|
|
query conditions, and how to determine whether a row that contains
|
|
|
|
some of the key values in a query actually satisfies the query.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
One advantage of <acronym>GIN</acronym> is that it allows the development
|
|
|
|
of custom data types with the appropriate access methods, by
|
|
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
|
|
This is much the same advantage as using <acronym>GiST</acronym>.
|
|
|
|
</para>
|
|
|
|
|
2006-11-23 06:58:01 +01:00
|
|
|
<para>
|
|
|
|
The <acronym>GIN</acronym>
|
|
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
|
|
maintained by Teodor Sigaev and Oleg Bartunov. There is more
|
|
|
|
information about <acronym>GIN</acronym> on their
|
2008-07-23 00:05:24 +02:00
|
|
|
<ulink url="http://www.sai.msu.su/~megera/wiki/Gin">website</ulink>.
|
2006-11-23 06:58:01 +01:00
|
|
|
</para>
|
2006-09-14 13:16:27 +02:00
|
|
|
</sect1>
|
|
|
|
|
Add an in-core GiST index opclass for inet/cidr types.
This operator class can accelerate subnet/supernet tests as well as
btree-equivalent ordered comparisons. It also handles a new network
operator inet && inet (overlaps, a/k/a "is supernet or subnet of"),
which is expected to be useful in exclusion constraints.
Ideally this opclass would be the default for GiST with inet/cidr data,
but we can't mark it that way until we figure out how to do a more or
less graceful transition from the current situation, in which the
really-completely-bogus inet/cidr opclasses in contrib/btree_gist are
marked as default. Having the opclass in core and not default is better
than not having it at all, though.
While at it, add new documentation sections to allow us to officially
document GiST/GIN/SP-GiST opclasses, something there was never a clear
place to do before. I filled these in with some simple tables listing
the existing opclasses and the operators they support, but there's
certainly scope to put more information there.
Emre Hasegeli, reviewed by Andreas Karlsson, further hacking by me
2014-04-08 21:46:14 +02:00
|
|
|
<sect1 id="gin-builtin-opclasses">
|
|
|
|
<title>Built-in Operator Classes</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The core <productname>PostgreSQL</> distribution
|
|
|
|
includes the <acronym>GIN</acronym> operator classes shown in
|
|
|
|
<xref linkend="gin-builtin-opclasses-table">.
|
|
|
|
(Some of the optional modules described in <xref linkend="contrib">
|
|
|
|
provide additional <acronym>GIN</acronym> operator classes.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<table id="gin-builtin-opclasses-table">
|
|
|
|
<title>Built-in <acronym>GIN</acronym> Operator Classes</title>
|
|
|
|
<tgroup cols="3">
|
|
|
|
<thead>
|
|
|
|
<row>
|
|
|
|
<entry>Name</entry>
|
|
|
|
<entry>Indexed Data Type</entry>
|
|
|
|
<entry>Indexable Operators</entry>
|
|
|
|
</row>
|
|
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_abstime_ops</></entry>
|
|
|
|
<entry><type>abstime[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_bit_ops</></entry>
|
|
|
|
<entry><type>bit[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_bool_ops</></entry>
|
|
|
|
<entry><type>boolean[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_bpchar_ops</></entry>
|
|
|
|
<entry><type>character[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_bytea_ops</></entry>
|
|
|
|
<entry><type>bytea[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_char_ops</></entry>
|
|
|
|
<entry><type>"char"[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_cidr_ops</></entry>
|
|
|
|
<entry><type>cidr[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_date_ops</></entry>
|
|
|
|
<entry><type>date[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_float4_ops</></entry>
|
|
|
|
<entry><type>float4[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_float8_ops</></entry>
|
|
|
|
<entry><type>float8[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_inet_ops</></entry>
|
|
|
|
<entry><type>inet[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_int2_ops</></entry>
|
|
|
|
<entry><type>smallint[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_int4_ops</></entry>
|
|
|
|
<entry><type>integer[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_int8_ops</></entry>
|
|
|
|
<entry><type>bigint[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_interval_ops</></entry>
|
|
|
|
<entry><type>interval[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_macaddr_ops</></entry>
|
|
|
|
<entry><type>macaddr[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_money_ops</></entry>
|
|
|
|
<entry><type>money[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_name_ops</></entry>
|
|
|
|
<entry><type>name[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_numeric_ops</></entry>
|
|
|
|
<entry><type>numeric[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_oid_ops</></entry>
|
|
|
|
<entry><type>oid[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_oidvector_ops</></entry>
|
|
|
|
<entry><type>oidvector[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_reltime_ops</></entry>
|
|
|
|
<entry><type>reltime[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_text_ops</></entry>
|
|
|
|
<entry><type>text[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_time_ops</></entry>
|
|
|
|
<entry><type>time[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_timestamp_ops</></entry>
|
|
|
|
<entry><type>timestamp[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_timestamptz_ops</></entry>
|
|
|
|
<entry><type>timestamp with time zone[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_timetz_ops</></entry>
|
|
|
|
<entry><type>time with time zone[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_tinterval_ops</></entry>
|
|
|
|
<entry><type>tinterval[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_varbit_ops</></entry>
|
|
|
|
<entry><type>bit varying[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>_varchar_ops</></entry>
|
|
|
|
<entry><type>character varying[]</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>&&</>
|
|
|
|
<literal><@</>
|
|
|
|
<literal>=</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>jsonb_ops</></entry>
|
|
|
|
<entry><type>jsonb</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>?</>
|
|
|
|
<literal>?&</>
|
|
|
|
<literal>?|</>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
2014-05-11 18:06:04 +02:00
|
|
|
<entry><literal>jsonb_path_ops</></entry>
|
Add an in-core GiST index opclass for inet/cidr types.
This operator class can accelerate subnet/supernet tests as well as
btree-equivalent ordered comparisons. It also handles a new network
operator inet && inet (overlaps, a/k/a "is supernet or subnet of"),
which is expected to be useful in exclusion constraints.
Ideally this opclass would be the default for GiST with inet/cidr data,
but we can't mark it that way until we figure out how to do a more or
less graceful transition from the current situation, in which the
really-completely-bogus inet/cidr opclasses in contrib/btree_gist are
marked as default. Having the opclass in core and not default is better
than not having it at all, though.
While at it, add new documentation sections to allow us to officially
document GiST/GIN/SP-GiST opclasses, something there was never a clear
place to do before. I filled these in with some simple tables listing
the existing opclasses and the operators they support, but there's
certainly scope to put more information there.
Emre Hasegeli, reviewed by Andreas Karlsson, further hacking by me
2014-04-08 21:46:14 +02:00
|
|
|
<entry><type>jsonb</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>@></>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
<row>
|
|
|
|
<entry><literal>tsvector_ops</></entry>
|
|
|
|
<entry><type>tsvector</></entry>
|
|
|
|
<entry>
|
|
|
|
<literal>@@</>
|
|
|
|
<literal>@@@</>
|
|
|
|
</entry>
|
|
|
|
</row>
|
|
|
|
</tbody>
|
|
|
|
</tgroup>
|
|
|
|
</table>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Of the two operator classes for type <type>jsonb</>, <literal>jsonb_ops</>
|
2014-05-11 18:06:04 +02:00
|
|
|
is the default. <literal>jsonb_path_ops</> supports fewer operators but
|
2014-05-09 22:33:25 +02:00
|
|
|
offers better performance for those operators.
|
2014-05-11 00:56:52 +02:00
|
|
|
See <xref linkend="json-indexing"> for details.
|
Add an in-core GiST index opclass for inet/cidr types.
This operator class can accelerate subnet/supernet tests as well as
btree-equivalent ordered comparisons. It also handles a new network
operator inet && inet (overlaps, a/k/a "is supernet or subnet of"),
which is expected to be useful in exclusion constraints.
Ideally this opclass would be the default for GiST with inet/cidr data,
but we can't mark it that way until we figure out how to do a more or
less graceful transition from the current situation, in which the
really-completely-bogus inet/cidr opclasses in contrib/btree_gist are
marked as default. Having the opclass in core and not default is better
than not having it at all, though.
While at it, add new documentation sections to allow us to officially
document GiST/GIN/SP-GiST opclasses, something there was never a clear
place to do before. I filled these in with some simple tables listing
the existing opclasses and the operators they support, but there's
certainly scope to put more information there.
Emre Hasegeli, reviewed by Andreas Karlsson, further hacking by me
2014-04-08 21:46:14 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<sect1 id="gin-extensibility">
|
|
|
|
<title>Extensibility</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>GIN</acronym> interface has a high level of abstraction,
|
2006-12-02 00:46:46 +01:00
|
|
|
requiring the access method implementer only to implement the semantics of
|
2006-09-14 13:16:27 +02:00
|
|
|
the data type being accessed. The <acronym>GIN</acronym> layer itself
|
|
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2008-05-16 18:31:02 +02:00
|
|
|
All it takes to get a <acronym>GIN</acronym> access method working is to
|
2014-03-12 16:13:22 +01:00
|
|
|
implement a few user-defined methods, which define the behavior of
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
keys in the tree and the relationships between keys, indexed items,
|
2006-12-02 00:46:46 +01:00
|
|
|
and indexable queries. In short, <acronym>GIN</acronym> combines
|
|
|
|
extensibility with generality, code reuse, and a clean interface.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-03-12 16:13:22 +01:00
|
|
|
There are three methods that an operator class for
|
|
|
|
<acronym>GIN</acronym> must provide:
|
2006-09-14 13:16:27 +02:00
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><function>int compare(Datum a, Datum b)</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Compares two keys (not indexed items!) and returns an integer less than
|
2006-12-02 00:46:46 +01:00
|
|
|
zero, zero, or greater than zero, indicating whether the first key is
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
less than, equal to, or greater than the second. Null keys are never
|
|
|
|
passed to this function.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<term><function>Datum *extractValue(Datum itemValue, int32 *nkeys,
|
|
|
|
bool **nullFlags)</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Returns a palloc'd array of keys given an item to be indexed. The
|
2006-12-02 00:46:46 +01:00
|
|
|
number of returned keys must be stored into <literal>*nkeys</>.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
If any of the keys can be null, also palloc an array of
|
2013-05-21 03:13:13 +02:00
|
|
|
<literal>*nkeys</> <type>bool</type> fields, store its address at
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<literal>*nullFlags</>, and set these null flags as needed.
|
2012-06-07 23:06:20 +02:00
|
|
|
<literal>*nullFlags</> can be left <symbol>NULL</symbol> (its initial value)
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
if all keys are non-null.
|
2012-06-07 23:06:20 +02:00
|
|
|
The return value can be <symbol>NULL</symbol> if the item contains no keys.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><function>Datum *extractQuery(Datum query, int32 *nkeys,
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
StrategyNumber n, bool **pmatch, Pointer **extra_data,
|
|
|
|
bool **nullFlags, int32 *searchMode)</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Returns a palloc'd array of keys given a value to be queried; that is,
|
2006-12-02 00:46:46 +01:00
|
|
|
<literal>query</> is the value on the right-hand side of an
|
|
|
|
indexable operator whose left-hand side is the indexed column.
|
|
|
|
<literal>n</> is the strategy number of the operator within the
|
|
|
|
operator class (see <xref linkend="xindex-strategies">).
|
|
|
|
Often, <function>extractQuery</> will need
|
|
|
|
to consult <literal>n</> to determine the data type of
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<literal>query</> and the method it should use to extract key values.
|
2006-12-02 00:46:46 +01:00
|
|
|
The number of returned keys must be stored into <literal>*nkeys</>.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
If any of the keys can be null, also palloc an array of
|
2013-05-21 03:13:13 +02:00
|
|
|
<literal>*nkeys</> <type>bool</type> fields, store its address at
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<literal>*nullFlags</>, and set these null flags as needed.
|
2013-05-21 03:13:13 +02:00
|
|
|
<literal>*nullFlags</> can be left <symbol>NULL</symbol> (its initial value)
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
if all keys are non-null.
|
2013-05-21 03:13:13 +02:00
|
|
|
The return value can be <symbol>NULL</symbol> if the <literal>query</> contains no keys.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<literal>searchMode</> is an output argument that allows
|
|
|
|
<function>extractQuery</> to specify details about how the search
|
|
|
|
will be done.
|
|
|
|
If <literal>*searchMode</> is set to
|
|
|
|
<literal>GIN_SEARCH_MODE_DEFAULT</> (which is the value it is
|
|
|
|
initialized to before call), only items that match at least one of
|
|
|
|
the returned keys are considered candidate matches.
|
|
|
|
If <literal>*searchMode</> is set to
|
|
|
|
<literal>GIN_SEARCH_MODE_INCLUDE_EMPTY</>, then in addition to items
|
|
|
|
containing at least one matching key, items that contain no keys at
|
|
|
|
all are considered candidate matches. (This mode is useful for
|
|
|
|
implementing is-subset-of operators, for example.)
|
|
|
|
If <literal>*searchMode</> is set to <literal>GIN_SEARCH_MODE_ALL</>,
|
|
|
|
then all non-null items in the index are considered candidate
|
|
|
|
matches, whether they match any of the returned keys or not. (This
|
|
|
|
mode is much slower than the other two choices, since it requires
|
|
|
|
scanning essentially the entire index, but it may be necessary to
|
|
|
|
implement corner cases correctly. An operator that needs this mode
|
|
|
|
in most cases is probably not a good candidate for a GIN operator
|
|
|
|
class.)
|
|
|
|
The symbols to use for setting this mode are defined in
|
|
|
|
<filename>access/gin.h</>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2008-05-16 18:31:02 +02:00
|
|
|
<literal>pmatch</> is an output argument for use when partial match
|
|
|
|
is supported. To use it, <function>extractQuery</> must allocate
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
an array of <literal>*nkeys</> booleans and store its address at
|
2008-05-16 18:31:02 +02:00
|
|
|
<literal>*pmatch</>. Each element of the array should be set to TRUE
|
|
|
|
if the corresponding key requires partial match, FALSE if not.
|
2013-05-21 03:13:13 +02:00
|
|
|
If <literal>*pmatch</> is set to <symbol>NULL</symbol> then GIN assumes partial match
|
|
|
|
is not required. The variable is initialized to <symbol>NULL</symbol> before call,
|
2008-05-16 18:31:02 +02:00
|
|
|
so this argument can simply be ignored by operator classes that do
|
|
|
|
not support partial match.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2009-03-25 23:19:02 +01:00
|
|
|
<literal>extra_data</> is an output argument that allows
|
|
|
|
<function>extractQuery</> to pass additional data to the
|
|
|
|
<function>consistent</> and <function>comparePartial</> methods.
|
|
|
|
To use it, <function>extractQuery</> must allocate
|
|
|
|
an array of <literal>*nkeys</> Pointers and store its address at
|
|
|
|
<literal>*extra_data</>, then store whatever it wants to into the
|
2013-05-21 03:13:13 +02:00
|
|
|
individual pointers. The variable is initialized to <symbol>NULL</symbol> before
|
2009-03-25 23:19:02 +01:00
|
|
|
call, so this argument can simply be ignored by operator classes that
|
|
|
|
do not require extra data. If <literal>*extra_data</> is set, the
|
|
|
|
whole array is passed to the <function>consistent</> method, and
|
|
|
|
the appropriate element to the <function>comparePartial</> method.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
2008-05-16 18:31:02 +02:00
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
2014-03-12 16:13:22 +01:00
|
|
|
</variablelist>
|
|
|
|
|
|
|
|
An operator class must also provide a function to check if an indexed item
|
|
|
|
matches the query. It comes in two flavors, a boolean <function>consistent</>
|
|
|
|
function, and a ternary <function>triConsistent</> function.
|
|
|
|
<function>triConsistent</> covers the functionality of both, so providing
|
|
|
|
triConsistent alone is sufficient. However, if the boolean variant is
|
2014-03-13 14:01:45 +01:00
|
|
|
significantly cheaper to calculate, it can be advantageous to provide both.
|
2014-03-12 16:13:22 +01:00
|
|
|
If only the boolean variant is provided, some optimizations that depend on
|
2014-07-09 05:29:09 +02:00
|
|
|
refuting index items before fetching all the keys are disabled.
|
2006-09-14 13:16:27 +02:00
|
|
|
|
2014-03-12 16:13:22 +01:00
|
|
|
<variablelist>
|
2006-09-14 13:16:27 +02:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><function>bool consistent(bool check[], StrategyNumber n, Datum query,
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
int32 nkeys, Pointer extra_data[], bool *recheck,
|
|
|
|
Datum queryKeys[], bool nullFlags[])</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Returns TRUE if an indexed item satisfies the query operator with
|
|
|
|
strategy number <literal>n</> (or might satisfy it, if the recheck
|
|
|
|
indication is returned). This function does not have direct access
|
|
|
|
to the indexed item's value, since <acronym>GIN</acronym> does not
|
|
|
|
store items explicitly. Rather, what is available is knowledge
|
|
|
|
about which key values extracted from the query appear in a given
|
|
|
|
indexed item. The <literal>check</> array has length
|
2009-03-25 23:19:02 +01:00
|
|
|
<literal>nkeys</>, which is the same as the number of keys previously
|
|
|
|
returned by <function>extractQuery</> for this <literal>query</> datum.
|
|
|
|
Each element of the
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<literal>check</> array is TRUE if the indexed item contains the
|
2014-03-13 14:01:45 +01:00
|
|
|
corresponding query key, i.e., if (check[i] == TRUE) the i-th key of the
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<function>extractQuery</> result array is present in the indexed item.
|
|
|
|
The original <literal>query</> datum is
|
|
|
|
passed in case the <function>consistent</> method needs to consult it,
|
|
|
|
and so are the <literal>queryKeys[]</> and <literal>nullFlags[]</>
|
|
|
|
arrays previously returned by <function>extractQuery</>.
|
2009-03-25 23:19:02 +01:00
|
|
|
<literal>extra_data</> is the extra-data array returned by
|
2013-05-21 03:13:13 +02:00
|
|
|
<function>extractQuery</>, or <symbol>NULL</symbol> if none.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When <function>extractQuery</> returns a null key in
|
|
|
|
<literal>queryKeys[]</>, the corresponding <literal>check[]</> element
|
|
|
|
is TRUE if the indexed item contains a null key; that is, the
|
|
|
|
semantics of <literal>check[]</> are like <literal>IS NOT DISTINCT
|
|
|
|
FROM</>. The <function>consistent</> function can examine the
|
|
|
|
corresponding <literal>nullFlags[]</> element if it needs to tell
|
|
|
|
the difference between a regular value match and a null match.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2008-04-14 19:05:34 +02:00
|
|
|
On success, <literal>*recheck</> should be set to TRUE if the heap
|
|
|
|
tuple needs to be rechecked against the query operator, or FALSE if
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
the index test is exact. That is, a FALSE return value guarantees
|
|
|
|
that the heap tuple does not match the query; a TRUE return value with
|
|
|
|
<literal>*recheck</> set to FALSE guarantees that the heap tuple does
|
|
|
|
match the query; and a TRUE return value with
|
|
|
|
<literal>*recheck</> set to TRUE means that the heap tuple might match
|
|
|
|
the query, so it needs to be fetched and rechecked by evaluating the
|
|
|
|
query operator directly against the originally indexed item.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
2014-03-12 16:13:22 +01:00
|
|
|
|
|
|
|
<varlistentry>
|
|
|
|
<term><function>GinLogicValue triConsistent(GinLogicValue check[], StrategyNumber n, Datum query,
|
|
|
|
int32 nkeys, Pointer extra_data[],
|
|
|
|
Datum queryKeys[], bool nullFlags[])</></term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
<function>triConsistent</> is similar to <function>consistent</>,
|
|
|
|
but instead of a boolean <literal>check[]</>, there are three possible
|
|
|
|
values for each key: <literal>GIN_TRUE</>, <literal>GIN_FALSE</> and
|
|
|
|
<literal>GIN_MAYBE</>. <literal>GIN_FALSE</> and <literal>GIN_TRUE</>
|
|
|
|
have the same meaning as regular boolean values.
|
|
|
|
<literal>GIN_MAYBE</> means that the presence of that key is not known.
|
|
|
|
When <literal>GIN_MAYBE</> values are present, the function should only
|
|
|
|
return GIN_TRUE if the item matches whether or not the index item
|
|
|
|
contains the corresponding query keys. Likewise, the function must
|
2014-07-09 05:29:09 +02:00
|
|
|
return GIN_FALSE only if the item does not match, whether or not it
|
2014-03-12 16:13:22 +01:00
|
|
|
contains the GIN_MAYBE keys. If the result depends on the GIN_MAYBE
|
2014-03-17 12:47:28 +01:00
|
|
|
entries, i.e. the match cannot be confirmed or refuted based on the
|
2014-03-12 16:13:22 +01:00
|
|
|
known query keys, the function must return GIN_MAYBE.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
When there are no GIN_MAYBE values in the <literal>check</> vector,
|
|
|
|
<literal>GIN_MAYBE</> return value is equivalent of setting
|
|
|
|
<literal>recheck</> flag in the boolean <function>consistent</> function.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
2006-09-14 13:16:27 +02:00
|
|
|
</variablelist>
|
|
|
|
|
2014-03-12 16:13:22 +01:00
|
|
|
Optionally, an operator class for <acronym>GIN</acronym> can supply the
|
|
|
|
following method:
|
2008-05-16 18:31:02 +02:00
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><function>int comparePartial(Datum partial_key, Datum key, StrategyNumber n,
|
|
|
|
Pointer extra_data)</></term>
|
2008-05-16 18:31:02 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Compare a partial-match query key to an index key. Returns an integer
|
2008-05-16 18:31:02 +02:00
|
|
|
whose sign indicates the result: less than zero means the index key
|
|
|
|
does not match the query, but the index scan should continue; zero
|
|
|
|
means that the index key does match the query; greater than zero
|
|
|
|
indicates that the index scan should stop because no more matches
|
|
|
|
are possible. The strategy number <literal>n</> of the operator
|
|
|
|
that generated the partial match query is provided, in case its
|
2009-03-25 23:19:02 +01:00
|
|
|
semantics are needed to determine when to end the scan. Also,
|
|
|
|
<literal>extra_data</> is the corresponding element of the extra-data
|
2013-05-21 03:13:13 +02:00
|
|
|
array made by <function>extractQuery</>, or <symbol>NULL</symbol> if none.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
Null keys are never passed to this function.
|
2008-05-16 18:31:02 +02:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
2010-08-17 06:37:21 +02:00
|
|
|
</para>
|
2008-05-16 18:31:02 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
To support <quote>partial match</> queries, an operator class must
|
|
|
|
provide the <function>comparePartial</> method, and its
|
|
|
|
<function>extractQuery</> method must set the <literal>pmatch</>
|
|
|
|
parameter when a partial-match query is encountered. See
|
|
|
|
<xref linkend="gin-partial-match"> for details.
|
|
|
|
</para>
|
|
|
|
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<para>
|
|
|
|
The actual data types of the various <literal>Datum</> values mentioned
|
|
|
|
above vary depending on the operator class. The item values passed to
|
|
|
|
<function>extractValue</> are always of the operator class's input type, and
|
|
|
|
all key values must be of the class's <literal>STORAGE</> type. The type of
|
2014-03-12 16:13:22 +01:00
|
|
|
the <literal>query</> argument passed to <function>extractQuery</>,
|
|
|
|
<function>consistent</> and <function>triConsistent</> is whatever is
|
|
|
|
specified as the right-hand input
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
type of the class member operator identified by the strategy number.
|
|
|
|
This need not be the same as the item type, so long as key values of the
|
|
|
|
correct type can be extracted from it.
|
|
|
|
</para>
|
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
</sect1>
|
|
|
|
|
2006-12-02 00:46:46 +01:00
|
|
|
<sect1 id="gin-implementation">
|
|
|
|
<title>Implementation</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Internally, a <acronym>GIN</acronym> index contains a B-tree index
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
constructed over keys, where each key is an element of one or more indexed
|
|
|
|
items (a member of an array, for example) and where each tuple in a leaf
|
|
|
|
page contains either a pointer to a B-tree of heap pointers (a
|
|
|
|
<quote>posting tree</>), or a simple list of heap pointers (a <quote>posting
|
|
|
|
list</>) when the list is small enough to fit into a single index tuple along
|
|
|
|
with the key value.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2013-05-21 03:13:13 +02:00
|
|
|
As of <productname>PostgreSQL</productname> 9.1, null key values can be
|
|
|
|
included in the index. Also, placeholder nulls are included in the index
|
|
|
|
for indexed items that are null or contain no keys according to
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<function>extractValue</>. This allows searches that should find empty
|
|
|
|
items to do so.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2012-06-07 23:06:20 +02:00
|
|
|
Multicolumn <acronym>GIN</acronym> indexes are implemented by building
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
a single B-tree over composite values (column number, key value). The
|
|
|
|
key values for different columns can be of different types.
|
2006-12-02 00:46:46 +01:00
|
|
|
</para>
|
|
|
|
|
2009-03-24 21:17:18 +01:00
|
|
|
<sect2 id="gin-fast-update">
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>GIN Fast Update Technique</title>
|
2009-03-24 21:17:18 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
Updating a <acronym>GIN</acronym> index tends to be slow because of the
|
|
|
|
intrinsic nature of inverted indexes: inserting or updating one heap row
|
|
|
|
can cause many inserts into the index (one for each key extracted
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
from the indexed item). As of <productname>PostgreSQL</productname> 8.4,
|
2009-03-24 21:17:18 +01:00
|
|
|
<acronym>GIN</> is capable of postponing much of this work by inserting
|
|
|
|
new tuples into a temporary, unsorted list of pending entries.
|
|
|
|
When the table is vacuumed, or if the pending list becomes too large
|
|
|
|
(larger than <xref linkend="guc-work-mem">), the entries are moved to the
|
|
|
|
main <acronym>GIN</acronym> data structure using the same bulk insert
|
|
|
|
techniques used during initial index creation. This greatly improves
|
|
|
|
<acronym>GIN</acronym> index update speed, even counting the additional
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
vacuum overhead. Moreover the overhead work can be done by a background
|
2009-03-24 21:17:18 +01:00
|
|
|
process instead of in foreground query processing.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The main disadvantage of this approach is that searches must scan the list
|
|
|
|
of pending entries in addition to searching the regular index, and so
|
|
|
|
a large list of pending entries will slow searches significantly.
|
|
|
|
Another disadvantage is that, while most updates are fast, an update
|
|
|
|
that causes the pending list to become <quote>too large</> will incur an
|
|
|
|
immediate cleanup cycle and thus be much slower than other updates.
|
|
|
|
Proper use of autovacuum can minimize both of these problems.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If consistent response time is more important than update speed,
|
|
|
|
use of pending entries can be disabled by turning off the
|
|
|
|
<literal>FASTUPDATE</literal> storage parameter for a
|
2010-04-03 09:23:02 +02:00
|
|
|
<acronym>GIN</acronym> index. See <xref linkend="sql-createindex">
|
|
|
|
for details.
|
2009-03-24 21:17:18 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2008-05-16 18:31:02 +02:00
|
|
|
<sect2 id="gin-partial-match">
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Partial Match Algorithm</title>
|
2009-03-24 21:17:18 +01:00
|
|
|
|
2008-05-16 18:31:02 +02:00
|
|
|
<para>
|
|
|
|
GIN can support <quote>partial match</> queries, in which the query
|
|
|
|
does not determine an exact match for one or more keys, but the possible
|
|
|
|
matches fall within a reasonably narrow range of key values (within the
|
|
|
|
key sorting order determined by the <function>compare</> support method).
|
|
|
|
The <function>extractQuery</> method, instead of returning a key value
|
|
|
|
to be matched exactly, returns a key value that is the lower bound of
|
|
|
|
the range to be searched, and sets the <literal>pmatch</> flag true.
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
The key range is then scanned using the <function>comparePartial</>
|
|
|
|
method. <function>comparePartial</> must return zero for a matching
|
|
|
|
index key, less than zero for a non-match that is still within the range
|
2008-05-16 18:31:02 +02:00
|
|
|
to be searched, or greater than zero if the index key is past the range
|
|
|
|
that could match.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2006-12-02 00:46:46 +01:00
|
|
|
</sect1>
|
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<sect1 id="gin-tips">
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>GIN Tips and Tricks</title>
|
2006-09-14 13:16:27 +02:00
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2011-01-29 19:00:18 +01:00
|
|
|
<term>Create vs. insert</term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
2006-12-02 00:46:46 +01:00
|
|
|
<para>
|
2009-03-24 21:17:18 +01:00
|
|
|
Insertion into a <acronym>GIN</acronym> index can be slow
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
due to the likelihood of many keys being inserted for each item.
|
2006-12-02 00:46:46 +01:00
|
|
|
So, for bulk insertions into a table it is advisable to drop the GIN
|
|
|
|
index and recreate it after finishing bulk insertion.
|
|
|
|
</para>
|
2009-03-24 21:17:18 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
As of <productname>PostgreSQL</productname> 8.4, this advice is less
|
|
|
|
necessary since delayed indexing is used (see <xref
|
|
|
|
linkend="gin-fast-update"> for details). But for very large updates
|
|
|
|
it may still be best to drop and recreate the index.
|
|
|
|
</para>
|
2006-09-14 13:16:27 +02:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2007-11-16 04:23:07 +01:00
|
|
|
<varlistentry>
|
|
|
|
<term><xref linkend="guc-maintenance-work-mem"></term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Build time for a <acronym>GIN</acronym> index is very sensitive to
|
|
|
|
the <varname>maintenance_work_mem</> setting; it doesn't pay to
|
|
|
|
skimp on work memory during index creation.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2009-03-24 21:17:18 +01:00
|
|
|
<varlistentry>
|
|
|
|
<term><xref linkend="guc-work-mem"></term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
During a series of insertions into an existing <acronym>GIN</acronym>
|
|
|
|
index that has <literal>FASTUPDATE</> enabled, the system will clean up
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
the pending-entry list whenever the list grows larger than
|
2009-03-24 21:17:18 +01:00
|
|
|
<varname>work_mem</>. To avoid fluctuations in observed response time,
|
|
|
|
it's desirable to have pending-list cleanup occur in the background
|
|
|
|
(i.e., via autovacuum). Foreground cleanup operations can be avoided by
|
|
|
|
increasing <varname>work_mem</> or making autovacuum more aggressive.
|
|
|
|
However, enlarging <varname>work_mem</> means that if a foreground
|
|
|
|
cleanup does occur, it will take even longer.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<varlistentry>
|
2006-12-02 00:46:46 +01:00
|
|
|
<term><xref linkend="guc-gin-fuzzy-search-limit"></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
2006-12-02 00:46:46 +01:00
|
|
|
<para>
|
|
|
|
The primary goal of developing <acronym>GIN</acronym> indexes was
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
to create support for highly scalable full-text search in
|
2006-12-02 00:46:46 +01:00
|
|
|
<productname>PostgreSQL</productname>, and there are often situations when
|
|
|
|
a full-text search returns a very large set of results. Moreover, this
|
|
|
|
often happens when the query contains very frequent words, so that the
|
|
|
|
large result set is not even useful. Since reading many
|
|
|
|
tuples from the disk and sorting them could take a lot of time, this is
|
|
|
|
unacceptable for production. (Note that the index search itself is very
|
|
|
|
fast.)
|
|
|
|
</para>
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
To facilitate controlled execution of such queries,
|
2007-11-16 04:23:07 +01:00
|
|
|
<acronym>GIN</acronym> has a configurable soft upper limit on the
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
number of rows returned: the
|
2006-12-02 00:46:46 +01:00
|
|
|
<varname>gin_fuzzy_search_limit</varname> configuration parameter.
|
|
|
|
It is set to 0 (meaning no limit) by default.
|
|
|
|
If a non-zero limit is set, then the returned set is a subset of
|
|
|
|
the whole result set, chosen at random.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
<quote>Soft</quote> means that the actual number of returned results
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
could differ somewhat from the specified limit, depending on the query
|
2006-12-02 00:46:46 +01:00
|
|
|
and the quality of the system's random number generator.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<para>
|
|
|
|
From experience, values in the thousands (e.g., 5000 — 20000)
|
|
|
|
work well.
|
|
|
|
</para>
|
2006-09-14 13:16:27 +02:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
2006-09-14 15:40:28 +02:00
|
|
|
</variablelist>
|
2006-09-14 13:16:27 +02:00
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="gin-limit">
|
|
|
|
<title>Limitations</title>
|
|
|
|
|
|
|
|
<para>
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
<acronym>GIN</acronym> assumes that indexable operators are strict. This
|
2013-05-21 03:13:13 +02:00
|
|
|
means that <function>extractValue</> will not be called at all on a null
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
item value (instead, a placeholder index entry is created automatically),
|
2013-05-21 03:13:13 +02:00
|
|
|
and <function>extractQuery</function> will not be called on a null query
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
value either (instead, the query is presumed to be unsatisfiable). Note
|
2013-05-21 03:13:13 +02:00
|
|
|
however that null key values contained within a non-null composite item
|
Fix GIN to support null keys, empty and null items, and full index scans.
Per my recent proposal(s). Null key datums can now be returned by
extractValue and extractQuery functions, and will be stored in the index.
Also, placeholder entries are made for indexable items that are NULL or
contain no keys according to extractValue. This means that the index is
now always complete, having at least one entry for every indexed heap TID,
and so we can get rid of the prohibition on full-index scans. A full-index
scan is implemented much the same way as partial-match scans were already:
we build a bitmap representing all the TIDs found in the index, and then
drive the results off that.
Also, introduce a concept of a "search mode" that can be requested by
extractQuery when the operator requires matching to empty items (this is
just as cheap as matching to a single key) or requires a full index scan
(which is not so cheap, but it sure beats failing or giving wrong answers).
The behavior remains backward compatible for opclasses that don't return
any null keys or request a non-default search mode.
Using these features, we can now make the GIN index opclass for anyarray
behave in a way that matches the actual anyarray operators for &&, <@, @>,
and = ... which it failed to do before in assorted corner cases.
This commit fixes the core GIN code and ginarrayprocs.c, updates the
documentation, and adds some simple regression test cases for the new
behaviors using the array operators. The tsearch and contrib GIN opclass
support functions still need to be looked over and probably fixed.
Another thing I intend to fix separately is that this is pretty inefficient
for cases where more than one scan condition needs a full-index search:
we'll run duplicate GinScanEntrys, each one of which builds a large bitmap.
There is some existing logic to merge duplicate GinScanEntrys but it needs
refactoring to make it work for entries belonging to different scan keys.
Note that most of gin.h has been split out into a new file gin_private.h,
so that gin.h doesn't export anything that's not supposed to be used by GIN
opclasses or the rest of the backend. I did quite a bit of other code
beautification work as well, mostly fixing comments and choosing more
appropriate names for things.
2011-01-08 01:16:24 +01:00
|
|
|
or query value are supported.
|
2006-09-14 13:16:27 +02:00
|
|
|
</para>
|
|
|
|
</sect1>
|
2006-09-14 23:15:07 +02:00
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<sect1 id="gin-examples">
|
|
|
|
<title>Examples</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
2007-11-14 00:36:26 +01:00
|
|
|
<acronym>GIN</acronym> operator classes for <type>tsvector</> and
|
2008-05-16 18:31:02 +02:00
|
|
|
for one-dimensional arrays of all internal types. Prefix searching in
|
|
|
|
<type>tsvector</> is implemented using the <acronym>GIN</> partial match
|
|
|
|
feature.
|
|
|
|
The following <filename>contrib</> modules also contain
|
|
|
|
<acronym>GIN</acronym> operator classes:
|
2006-12-02 00:46:46 +01:00
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<variablelist>
|
2009-03-25 23:19:02 +01:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>btree_gin</></term>
|
2009-03-25 23:19:02 +01:00
|
|
|
<listitem>
|
2010-08-17 06:37:21 +02:00
|
|
|
<para>B-tree equivalent functionality for several data types</para>
|
2009-03-25 23:19:02 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2007-11-14 00:36:26 +01:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>hstore</></term>
|
2007-11-14 00:36:26 +01:00
|
|
|
<listitem>
|
|
|
|
<para>Module for storing (key, value) pairs</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2006-09-14 13:16:27 +02:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>intarray</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
2010-08-17 06:37:21 +02:00
|
|
|
<para>Enhanced support for <type>int[]</type></para>
|
2006-09-14 13:16:27 +02:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>pg_trgm</></term>
|
2006-09-14 13:16:27 +02:00
|
|
|
<listitem>
|
2007-11-14 00:36:26 +01:00
|
|
|
<para>Text similarity using trigram matching</para>
|
2006-09-14 13:16:27 +02:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
2006-09-14 23:15:07 +02:00
|
|
|
</variablelist>
|
2010-08-17 06:37:21 +02:00
|
|
|
</para>
|
2006-09-14 23:15:07 +02:00
|
|
|
</sect1>
|
2006-09-14 13:16:27 +02:00
|
|
|
|
|
|
|
</chapter>
|