357 lines
13 KiB
Plaintext
357 lines
13 KiB
Plaintext
<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.15 2008/05/16 16:31:01 tgl Exp $ -->
|
|
|
|
<chapter id="GIN">
|
|
<title>GIN Indexes</title>
|
|
|
|
<indexterm>
|
|
<primary>index</primary>
|
|
<secondary>GIN</secondary>
|
|
</indexterm>
|
|
|
|
<sect1 id="gin-intro">
|
|
<title>Introduction</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> stands for Generalized Inverted Index. It is
|
|
an index structure storing a set of (key, posting list) pairs, where
|
|
a <quote>posting list</> is a set of rows in which the key occurs. Each
|
|
indexed value can contain many keys, so the same row ID can appear in
|
|
multiple posting lists.
|
|
</para>
|
|
|
|
<para>
|
|
It is generalized in the sense that a <acronym>GIN</acronym> index
|
|
does not need to be aware of the operation that it accelerates.
|
|
Instead, it uses custom strategies defined for particular data types.
|
|
</para>
|
|
|
|
<para>
|
|
One advantage of <acronym>GIN</acronym> is that it allows the development
|
|
of custom data types with the appropriate access methods, by
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
This is much the same advantage as using <acronym>GiST</acronym>.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym>
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
maintained by Teodor Sigaev and Oleg Bartunov. There is more
|
|
information about <acronym>GIN</acronym> on their
|
|
<ulink url="http://www.sai.msu.su/~megera/oddmuse/index.cgi/Gin">website</ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-extensibility">
|
|
<title>Extensibility</title>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym> interface has a high level of abstraction,
|
|
requiring the access method implementer only to implement the semantics of
|
|
the data type being accessed. The <acronym>GIN</acronym> layer itself
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
</para>
|
|
|
|
<para>
|
|
All it takes to get a <acronym>GIN</acronym> access method working is to
|
|
implement four (or five) user-defined methods, which define the behavior of
|
|
keys in the tree and the relationships between keys, indexed values,
|
|
and indexable queries. In short, <acronym>GIN</acronym> combines
|
|
extensibility with generality, code reuse, and a clean interface.
|
|
</para>
|
|
|
|
<para>
|
|
The four methods that an operator class for
|
|
<acronym>GIN</acronym> must provide are:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>int compare(Datum a, Datum b)</term>
|
|
<listitem>
|
|
<para>
|
|
Compares keys (not indexed values!) and returns an integer less than
|
|
zero, zero, or greater than zero, indicating whether the first key is
|
|
less than, equal to, or greater than the second.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum *extractValue(Datum inputValue, int32 *nkeys)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be indexed. The
|
|
number of returned keys must be stored into <literal>*nkeys</>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum *extractQuery(Datum query, int32 *nkeys,
|
|
StrategyNumber n, bool **pmatch)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be queried; that is,
|
|
<literal>query</> is the value on the right-hand side of an
|
|
indexable operator whose left-hand side is the indexed column.
|
|
<literal>n</> is the strategy number of the operator within the
|
|
operator class (see <xref linkend="xindex-strategies">).
|
|
Often, <function>extractQuery</> will need
|
|
to consult <literal>n</> to determine the data type of
|
|
<literal>query</> and the key values that need to be extracted.
|
|
The number of returned keys must be stored into <literal>*nkeys</>.
|
|
If the query contains no keys then <function>extractQuery</>
|
|
should store 0 or -1 into <literal>*nkeys</>, depending on the
|
|
semantics of the operator. 0 means that every
|
|
value matches the <literal>query</> and a sequential scan should be
|
|
produced. -1 means nothing can match the <literal>query</>.
|
|
<literal>pmatch</> is an output argument for use when partial match
|
|
is supported. To use it, <function>extractQuery</> must allocate
|
|
an array of <literal>*nkeys</> booleans and store its address at
|
|
<literal>*pmatch</>. Each element of the array should be set to TRUE
|
|
if the corresponding key requires partial match, FALSE if not.
|
|
If <literal>*pmatch</> is set to NULL then GIN assumes partial match
|
|
is not required. The variable is initialized to NULL before call,
|
|
so this argument can simply be ignored by operator classes that do
|
|
not support partial match.
|
|
</para>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>bool consistent(bool check[], StrategyNumber n, Datum query, bool *recheck)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns TRUE if the indexed value satisfies the query operator with
|
|
strategy number <literal>n</> (or might satisfy, if the recheck
|
|
indication is returned). The <literal>check</> array has
|
|
the same length as the number of keys previously returned by
|
|
<function>extractQuery</> for this query. Each element of the
|
|
<literal>check</> array is TRUE if the indexed value contains the
|
|
corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
|
|
<function>extractQuery</> result array is present in the indexed value.
|
|
The original <literal>query</> datum (not the extracted key array!) is
|
|
passed in case the <function>consistent</> method needs to consult it.
|
|
On success, <literal>*recheck</> should be set to TRUE if the heap
|
|
tuple needs to be rechecked against the query operator, or FALSE if
|
|
the index test is exact.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
<para>
|
|
Optionally, an operator class for
|
|
<acronym>GIN</acronym> can supply a fifth method:
|
|
</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term>int comparePartial(Datum partial_key, Datum key, StrategyNumber n)</term>
|
|
<listitem>
|
|
<para>
|
|
Compare a partial-match query to an index key. Returns an integer
|
|
whose sign indicates the result: less than zero means the index key
|
|
does not match the query, but the index scan should continue; zero
|
|
means that the index key does match the query; greater than zero
|
|
indicates that the index scan should stop because no more matches
|
|
are possible. The strategy number <literal>n</> of the operator
|
|
that generated the partial match query is provided, in case its
|
|
semantics are needed to determine when to end the scan.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
<para>
|
|
To support <quote>partial match</> queries, an operator class must
|
|
provide the <function>comparePartial</> method, and its
|
|
<function>extractQuery</> method must set the <literal>pmatch</>
|
|
parameter when a partial-match query is encountered. See
|
|
<xref linkend="gin-partial-match"> for details.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-implementation">
|
|
<title>Implementation</title>
|
|
|
|
<para>
|
|
Internally, a <acronym>GIN</acronym> index contains a B-tree index
|
|
constructed over keys, where each key is an element of the indexed value
|
|
(a member of an array, for example) and where each tuple in a leaf page is
|
|
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
|
|
list of heap pointers (PL, posting list) if the list is small enough.
|
|
</para>
|
|
|
|
<sect2 id="gin-partial-match">
|
|
<title>Partial match algorithm</title>
|
|
|
|
<para>
|
|
GIN can support <quote>partial match</> queries, in which the query
|
|
does not determine an exact match for one or more keys, but the possible
|
|
matches fall within a reasonably narrow range of key values (within the
|
|
key sorting order determined by the <function>compare</> support method).
|
|
The <function>extractQuery</> method, instead of returning a key value
|
|
to be matched exactly, returns a key value that is the lower bound of
|
|
the range to be searched, and sets the <literal>pmatch</> flag true.
|
|
The key range is then searched using the <function>comparePartial</>
|
|
method. <function>comparePartial</> must return zero for an actual
|
|
match, less than zero for a non-match that is still within the range
|
|
to be searched, or greater than zero if the index key is past the range
|
|
that could match.
|
|
</para>
|
|
|
|
<para>
|
|
During a partial-match scan, all <literal>itemPointer</>s for matching keys
|
|
are OR'ed into a <literal>TIDBitmap</>.
|
|
The scan fails if the <literal>TIDBitmap</> becomes lossy.
|
|
In this case an error message will be reported with advice
|
|
to increase <literal>work_mem</>.
|
|
</para>
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-tips">
|
|
<title>GIN tips and tricks</title>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>Create vs insert</term>
|
|
<listitem>
|
|
<para>
|
|
In most cases, insertion into a <acronym>GIN</acronym> index is slow
|
|
due to the likelihood of many keys being inserted for each value.
|
|
So, for bulk insertions into a table it is advisable to drop the GIN
|
|
index and recreate it after finishing bulk insertion.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-maintenance-work-mem"></term>
|
|
<listitem>
|
|
<para>
|
|
Build time for a <acronym>GIN</acronym> index is very sensitive to
|
|
the <varname>maintenance_work_mem</> setting; it doesn't pay to
|
|
skimp on work memory during index creation.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-gin-fuzzy-search-limit"></term>
|
|
<listitem>
|
|
<para>
|
|
The primary goal of developing <acronym>GIN</acronym> indexes was
|
|
to create support for highly scalable, full-text search in
|
|
<productname>PostgreSQL</productname>, and there are often situations when
|
|
a full-text search returns a very large set of results. Moreover, this
|
|
often happens when the query contains very frequent words, so that the
|
|
large result set is not even useful. Since reading many
|
|
tuples from the disk and sorting them could take a lot of time, this is
|
|
unacceptable for production. (Note that the index search itself is very
|
|
fast.)
|
|
</para>
|
|
<para>
|
|
To facilitate controlled execution of such queries
|
|
<acronym>GIN</acronym> has a configurable soft upper limit on the
|
|
number of rows returned, the
|
|
<varname>gin_fuzzy_search_limit</varname> configuration parameter.
|
|
It is set to 0 (meaning no limit) by default.
|
|
If a non-zero limit is set, then the returned set is a subset of
|
|
the whole result set, chosen at random.
|
|
</para>
|
|
<para>
|
|
<quote>Soft</quote> means that the actual number of returned results
|
|
could differ slightly from the specified limit, depending on the query
|
|
and the quality of the system's random number generator.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-limit">
|
|
<title>Limitations</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> doesn't support full index scans: because there are
|
|
often many keys per value, each heap pointer would be returned many times,
|
|
and there is no easy way to prevent this.
|
|
</para>
|
|
|
|
<para>
|
|
When <function>extractQuery</function> returns zero keys,
|
|
<acronym>GIN</acronym> will emit an error. Depending on the operator,
|
|
a void query might match all, some, or none of the indexed values (for
|
|
example, every array contains the empty array, but does not overlap the
|
|
empty array), and <acronym>GIN</acronym> cannot determine the correct
|
|
answer, nor produce a full-index-scan result if it could determine that
|
|
that was correct.
|
|
</para>
|
|
|
|
<para>
|
|
It is not an error for <function>extractValue</> to return zero keys,
|
|
but in this case the indexed value will be unrepresented in the index.
|
|
This is another reason why full index scan is not useful — it would
|
|
miss such rows.
|
|
</para>
|
|
|
|
<para>
|
|
It is possible for an operator class to circumvent the restriction against
|
|
full index scan. To do that, <function>extractValue</> must return at least
|
|
one (possibly dummy) key for every indexed value, and
|
|
<function>extractQuery</function> must convert an unrestricted search into
|
|
a partial-match query that will scan the whole index. This is inefficient
|
|
but might be necessary to avoid corner-case failures with operators such
|
|
as LIKE. Note however that failure could still occur if the intermediate
|
|
<literal>TIDBitmap</> becomes lossy.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-examples">
|
|
<title>Examples</title>
|
|
|
|
<para>
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
|
<acronym>GIN</acronym> operator classes for <type>tsvector</> and
|
|
for one-dimensional arrays of all internal types. Prefix searching in
|
|
<type>tsvector</> is implemented using the <acronym>GIN</> partial match
|
|
feature.
|
|
The following <filename>contrib</> modules also contain
|
|
<acronym>GIN</acronym> operator classes:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>hstore</term>
|
|
<listitem>
|
|
<para>Module for storing (key, value) pairs</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>intarray</term>
|
|
<listitem>
|
|
<para>Enhanced support for int4[]</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>pg_trgm</term>
|
|
<listitem>
|
|
<para>Text similarity using trigram matching</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</sect1>
|
|
|
|
</chapter>
|