mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-10-04 00:36:50 +02:00
4ab7ea5ace
detection of tabs are added in the future.
261 lines
9.2 KiB
Plaintext
261 lines
9.2 KiB
Plaintext
<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.11 2007/02/16 03:50:29 momjian Exp $ -->
|
|
|
|
<chapter id="GIN">
|
|
<title>GIN Indexes</title>
|
|
|
|
<indexterm>
|
|
<primary>index</primary>
|
|
<secondary>GIN</secondary>
|
|
</indexterm>
|
|
|
|
<sect1 id="gin-intro">
|
|
<title>Introduction</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> stands for Generalized Inverted Index. It is
|
|
an index structure storing a set of (key, posting list) pairs, where
|
|
a <quote>posting list</> is a set of rows in which the key occurs. Each
|
|
indexed value can contain many keys, so the same row ID can appear in
|
|
multiple posting lists.
|
|
</para>
|
|
|
|
<para>
|
|
It is generalized in the sense that a <acronym>GIN</acronym> index
|
|
does not need to be aware of the operation that it accelerates.
|
|
Instead, it uses custom strategies defined for particular data types.
|
|
</para>
|
|
|
|
<para>
|
|
One advantage of <acronym>GIN</acronym> is that it allows the development
|
|
of custom data types with the appropriate access methods, by
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
This is much the same advantage as using <acronym>GiST</acronym>.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym>
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
maintained by Teodor Sigaev and Oleg Bartunov. There is more
|
|
information about <acronym>GIN</acronym> on their
|
|
<ulink url="http://www.sai.msu.su/~megera/oddmuse/index.cgi/Gin">website</ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-extensibility">
|
|
<title>Extensibility</title>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym> interface has a high level of abstraction,
|
|
requiring the access method implementer only to implement the semantics of
|
|
the data type being accessed. The <acronym>GIN</acronym> layer itself
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
</para>
|
|
|
|
<para>
|
|
All it takes to get a <acronym>GIN</acronym> access method working
|
|
is to implement four user-defined methods, which define the behavior of
|
|
keys in the tree and the relationships between keys, indexed values,
|
|
and indexable queries. In short, <acronym>GIN</acronym> combines
|
|
extensibility with generality, code reuse, and a clean interface.
|
|
</para>
|
|
|
|
<para>
|
|
The four methods that an index operator class for
|
|
<acronym>GIN</acronym> must provide are:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>int compare(Datum a, Datum b)</term>
|
|
<listitem>
|
|
<para>
|
|
Compares keys (not indexed values!) and returns an integer less than
|
|
zero, zero, or greater than zero, indicating whether the first key is
|
|
less than, equal to, or greater than the second.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum* extractValue(Datum inputValue, int32 *nkeys)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be indexed. The
|
|
number of returned keys must be stored into <literal>*nkeys</>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum* extractQuery(Datum query, int32 *nkeys,
|
|
StrategyNumber n)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be queried; that is,
|
|
<literal>query</> is the value on the right-hand side of an
|
|
indexable operator whose left-hand side is the indexed column.
|
|
<literal>n</> is the strategy number of the operator within the
|
|
operator class (see <xref linkend="xindex-strategies">).
|
|
Often, <function>extractQuery</> will need
|
|
to consult <literal>n</> to determine the data type of
|
|
<literal>query</> and the key values that need to be extracted.
|
|
The number of returned keys must be stored into <literal>*nkeys</>.
|
|
If number of keys is equal to zero then <function>extractQuery</>
|
|
should store 0 or -1 into <literal>*nkeys</>. 0 means that any
|
|
row matches the <literal>query</> and sequence scan should be
|
|
produced. -1 means nothing can satisfy <literal>query</>.
|
|
Choice of value should be based on semantics meaning of operation with
|
|
given strategy number.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>bool consistent(bool check[], StrategyNumber n, Datum query)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns TRUE if the indexed value satisfies the query operator with
|
|
strategy number <literal>n</> (or would satisfy, if the operator is
|
|
marked RECHECK in the operator class). The <literal>check</> array has
|
|
the same length as the number of keys previously returned by
|
|
<function>extractQuery</> for this query. Each element of the
|
|
<literal>check</> array is TRUE if the indexed value contains the
|
|
corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
|
|
<function>extractQuery</> result array is present in the indexed value.
|
|
The original <literal>query</> datum (not the extracted key array!) is
|
|
passed in case the <function>consistent</> method needs to consult it.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-implementation">
|
|
<title>Implementation</title>
|
|
|
|
<para>
|
|
Internally, a <acronym>GIN</acronym> index contains a B-tree index
|
|
constructed over keys, where each key is an element of the indexed value
|
|
(a member of an array, for example) and where each tuple in a leaf page is
|
|
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
|
|
list of heap pointers (PL, posting list) if the list is small enough.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-tips">
|
|
<title>GIN tips and tricks</title>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>Create vs insert</term>
|
|
<listitem>
|
|
<para>
|
|
In most cases, insertion into a <acronym>GIN</acronym> index is slow
|
|
due to the likelihood of many keys being inserted for each value.
|
|
So, for bulk insertions into a table it is advisable to drop the GIN
|
|
index and recreate it after finishing bulk insertion.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-gin-fuzzy-search-limit"></term>
|
|
<listitem>
|
|
<para>
|
|
The primary goal of developing <acronym>GIN</acronym> indexes was
|
|
to create support for highly scalable, full-text search in
|
|
<productname>PostgreSQL</productname>, and there are often situations when
|
|
a full-text search returns a very large set of results. Moreover, this
|
|
often happens when the query contains very frequent words, so that the
|
|
large result set is not even useful. Since reading many
|
|
tuples from the disk and sorting them could take a lot of time, this is
|
|
unacceptable for production. (Note that the index search itself is very
|
|
fast.)
|
|
</para>
|
|
<para>
|
|
To facilitate controlled execution of such queries
|
|
<acronym>GIN</acronym> has a configurable soft upper limit on the size
|
|
of the returned set, the
|
|
<varname>gin_fuzzy_search_limit</varname> configuration parameter.
|
|
It is set to 0 (meaning no limit) by default.
|
|
If a non-zero limit is set, then the returned set is a subset of
|
|
the whole result set, chosen at random.
|
|
</para>
|
|
<para>
|
|
<quote>Soft</quote> means that the actual number of returned results
|
|
could differ slightly from the specified limit, depending on the query
|
|
and the quality of the system's random number generator.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-limit">
|
|
<title>Limitations</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> doesn't support full index scans: because there are
|
|
often many keys per value, each heap pointer would be returned many times,
|
|
and there is no easy way to prevent this.
|
|
</para>
|
|
|
|
<para>
|
|
When <function>extractQuery</function> returns zero keys,
|
|
<acronym>GIN</acronym> will emit an error. Depending on the operator,
|
|
a void query might match all, some, or none of the indexed values (for
|
|
example, every array contains the empty array, but does not overlap the
|
|
empty array), and <acronym>GIN</acronym> cannot determine the correct
|
|
answer, nor produce a full-index-scan result if it could determine that
|
|
that was correct.
|
|
</para>
|
|
|
|
<para>
|
|
It is not an error for <function>extractValue</> to return zero keys,
|
|
but in this case the indexed value will be unrepresented in the index.
|
|
This is another reason why full index scan is not useful — it would
|
|
miss such rows.
|
|
</para>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> searches keys only by equality matching. This might
|
|
be improved in future.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-examples">
|
|
<title>Examples</title>
|
|
|
|
<para>
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
|
<acronym>GIN</acronym> classes for one-dimensional arrays of all internal
|
|
types. The following
|
|
<filename>contrib</> modules also contain <acronym>GIN</acronym>
|
|
operator classes:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>intarray</term>
|
|
<listitem>
|
|
<para>Enhanced support for int4[]</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>tsearch2</term>
|
|
<listitem>
|
|
<para>Support for inverted text indexing. This is much faster for very
|
|
large, mostly-static sets of documents.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</sect1>
|
|
|
|
</chapter>
|