944 lines
34 KiB
Plaintext
944 lines
34 KiB
Plaintext
<!-- doc/src/sgml/gin.sgml -->
|
|
|
|
<chapter id="GIN">
|
|
<title>GIN Indexes</title>
|
|
|
|
<indexterm>
|
|
<primary>index</primary>
|
|
<secondary>GIN</secondary>
|
|
</indexterm>
|
|
|
|
<sect1 id="gin-intro">
|
|
<title>Introduction</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> stands for Generalized Inverted Index.
|
|
<acronym>GIN</acronym> is designed for handling cases where the items
|
|
to be indexed are composite values, and the queries to be handled by
|
|
the index need to search for element values that appear within
|
|
the composite items. For example, the items could be documents,
|
|
and the queries could be searches for documents containing specific words.
|
|
</para>
|
|
|
|
<para>
|
|
We use the word <firstterm>item</> to refer to a composite value that
|
|
is to be indexed, and the word <firstterm>key</> to refer to an element
|
|
value. <acronym>GIN</acronym> always stores and searches for keys,
|
|
not item values per se.
|
|
</para>
|
|
|
|
<para>
|
|
A <acronym>GIN</acronym> index stores a set of (key, posting list) pairs,
|
|
where a <firstterm>posting list</> is a set of row IDs in which the key
|
|
occurs. The same row ID can appear in multiple posting lists, since
|
|
an item can contain more than one key. Each key value is stored only
|
|
once, so a <acronym>GIN</acronym> index is very compact for cases
|
|
where the same key appears many times.
|
|
</para>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> is generalized in the sense that the
|
|
<acronym>GIN</acronym> access method code does not need to know the
|
|
specific operations that it accelerates.
|
|
Instead, it uses custom strategies defined for particular data types.
|
|
The strategy defines how keys are extracted from indexed items and
|
|
query conditions, and how to determine whether a row that contains
|
|
some of the key values in a query actually satisfies the query.
|
|
</para>
|
|
|
|
<para>
|
|
One advantage of <acronym>GIN</acronym> is that it allows the development
|
|
of custom data types with the appropriate access methods, by
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
This is much the same advantage as using <acronym>GiST</acronym>.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym>
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
maintained by Teodor Sigaev and Oleg Bartunov. There is more
|
|
information about <acronym>GIN</acronym> on their
|
|
<ulink url="http://www.sai.msu.su/~megera/wiki/Gin">website</ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-builtin-opclasses">
|
|
<title>Built-in Operator Classes</title>
|
|
|
|
<para>
|
|
The core <productname>PostgreSQL</> distribution
|
|
includes the <acronym>GIN</acronym> operator classes shown in
|
|
<xref linkend="gin-builtin-opclasses-table">.
|
|
(Some of the optional modules described in <xref linkend="contrib">
|
|
provide additional <acronym>GIN</acronym> operator classes.)
|
|
</para>
|
|
|
|
<table id="gin-builtin-opclasses-table">
|
|
<title>Built-in <acronym>GIN</acronym> Operator Classes</title>
|
|
<tgroup cols="3">
|
|
<thead>
|
|
<row>
|
|
<entry>Name</entry>
|
|
<entry>Indexed Data Type</entry>
|
|
<entry>Indexable Operators</entry>
|
|
</row>
|
|
</thead>
|
|
<tbody>
|
|
<row>
|
|
<entry><literal>_abstime_ops</></entry>
|
|
<entry><type>abstime[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_bit_ops</></entry>
|
|
<entry><type>bit[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_bool_ops</></entry>
|
|
<entry><type>boolean[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_bpchar_ops</></entry>
|
|
<entry><type>character[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_bytea_ops</></entry>
|
|
<entry><type>bytea[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_char_ops</></entry>
|
|
<entry><type>"char"[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_cidr_ops</></entry>
|
|
<entry><type>cidr[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_date_ops</></entry>
|
|
<entry><type>date[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_float4_ops</></entry>
|
|
<entry><type>float4[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_float8_ops</></entry>
|
|
<entry><type>float8[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_inet_ops</></entry>
|
|
<entry><type>inet[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_int2_ops</></entry>
|
|
<entry><type>smallint[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_int4_ops</></entry>
|
|
<entry><type>integer[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_int8_ops</></entry>
|
|
<entry><type>bigint[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_interval_ops</></entry>
|
|
<entry><type>interval[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_macaddr_ops</></entry>
|
|
<entry><type>macaddr[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_money_ops</></entry>
|
|
<entry><type>money[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_name_ops</></entry>
|
|
<entry><type>name[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_numeric_ops</></entry>
|
|
<entry><type>numeric[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_oid_ops</></entry>
|
|
<entry><type>oid[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_oidvector_ops</></entry>
|
|
<entry><type>oidvector[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_reltime_ops</></entry>
|
|
<entry><type>reltime[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_text_ops</></entry>
|
|
<entry><type>text[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_time_ops</></entry>
|
|
<entry><type>time[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_timestamp_ops</></entry>
|
|
<entry><type>timestamp[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_timestamptz_ops</></entry>
|
|
<entry><type>timestamp with time zone[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_timetz_ops</></entry>
|
|
<entry><type>time with time zone[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_tinterval_ops</></entry>
|
|
<entry><type>tinterval[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_varbit_ops</></entry>
|
|
<entry><type>bit varying[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>_varchar_ops</></entry>
|
|
<entry><type>character varying[]</></entry>
|
|
<entry>
|
|
<literal>&&</>
|
|
<literal><@</>
|
|
<literal>=</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>jsonb_ops</></entry>
|
|
<entry><type>jsonb</></entry>
|
|
<entry>
|
|
<literal>?</>
|
|
<literal>?&</>
|
|
<literal>?|</>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>jsonb_path_ops</></entry>
|
|
<entry><type>jsonb</></entry>
|
|
<entry>
|
|
<literal>@></>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry><literal>tsvector_ops</></entry>
|
|
<entry><type>tsvector</></entry>
|
|
<entry>
|
|
<literal>@@</>
|
|
<literal>@@@</>
|
|
</entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
Of the two operator classes for type <type>jsonb</>, <literal>jsonb_ops</>
|
|
is the default. <literal>jsonb_path_ops</> supports fewer operators but
|
|
offers better performance for those operators.
|
|
See <xref linkend="json-indexing"> for details.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-extensibility">
|
|
<title>Extensibility</title>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym> interface has a high level of abstraction,
|
|
requiring the access method implementer only to implement the semantics of
|
|
the data type being accessed. The <acronym>GIN</acronym> layer itself
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
</para>
|
|
|
|
<para>
|
|
All it takes to get a <acronym>GIN</acronym> access method working is to
|
|
implement a few user-defined methods, which define the behavior of
|
|
keys in the tree and the relationships between keys, indexed items,
|
|
and indexable queries. In short, <acronym>GIN</acronym> combines
|
|
extensibility with generality, code reuse, and a clean interface.
|
|
</para>
|
|
|
|
<para>
|
|
There are three methods that an operator class for
|
|
<acronym>GIN</acronym> must provide:
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><function>int compare(Datum a, Datum b)</></term>
|
|
<listitem>
|
|
<para>
|
|
Compares two keys (not indexed items!) and returns an integer less than
|
|
zero, zero, or greater than zero, indicating whether the first key is
|
|
less than, equal to, or greater than the second. Null keys are never
|
|
passed to this function.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><function>Datum *extractValue(Datum itemValue, int32 *nkeys,
|
|
bool **nullFlags)</></term>
|
|
<listitem>
|
|
<para>
|
|
Returns a palloc'd array of keys given an item to be indexed. The
|
|
number of returned keys must be stored into <literal>*nkeys</>.
|
|
If any of the keys can be null, also palloc an array of
|
|
<literal>*nkeys</> <type>bool</type> fields, store its address at
|
|
<literal>*nullFlags</>, and set these null flags as needed.
|
|
<literal>*nullFlags</> can be left <symbol>NULL</symbol> (its initial value)
|
|
if all keys are non-null.
|
|
The return value can be <symbol>NULL</symbol> if the item contains no keys.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><function>Datum *extractQuery(Datum query, int32 *nkeys,
|
|
StrategyNumber n, bool **pmatch, Pointer **extra_data,
|
|
bool **nullFlags, int32 *searchMode)</></term>
|
|
<listitem>
|
|
<para>
|
|
Returns a palloc'd array of keys given a value to be queried; that is,
|
|
<literal>query</> is the value on the right-hand side of an
|
|
indexable operator whose left-hand side is the indexed column.
|
|
<literal>n</> is the strategy number of the operator within the
|
|
operator class (see <xref linkend="xindex-strategies">).
|
|
Often, <function>extractQuery</> will need
|
|
to consult <literal>n</> to determine the data type of
|
|
<literal>query</> and the method it should use to extract key values.
|
|
The number of returned keys must be stored into <literal>*nkeys</>.
|
|
If any of the keys can be null, also palloc an array of
|
|
<literal>*nkeys</> <type>bool</type> fields, store its address at
|
|
<literal>*nullFlags</>, and set these null flags as needed.
|
|
<literal>*nullFlags</> can be left <symbol>NULL</symbol> (its initial value)
|
|
if all keys are non-null.
|
|
The return value can be <symbol>NULL</symbol> if the <literal>query</> contains no keys.
|
|
</para>
|
|
|
|
<para>
|
|
<literal>searchMode</> is an output argument that allows
|
|
<function>extractQuery</> to specify details about how the search
|
|
will be done.
|
|
If <literal>*searchMode</> is set to
|
|
<literal>GIN_SEARCH_MODE_DEFAULT</> (which is the value it is
|
|
initialized to before call), only items that match at least one of
|
|
the returned keys are considered candidate matches.
|
|
If <literal>*searchMode</> is set to
|
|
<literal>GIN_SEARCH_MODE_INCLUDE_EMPTY</>, then in addition to items
|
|
containing at least one matching key, items that contain no keys at
|
|
all are considered candidate matches. (This mode is useful for
|
|
implementing is-subset-of operators, for example.)
|
|
If <literal>*searchMode</> is set to <literal>GIN_SEARCH_MODE_ALL</>,
|
|
then all non-null items in the index are considered candidate
|
|
matches, whether they match any of the returned keys or not. (This
|
|
mode is much slower than the other two choices, since it requires
|
|
scanning essentially the entire index, but it may be necessary to
|
|
implement corner cases correctly. An operator that needs this mode
|
|
in most cases is probably not a good candidate for a GIN operator
|
|
class.)
|
|
The symbols to use for setting this mode are defined in
|
|
<filename>access/gin.h</>.
|
|
</para>
|
|
|
|
<para>
|
|
<literal>pmatch</> is an output argument for use when partial match
|
|
is supported. To use it, <function>extractQuery</> must allocate
|
|
an array of <literal>*nkeys</> booleans and store its address at
|
|
<literal>*pmatch</>. Each element of the array should be set to TRUE
|
|
if the corresponding key requires partial match, FALSE if not.
|
|
If <literal>*pmatch</> is set to <symbol>NULL</symbol> then GIN assumes partial match
|
|
is not required. The variable is initialized to <symbol>NULL</symbol> before call,
|
|
so this argument can simply be ignored by operator classes that do
|
|
not support partial match.
|
|
</para>
|
|
|
|
<para>
|
|
<literal>extra_data</> is an output argument that allows
|
|
<function>extractQuery</> to pass additional data to the
|
|
<function>consistent</> and <function>comparePartial</> methods.
|
|
To use it, <function>extractQuery</> must allocate
|
|
an array of <literal>*nkeys</> pointers and store its address at
|
|
<literal>*extra_data</>, then store whatever it wants to into the
|
|
individual pointers. The variable is initialized to <symbol>NULL</symbol> before
|
|
call, so this argument can simply be ignored by operator classes that
|
|
do not require extra data. If <literal>*extra_data</> is set, the
|
|
whole array is passed to the <function>consistent</> method, and
|
|
the appropriate element to the <function>comparePartial</> method.
|
|
</para>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
An operator class must also provide a function to check if an indexed item
|
|
matches the query. It comes in two flavors, a boolean <function>consistent</>
|
|
function, and a ternary <function>triConsistent</> function.
|
|
<function>triConsistent</> covers the functionality of both, so providing
|
|
<function>triConsistent</> alone is sufficient. However, if the boolean
|
|
variant is significantly cheaper to calculate, it can be advantageous to
|
|
provide both. If only the boolean variant is provided, some optimizations
|
|
that depend on refuting index items before fetching all the keys are
|
|
disabled.
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><function>bool consistent(bool check[], StrategyNumber n, Datum query,
|
|
int32 nkeys, Pointer extra_data[], bool *recheck,
|
|
Datum queryKeys[], bool nullFlags[])</></term>
|
|
<listitem>
|
|
<para>
|
|
Returns TRUE if an indexed item satisfies the query operator with
|
|
strategy number <literal>n</> (or might satisfy it, if the recheck
|
|
indication is returned). This function does not have direct access
|
|
to the indexed item's value, since <acronym>GIN</acronym> does not
|
|
store items explicitly. Rather, what is available is knowledge
|
|
about which key values extracted from the query appear in a given
|
|
indexed item. The <literal>check</> array has length
|
|
<literal>nkeys</>, which is the same as the number of keys previously
|
|
returned by <function>extractQuery</> for this <literal>query</> datum.
|
|
Each element of the
|
|
<literal>check</> array is TRUE if the indexed item contains the
|
|
corresponding query key, i.e., if (check[i] == TRUE) the i-th key of the
|
|
<function>extractQuery</> result array is present in the indexed item.
|
|
The original <literal>query</> datum is
|
|
passed in case the <function>consistent</> method needs to consult it,
|
|
and so are the <literal>queryKeys[]</> and <literal>nullFlags[]</>
|
|
arrays previously returned by <function>extractQuery</>.
|
|
<literal>extra_data</> is the extra-data array returned by
|
|
<function>extractQuery</>, or <symbol>NULL</symbol> if none.
|
|
</para>
|
|
|
|
<para>
|
|
When <function>extractQuery</> returns a null key in
|
|
<literal>queryKeys[]</>, the corresponding <literal>check[]</> element
|
|
is TRUE if the indexed item contains a null key; that is, the
|
|
semantics of <literal>check[]</> are like <literal>IS NOT DISTINCT
|
|
FROM</>. The <function>consistent</> function can examine the
|
|
corresponding <literal>nullFlags[]</> element if it needs to tell
|
|
the difference between a regular value match and a null match.
|
|
</para>
|
|
|
|
<para>
|
|
On success, <literal>*recheck</> should be set to TRUE if the heap
|
|
tuple needs to be rechecked against the query operator, or FALSE if
|
|
the index test is exact. That is, a FALSE return value guarantees
|
|
that the heap tuple does not match the query; a TRUE return value with
|
|
<literal>*recheck</> set to FALSE guarantees that the heap tuple does
|
|
match the query; and a TRUE return value with
|
|
<literal>*recheck</> set to TRUE means that the heap tuple might match
|
|
the query, so it needs to be fetched and rechecked by evaluating the
|
|
query operator directly against the originally indexed item.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><function>GinTernaryValue triConsistent(GinTernaryValue check[], StrategyNumber n, Datum query,
|
|
int32 nkeys, Pointer extra_data[],
|
|
Datum queryKeys[], bool nullFlags[])</></term>
|
|
<listitem>
|
|
<para>
|
|
<function>triConsistent</> is similar to <function>consistent</>,
|
|
but instead of booleans in the <literal>check</> vector, there are
|
|
three possible values for each
|
|
key: <literal>GIN_TRUE</>, <literal>GIN_FALSE</> and
|
|
<literal>GIN_MAYBE</>. <literal>GIN_FALSE</> and <literal>GIN_TRUE</>
|
|
have the same meaning as regular boolean values, while
|
|
<literal>GIN_MAYBE</> means that the presence of that key is not known.
|
|
When <literal>GIN_MAYBE</> values are present, the function should only
|
|
return <literal>GIN_TRUE</> if the item certainly matches whether or
|
|
not the index item contains the corresponding query keys. Likewise, the
|
|
function must return <literal>GIN_FALSE</> only if the item certainly
|
|
does not match, whether or not it contains the <literal>GIN_MAYBE</>
|
|
keys. If the result depends on the <literal>GIN_MAYBE</> entries, i.e.,
|
|
the match cannot be confirmed or refuted based on the known query keys,
|
|
the function must return <literal>GIN_MAYBE</>.
|
|
</para>
|
|
<para>
|
|
When there are no <literal>GIN_MAYBE</> values in the <literal>check</>
|
|
vector, a <literal>GIN_MAYBE</> return value is the equivalent of
|
|
setting the <literal>recheck</> flag in the
|
|
boolean <function>consistent</> function.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
Optionally, an operator class for <acronym>GIN</acronym> can supply the
|
|
following method:
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><function>int comparePartial(Datum partial_key, Datum key, StrategyNumber n,
|
|
Pointer extra_data)</></term>
|
|
<listitem>
|
|
<para>
|
|
Compare a partial-match query key to an index key. Returns an integer
|
|
whose sign indicates the result: less than zero means the index key
|
|
does not match the query, but the index scan should continue; zero
|
|
means that the index key does match the query; greater than zero
|
|
indicates that the index scan should stop because no more matches
|
|
are possible. The strategy number <literal>n</> of the operator
|
|
that generated the partial match query is provided, in case its
|
|
semantics are needed to determine when to end the scan. Also,
|
|
<literal>extra_data</> is the corresponding element of the extra-data
|
|
array made by <function>extractQuery</>, or <symbol>NULL</symbol> if none.
|
|
Null keys are never passed to this function.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</para>
|
|
|
|
<para>
|
|
To support <quote>partial match</> queries, an operator class must
|
|
provide the <function>comparePartial</> method, and its
|
|
<function>extractQuery</> method must set the <literal>pmatch</>
|
|
parameter when a partial-match query is encountered. See
|
|
<xref linkend="gin-partial-match"> for details.
|
|
</para>
|
|
|
|
<para>
|
|
The actual data types of the various <literal>Datum</> values mentioned
|
|
above vary depending on the operator class. The item values passed to
|
|
<function>extractValue</> are always of the operator class's input type, and
|
|
all key values must be of the class's <literal>STORAGE</> type. The type of
|
|
the <literal>query</> argument passed to <function>extractQuery</>,
|
|
<function>consistent</> and <function>triConsistent</> is whatever is the
|
|
right-hand input type of the class member operator identified by the
|
|
strategy number. This need not be the same as the indexed type, so long as
|
|
key values of the correct type can be extracted from it. However, it is
|
|
recommended that the SQL declarations of these three support functions use
|
|
the opclass's indexed data type for the <literal>query</> argument, even
|
|
though the actual type might be something else depending on the operator.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-implementation">
|
|
<title>Implementation</title>
|
|
|
|
<para>
|
|
Internally, a <acronym>GIN</acronym> index contains a B-tree index
|
|
constructed over keys, where each key is an element of one or more indexed
|
|
items (a member of an array, for example) and where each tuple in a leaf
|
|
page contains either a pointer to a B-tree of heap pointers (a
|
|
<quote>posting tree</>), or a simple list of heap pointers (a <quote>posting
|
|
list</>) when the list is small enough to fit into a single index tuple along
|
|
with the key value.
|
|
</para>
|
|
|
|
<para>
|
|
As of <productname>PostgreSQL</productname> 9.1, null key values can be
|
|
included in the index. Also, placeholder nulls are included in the index
|
|
for indexed items that are null or contain no keys according to
|
|
<function>extractValue</>. This allows searches that should find empty
|
|
items to do so.
|
|
</para>
|
|
|
|
<para>
|
|
Multicolumn <acronym>GIN</acronym> indexes are implemented by building
|
|
a single B-tree over composite values (column number, key value). The
|
|
key values for different columns can be of different types.
|
|
</para>
|
|
|
|
<sect2 id="gin-fast-update">
|
|
<title>GIN Fast Update Technique</title>
|
|
|
|
<para>
|
|
Updating a <acronym>GIN</acronym> index tends to be slow because of the
|
|
intrinsic nature of inverted indexes: inserting or updating one heap row
|
|
can cause many inserts into the index (one for each key extracted
|
|
from the indexed item). As of <productname>PostgreSQL</productname> 8.4,
|
|
<acronym>GIN</> is capable of postponing much of this work by inserting
|
|
new tuples into a temporary, unsorted list of pending entries.
|
|
When the table is vacuumed or autoanalyzed, or when
|
|
<function>gin_clean_pending_list</function> function is called, or if the
|
|
pending list becomes larger than
|
|
<xref linkend="guc-gin-pending-list-limit">, the entries are moved to the
|
|
main <acronym>GIN</acronym> data structure using the same bulk insert
|
|
techniques used during initial index creation. This greatly improves
|
|
<acronym>GIN</acronym> index update speed, even counting the additional
|
|
vacuum overhead. Moreover the overhead work can be done by a background
|
|
process instead of in foreground query processing.
|
|
</para>
|
|
|
|
<para>
|
|
The main disadvantage of this approach is that searches must scan the list
|
|
of pending entries in addition to searching the regular index, and so
|
|
a large list of pending entries will slow searches significantly.
|
|
Another disadvantage is that, while most updates are fast, an update
|
|
that causes the pending list to become <quote>too large</> will incur an
|
|
immediate cleanup cycle and thus be much slower than other updates.
|
|
Proper use of autovacuum can minimize both of these problems.
|
|
</para>
|
|
|
|
<para>
|
|
If consistent response time is more important than update speed,
|
|
use of pending entries can be disabled by turning off the
|
|
<literal>fastupdate</literal> storage parameter for a
|
|
<acronym>GIN</acronym> index. See <xref linkend="sql-createindex">
|
|
for details.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2 id="gin-partial-match">
|
|
<title>Partial Match Algorithm</title>
|
|
|
|
<para>
|
|
GIN can support <quote>partial match</> queries, in which the query
|
|
does not determine an exact match for one or more keys, but the possible
|
|
matches fall within a reasonably narrow range of key values (within the
|
|
key sorting order determined by the <function>compare</> support method).
|
|
The <function>extractQuery</> method, instead of returning a key value
|
|
to be matched exactly, returns a key value that is the lower bound of
|
|
the range to be searched, and sets the <literal>pmatch</> flag true.
|
|
The key range is then scanned using the <function>comparePartial</>
|
|
method. <function>comparePartial</> must return zero for a matching
|
|
index key, less than zero for a non-match that is still within the range
|
|
to be searched, or greater than zero if the index key is past the range
|
|
that could match.
|
|
</para>
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-tips">
|
|
<title>GIN Tips and Tricks</title>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>Create vs. insert</term>
|
|
<listitem>
|
|
<para>
|
|
Insertion into a <acronym>GIN</acronym> index can be slow
|
|
due to the likelihood of many keys being inserted for each item.
|
|
So, for bulk insertions into a table it is advisable to drop the GIN
|
|
index and recreate it after finishing bulk insertion.
|
|
</para>
|
|
|
|
<para>
|
|
As of <productname>PostgreSQL</productname> 8.4, this advice is less
|
|
necessary since delayed indexing is used (see <xref
|
|
linkend="gin-fast-update"> for details). But for very large updates
|
|
it may still be best to drop and recreate the index.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-maintenance-work-mem"></term>
|
|
<listitem>
|
|
<para>
|
|
Build time for a <acronym>GIN</acronym> index is very sensitive to
|
|
the <varname>maintenance_work_mem</> setting; it doesn't pay to
|
|
skimp on work memory during index creation.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-gin-pending-list-limit"></term>
|
|
<listitem>
|
|
<para>
|
|
During a series of insertions into an existing <acronym>GIN</acronym>
|
|
index that has <literal>fastupdate</> enabled, the system will clean up
|
|
the pending-entry list whenever the list grows larger than
|
|
<varname>gin_pending_list_limit</>. To avoid fluctuations in observed
|
|
response time, it's desirable to have pending-list cleanup occur in the
|
|
background (i.e., via autovacuum). Foreground cleanup operations
|
|
can be avoided by increasing <varname>gin_pending_list_limit</>
|
|
or making autovacuum more aggressive.
|
|
However, enlarging the threshold of the cleanup operation means that
|
|
if a foreground cleanup does occur, it will take even longer.
|
|
</para>
|
|
<para>
|
|
<varname>gin_pending_list_limit</> can be overridden for individual
|
|
GIN indexes by changing storage parameters, and which allows each
|
|
GIN index to have its own cleanup threshold.
|
|
For example, it's possible to increase the threshold only for the GIN
|
|
index which can be updated heavily, and decrease it otherwise.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-gin-fuzzy-search-limit"></term>
|
|
<listitem>
|
|
<para>
|
|
The primary goal of developing <acronym>GIN</acronym> indexes was
|
|
to create support for highly scalable full-text search in
|
|
<productname>PostgreSQL</productname>, and there are often situations when
|
|
a full-text search returns a very large set of results. Moreover, this
|
|
often happens when the query contains very frequent words, so that the
|
|
large result set is not even useful. Since reading many
|
|
tuples from the disk and sorting them could take a lot of time, this is
|
|
unacceptable for production. (Note that the index search itself is very
|
|
fast.)
|
|
</para>
|
|
<para>
|
|
To facilitate controlled execution of such queries,
|
|
<acronym>GIN</acronym> has a configurable soft upper limit on the
|
|
number of rows returned: the
|
|
<varname>gin_fuzzy_search_limit</varname> configuration parameter.
|
|
It is set to 0 (meaning no limit) by default.
|
|
If a non-zero limit is set, then the returned set is a subset of
|
|
the whole result set, chosen at random.
|
|
</para>
|
|
<para>
|
|
<quote>Soft</quote> means that the actual number of returned results
|
|
could differ somewhat from the specified limit, depending on the query
|
|
and the quality of the system's random number generator.
|
|
</para>
|
|
<para>
|
|
From experience, values in the thousands (e.g., 5000 — 20000)
|
|
work well.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-limit">
|
|
<title>Limitations</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> assumes that indexable operators are strict. This
|
|
means that <function>extractValue</> will not be called at all on a null
|
|
item value (instead, a placeholder index entry is created automatically),
|
|
and <function>extractQuery</function> will not be called on a null query
|
|
value either (instead, the query is presumed to be unsatisfiable). Note
|
|
however that null key values contained within a non-null composite item
|
|
or query value are supported.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-examples">
|
|
<title>Examples</title>
|
|
|
|
<para>
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
|
<acronym>GIN</acronym> operator classes for <type>tsvector</> and
|
|
for one-dimensional arrays of all internal types. Prefix searching in
|
|
<type>tsvector</> is implemented using the <acronym>GIN</> partial match
|
|
feature.
|
|
The following <filename>contrib</> modules also contain
|
|
<acronym>GIN</acronym> operator classes:
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><filename>btree_gin</></term>
|
|
<listitem>
|
|
<para>B-tree equivalent functionality for several data types</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>hstore</></term>
|
|
<listitem>
|
|
<para>Module for storing (key, value) pairs</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>intarray</></term>
|
|
<listitem>
|
|
<para>Enhanced support for <type>int[]</type></para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>pg_trgm</></term>
|
|
<listitem>
|
|
<para>Text similarity using trigram matching</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</para>
|
|
</sect1>
|
|
|
|
</chapter>
|