GIN Indexes
index
GIN
Introduction
GIN stands for Generalized Inverted Index. It is
an index structure storing a set of (key, posting list) pairs, where
a posting list> is a set of rows in which the key occurs. Each
indexed value can contain many keys, so the same row ID can appear in
multiple posting lists.
It is generalized in the sense that a GIN index
does not need to be aware of the operation that it accelerates.
Instead, it uses custom strategies defined for particular data types.
One advantage of GIN is that it allows the development
of custom data types with the appropriate access methods, by
an expert in the domain of the data type, rather than a database expert.
This is much the same advantage as using GiST.
The GIN
implementation in PostgreSQL is primarily
maintained by Teodor Sigaev and Oleg Bartunov. There is more
information about GIN on their
website.
Extensibility
The GIN interface has a high level of abstraction,
requiring the access method implementer only to implement the semantics of
the data type being accessed. The GIN layer itself
takes care of concurrency, logging and searching the tree structure.
All it takes to get a GIN access method working
is to implement four user-defined methods, which define the behavior of
keys in the tree and the relationships between keys, indexed values,
and indexable queries. In short, GIN combines
extensibility with generality, code reuse, and a clean interface.
The four methods that an index operator class for
GIN must provide are:
int compare(Datum a, Datum b)
Compares keys (not indexed values!) and returns an integer less than
zero, zero, or greater than zero, indicating whether the first key is
less than, equal to, or greater than the second.
Datum* extractValue(Datum inputValue, int32 *nkeys)
Returns an array of keys given a value to be indexed. The
number of returned keys must be stored into *nkeys>.
Datum* extractQuery(Datum query, int32 *nkeys,
StrategyNumber n)
Returns an array of keys given a value to be queried; that is,
query> is the value on the right-hand side of an
indexable operator whose left-hand side is the indexed column.
n> is the strategy number of the operator within the
operator class (see ).
Often, extractQuery> will need
to consult n> to determine the data type of
query> and the key values that need to be extracted.
The number of returned keys must be stored into *nkeys>.
If number of keys is equal to zero then extractQuery>
should store 0 or -1 into *nkeys>. 0 means that any
row matches the query> and sequence scan should be
produced. -1 means nothing can satisfy query>.
Choice of value should be based on semantics meaning of operation with
given strategy number.
bool consistent(bool check[], StrategyNumber n, Datum query)
Returns TRUE if the indexed value satisfies the query operator with
strategy number n> (or would satisfy, if the operator is
marked RECHECK in the operator class). The check> array has
the same length as the number of keys previously returned by
extractQuery> for this query. Each element of the
check> array is TRUE if the indexed value contains the
corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
extractQuery> result array is present in the indexed value.
The original query> datum (not the extracted key array!) is
passed in case the consistent> method needs to consult it.
Implementation
Internally, a GIN index contains a B-tree index
constructed over keys, where each key is an element of the indexed value
(a member of an array, for example) and where each tuple in a leaf page is
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
list of heap pointers (PL, posting list) if the list is small enough.
GIN tips and tricks
Create vs insert
In most cases, insertion into a GIN index is slow
due to the likelihood of many keys being inserted for each value.
So, for bulk insertions into a table it is advisable to drop the GIN
index and recreate it after finishing bulk insertion.
The primary goal of developing GIN indexes was
to create support for highly scalable, full-text search in
PostgreSQL, and there are often situations when
a full-text search returns a very large set of results. Moreover, this
often happens when the query contains very frequent words, so that the
large result set is not even useful. Since reading many
tuples from the disk and sorting them could take a lot of time, this is
unacceptable for production. (Note that the index search itself is very
fast.)
To facilitate controlled execution of such queries
GIN has a configurable soft upper limit on the size
of the returned set, the
gin_fuzzy_search_limit configuration parameter.
It is set to 0 (meaning no limit) by default.
If a non-zero limit is set, then the returned set is a subset of
the whole result set, chosen at random.
Soft
means that the actual number of returned results
could differ slightly from the specified limit, depending on the query
and the quality of the system's random number generator.
Limitations
GIN doesn't support full index scans: because there are
often many keys per value, each heap pointer would be returned many times,
and there is no easy way to prevent this.
When extractQuery returns zero keys,
GIN will emit an error. Depending on the operator,
a void query might match all, some, or none of the indexed values (for
example, every array contains the empty array, but does not overlap the
empty array), and GIN cannot determine the correct
answer, nor produce a full-index-scan result if it could determine that
that was correct.
It is not an error for extractValue> to return zero keys,
but in this case the indexed value will be unrepresented in the index.
This is another reason why full index scan is not useful — it would
miss such rows.
GIN searches keys only by equality matching. This might
be improved in future.
Examples
The PostgreSQL source distribution includes
GIN classes for one-dimensional arrays of all internal
types. The following
contrib> modules also contain GIN
operator classes:
intarray
Enhanced support for int4[]
tsearch2
Support for inverted text indexing. This is much faster for very
large, mostly-static sets of documents.