GIN Indexes
index
GIN
Introduction
GIN stands for Generalized Inverted Index. It is
an index structure storing a set of (key, posting list) pairs, where
'posting list' is a set of rows in which the key occurs. Each
row may contain many keys.
It is generalized in the sense that a GIN index
does not need to be aware of the operation that it accelerates.
Instead, it uses custom strategies defined for particular data types.
One advantage of GIN is that it allows the development
of custom data types with the appropriate access methods, by
an expert in the domain of the data type, rather than a database expert.
This is much the same advantage as using GiST.
The GIN
implementation in PostgreSQL is primarily
maintained by Teodor Sigaev and Oleg Bartunov, and there is more
information on their
website.
Extensibility
The GIN interface has a high level of abstraction,
requiring the access method implementer to only implement the semantics of
the data type being accessed. The GIN layer itself
takes care of concurrency, logging and searching the tree structure.
All it takes to get a GIN access method working
is to implement four user-defined methods, which define the behavior of
keys in the tree. In short, GIN combines extensibility
along with generality, code reuse, and a clean interface.
Implementation
Internally, GIN consists of a B-tree index constructed
over keys, where each key is an element of the indexed value
(element of array, for example) and where each tuple in a leaf page is
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
list of heap pointers (PL, posting list) if the tuple is small enough.
There are four methods that an index operator class for
GIN must provide (prototypes are in pseudocode):
int compare( Datum a, Datum b )
Compares keys (not indexed values!) and returns an integer less than
zero, zero, or greater than zero, indicating whether the first key is
less than, equal to, or greater than the second.
Datum* extractValue(Datum inputValue, uint32 *nkeys)
Returns an array of keys of value to be indexed, nkeys should
contain the number of returned keys.
Datum* extractQuery(Datum query, uint32 nkeys,
StrategyNumber n)
Returns an array of keys of the query to be executed. n contains
the strategy number of the operation
(see ).
Depending on n, query may be different type.
bool consistent( bool check[], StrategyNumber n, Datum query)
Returns TRUE if the indexed value satisfies the query qualifier with
strategy n (or may satisfy in case of RECHECK mark in operator class).
Each element of the check array is TRUE if the indexed value has a
corresponding key in the query: if (check[i] == TRUE ) the i-th key of
the query is present in the indexed value.
GIN tips and trics
Create vs insert
In most cases, insertion into GIN index is slow
due to the likelihood of many keys being inserted for each value.
So, for bulk insertions into a table it is advisable to to drop the GIN
index and recreate it after finishing bulk insertion.
gin_fuzzy_search_limit
The primary goal of developing GIN indices was
support for highly scalable, full-text search in
PostgreSQL and there are often situations when
a full-text search returns a very large set of results. Since reading
tuples from the disk and sorting them could take a lot of time, this is
unacceptable for production. (Note that the index search itself is very
fast.)
Such queries usually contain very frequent words, so the results are not
very helpful. To facilitate execution of such queries
GIN has a configurable soft upper limit of the size
of the returned set, determined by the
gin_fuzzy_search_limit GUC variable. It is set to 0 by
default (no limit).
If a non-zero search limit is set, then the returned set is a subset of
the whole result set, chosen at random.
"Soft" means that the actual number of returned results could slightly
differ from the specified limit, depending on the query and the quality
of the system's random number generator.
Limitations
GIN doesn't support full index scans due to their
extremely inefficiency: because there are often many keys per value,
each heap pointer will returned several times.
When extractQuery returns zero keys, GIN will emit a
error: for different opclasses and strategies the semantic meaning of a void
query may be different (for example, any array contains the void array,
but they don't overlap the void array), and GIN can't
suggest reasonable answer.
GIN searches keys only by equality matching. This may
be improved in future.
Examples
The PostgreSQL source distribution includes
GIN classes for one-dimensional arrays of all internal
types. The following
contrib> modules also contain GIN
operator classes:
intarray
Enhanced support for int4[]
tsearch2
Support for inverted text indexing. This is much faster for very
large, mostly-static sets of documents.