GIN Indexes
index
GIN
Introduction
GIN stands for Generalized Inverted Index. It is
an index structure storing a set of (key, posting list) pairs, where
a posting list> is a set of rows in which the key occurs. Each
indexed value can contain many keys, so the same row ID can appear in
multiple posting lists.
It is generalized in the sense that a GIN index
does not need to be aware of the operation that it accelerates.
Instead, it uses custom strategies defined for particular data types.
One advantage of GIN is that it allows the development
of custom data types with the appropriate access methods, by
an expert in the domain of the data type, rather than a database expert.
This is much the same advantage as using GiST.
The GIN
implementation in PostgreSQL is primarily
maintained by Teodor Sigaev and Oleg Bartunov. There is more
information about GIN on their
website.
Extensibility
The GIN interface has a high level of abstraction,
requiring the access method implementer only to implement the semantics of
the data type being accessed. The GIN layer itself
takes care of concurrency, logging and searching the tree structure.
All it takes to get a GIN access method working is to
implement four (or five) user-defined methods, which define the behavior of
keys in the tree and the relationships between keys, indexed values,
and indexable queries. In short, GIN combines
extensibility with generality, code reuse, and a clean interface.
The four methods that an operator class for
GIN must provide are:
int compare(Datum a, Datum b)
Compares keys (not indexed values!) and returns an integer less than
zero, zero, or greater than zero, indicating whether the first key is
less than, equal to, or greater than the second.
Datum *extractValue(Datum inputValue, int32 *nkeys)
Returns an array of keys given a value to be indexed. The
number of returned keys must be stored into *nkeys>.
Datum *extractQuery(Datum query, int32 *nkeys,
StrategyNumber n, bool **pmatch)
Returns an array of keys given a value to be queried; that is,
query> is the value on the right-hand side of an
indexable operator whose left-hand side is the indexed column.
n> is the strategy number of the operator within the
operator class (see ).
Often, extractQuery> will need
to consult n> to determine the data type of
query> and the key values that need to be extracted.
The number of returned keys must be stored into *nkeys>.
If the query contains no keys then extractQuery>
should store 0 or -1 into *nkeys>, depending on the
semantics of the operator. 0 means that every
value matches the query> and a sequential scan should be
produced. -1 means nothing can match the query>.
pmatch> is an output argument for use when partial match
is supported. To use it, extractQuery> must allocate
an array of *nkeys> booleans and store its address at
*pmatch>. Each element of the array should be set to TRUE
if the corresponding key requires partial match, FALSE if not.
If *pmatch> is set to NULL then GIN assumes partial match
is not required. The variable is initialized to NULL before call,
so this argument can simply be ignored by operator classes that do
not support partial match.
bool consistent(bool check[], StrategyNumber n, Datum query, bool *recheck)
Returns TRUE if the indexed value satisfies the query operator with
strategy number n> (or might satisfy, if the recheck
indication is returned). The check> array has
the same length as the number of keys previously returned by
extractQuery> for this query. Each element of the
check> array is TRUE if the indexed value contains the
corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
extractQuery> result array is present in the indexed value.
The original query> datum (not the extracted key array!) is
passed in case the consistent> method needs to consult it.
On success, *recheck> should be set to TRUE if the heap
tuple needs to be rechecked against the query operator, or FALSE if
the index test is exact.
Optionally, an operator class for
GIN can supply a fifth method:
int comparePartial(Datum partial_key, Datum key, StrategyNumber n)
Compare a partial-match query to an index key. Returns an integer
whose sign indicates the result: less than zero means the index key
does not match the query, but the index scan should continue; zero
means that the index key does match the query; greater than zero
indicates that the index scan should stop because no more matches
are possible. The strategy number n> of the operator
that generated the partial match query is provided, in case its
semantics are needed to determine when to end the scan.
To support partial match> queries, an operator class must
provide the comparePartial> method, and its
extractQuery> method must set the pmatch>
parameter when a partial-match query is encountered. See
for details.
Implementation
Internally, a GIN index contains a B-tree index
constructed over keys, where each key is an element of the indexed value
(a member of an array, for example) and where each tuple in a leaf page is
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
list of heap pointers (PL, posting list) if the list is small enough.
Partial match algorithm
GIN can support partial match> queries, in which the query
does not determine an exact match for one or more keys, but the possible
matches fall within a reasonably narrow range of key values (within the
key sorting order determined by the compare> support method).
The extractQuery> method, instead of returning a key value
to be matched exactly, returns a key value that is the lower bound of
the range to be searched, and sets the pmatch> flag true.
The key range is then searched using the comparePartial>
method. comparePartial> must return zero for an actual
match, less than zero for a non-match that is still within the range
to be searched, or greater than zero if the index key is past the range
that could match.
During a partial-match scan, all itemPointer>s for matching keys
are OR'ed into a TIDBitmap>.
The scan fails if the TIDBitmap> becomes lossy.
In this case an error message will be reported with advice
to increase work_mem>.
GIN tips and tricks
Create vs insert
In most cases, insertion into a GIN index is slow
due to the likelihood of many keys being inserted for each value.
So, for bulk insertions into a table it is advisable to drop the GIN
index and recreate it after finishing bulk insertion.
Build time for a GIN index is very sensitive to
the maintenance_work_mem> setting; it doesn't pay to
skimp on work memory during index creation.
The primary goal of developing GIN indexes was
to create support for highly scalable, full-text search in
PostgreSQL, and there are often situations when
a full-text search returns a very large set of results. Moreover, this
often happens when the query contains very frequent words, so that the
large result set is not even useful. Since reading many
tuples from the disk and sorting them could take a lot of time, this is
unacceptable for production. (Note that the index search itself is very
fast.)
To facilitate controlled execution of such queries
GIN has a configurable soft upper limit on the
number of rows returned, the
gin_fuzzy_search_limit configuration parameter.
It is set to 0 (meaning no limit) by default.
If a non-zero limit is set, then the returned set is a subset of
the whole result set, chosen at random.
Soft
means that the actual number of returned results
could differ slightly from the specified limit, depending on the query
and the quality of the system's random number generator.
Limitations
GIN doesn't support full index scans: because there are
often many keys per value, each heap pointer would be returned many times,
and there is no easy way to prevent this.
When extractQuery returns zero keys,
GIN will emit an error. Depending on the operator,
a void query might match all, some, or none of the indexed values (for
example, every array contains the empty array, but does not overlap the
empty array), and GIN cannot determine the correct
answer, nor produce a full-index-scan result if it could determine that
that was correct.
It is not an error for extractValue> to return zero keys,
but in this case the indexed value will be unrepresented in the index.
This is another reason why full index scan is not useful — it would
miss such rows.
It is possible for an operator class to circumvent the restriction against
full index scan. To do that, extractValue> must return at least
one (possibly dummy) key for every indexed value, and
extractQuery must convert an unrestricted search into
a partial-match query that will scan the whole index. This is inefficient
but might be necessary to avoid corner-case failures with operators such
as LIKE. Note however that failure could still occur if the intermediate
TIDBitmap> becomes lossy.
Examples
The PostgreSQL source distribution includes
GIN operator classes for tsvector> and
for one-dimensional arrays of all internal types. Prefix searching in
tsvector> is implemented using the GIN> partial match
feature.
The following contrib> modules also contain
GIN operator classes:
hstore
Module for storing (key, value) pairs
intarray
Enhanced support for int4[]
pg_trgm
Text similarity using trigram matching