2010-09-20 22:08:53 +02:00
|
|
|
<!-- doc/src/sgml/gist.sgml -->
|
2003-09-29 20:18:35 +02:00
|
|
|
|
2005-01-08 23:13:38 +01:00
|
|
|
<chapter id="GiST">
|
2003-10-31 23:41:21 +01:00
|
|
|
<title>GiST Indexes</title>
|
|
|
|
|
2005-01-08 23:13:38 +01:00
|
|
|
<indexterm>
|
|
|
|
<primary>index</primary>
|
|
|
|
<secondary>GiST</secondary>
|
|
|
|
</indexterm>
|
2005-11-07 18:36:47 +01:00
|
|
|
|
|
|
|
<sect1 id="gist-intro">
|
|
|
|
<title>Introduction</title>
|
|
|
|
|
|
|
|
<para>
|
2003-10-31 23:41:21 +01:00
|
|
|
<acronym>GiST</acronym> stands for Generalized Search Tree. It is a
|
|
|
|
balanced, tree-structured access method, that acts as a base template in
|
2005-11-07 18:36:47 +01:00
|
|
|
which to implement arbitrary indexing schemes. B-trees, R-trees and many
|
2003-10-31 23:41:21 +01:00
|
|
|
other indexing schemes can be implemented in <acronym>GiST</acronym>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
One advantage of <acronym>GiST</acronym> is that it allows the development
|
|
|
|
of custom data types with the appropriate access methods, by
|
|
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2009-06-12 21:48:53 +02:00
|
|
|
Some of the information here is derived from the University of California
|
|
|
|
at Berkeley's GiST Indexing Project
|
|
|
|
<ulink url="http://gist.cs.berkeley.edu/">web site</ulink> and
|
|
|
|
Marcel Kornacker's thesis,
|
2005-10-21 15:59:05 +02:00
|
|
|
<ulink url="http://www.sai.msu.su/~megera/postgres/gist/papers/concurrency/access-methods-for-next-generation.pdf.gz">
|
2009-06-12 21:48:53 +02:00
|
|
|
Access Methods for Next-Generation Database Systems</ulink>.
|
2005-06-29 03:23:49 +02:00
|
|
|
The <acronym>GiST</acronym>
|
2003-10-31 23:41:21 +01:00
|
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
|
|
maintained by Teodor Sigaev and Oleg Bartunov, and there is more
|
2005-06-29 03:23:49 +02:00
|
|
|
information on their
|
2009-06-12 21:48:53 +02:00
|
|
|
<ulink url="http://www.sai.msu.su/~megera/postgres/gist/">web site</ulink>.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
2005-10-21 03:41:28 +02:00
|
|
|
<sect1 id="gist-extensibility">
|
2003-10-31 23:41:21 +01:00
|
|
|
<title>Extensibility</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Traditionally, implementing a new index access method meant a lot of
|
|
|
|
difficult work. It was necessary to understand the inner workings of the
|
|
|
|
database, such as the lock manager and Write-Ahead Log. The
|
|
|
|
<acronym>GiST</acronym> interface has a high level of abstraction,
|
2009-06-12 21:48:53 +02:00
|
|
|
requiring the access method implementer only to implement the semantics of
|
2003-10-31 23:41:21 +01:00
|
|
|
the data type being accessed. The <acronym>GiST</acronym> layer itself
|
|
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
2003-10-31 23:41:21 +01:00
|
|
|
<para>
|
|
|
|
This extensibility should not be confused with the extensibility of the
|
|
|
|
other standard search trees in terms of the data they can handle. For
|
2005-11-07 18:36:47 +01:00
|
|
|
example, <productname>PostgreSQL</productname> supports extensible B-trees
|
|
|
|
and hash indexes. That means that you can use
|
|
|
|
<productname>PostgreSQL</productname> to build a B-tree or hash over any
|
|
|
|
data type you want. But B-trees only support range predicates
|
2003-10-31 23:41:21 +01:00
|
|
|
(<literal><</literal>, <literal>=</literal>, <literal>></literal>),
|
2005-11-07 18:36:47 +01:00
|
|
|
and hash indexes only support equality queries.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
2003-10-31 23:41:21 +01:00
|
|
|
<para>
|
|
|
|
So if you index, say, an image collection with a
|
2005-11-07 18:36:47 +01:00
|
|
|
<productname>PostgreSQL</productname> B-tree, you can only issue queries
|
2003-10-31 23:41:21 +01:00
|
|
|
such as <quote>is imagex equal to imagey</quote>, <quote>is imagex less
|
2009-06-12 21:48:53 +02:00
|
|
|
than imagey</quote> and <quote>is imagex greater than imagey</quote>.
|
2003-10-31 23:41:21 +01:00
|
|
|
Depending on how you define <quote>equals</quote>, <quote>less than</quote>
|
|
|
|
and <quote>greater than</quote> in this context, this could be useful.
|
|
|
|
However, by using a <acronym>GiST</acronym> based index, you could create
|
|
|
|
ways to ask domain-specific questions, perhaps <quote>find all images of
|
|
|
|
horses</quote> or <quote>find all over-exposed images</quote>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
All it takes to get a <acronym>GiST</acronym> access method up and running
|
2010-12-04 05:49:06 +01:00
|
|
|
is to implement several user-defined methods, which define the behavior of
|
2003-10-31 23:41:21 +01:00
|
|
|
keys in the tree. Of course these methods have to be pretty fancy to
|
2005-11-07 18:36:47 +01:00
|
|
|
support fancy queries, but for all the standard queries (B-trees,
|
2003-10-31 23:41:21 +01:00
|
|
|
R-trees, etc.) they're relatively straightforward. In short,
|
|
|
|
<acronym>GiST</acronym> combines extensibility along with generality, code
|
|
|
|
reuse, and a clean interface.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
There are seven methods that an index operator class for
|
2010-12-04 05:49:06 +01:00
|
|
|
<acronym>GiST</acronym> must provide, and an eighth that is optional.
|
|
|
|
Correctness of the index is ensured
|
2009-06-12 21:48:53 +02:00
|
|
|
by proper implementation of the <function>same</>, <function>consistent</>
|
|
|
|
and <function>union</> methods, while efficiency (size and speed) of the
|
|
|
|
index will depend on the <function>penalty</> and <function>picksplit</>
|
|
|
|
methods.
|
2010-12-04 05:49:06 +01:00
|
|
|
The remaining two basic methods are <function>compress</> and
|
2009-06-12 21:48:53 +02:00
|
|
|
<function>decompress</>, which allow an index to have internal tree data of
|
|
|
|
a different type than the data it indexes. The leaves are to be of the
|
|
|
|
indexed data type, while the other tree nodes can be of any C struct (but
|
2010-08-17 06:37:21 +02:00
|
|
|
you still have to follow <productname>PostgreSQL</> data type rules here,
|
2009-06-12 21:48:53 +02:00
|
|
|
see about <literal>varlena</> for variable sized data). If the tree's
|
|
|
|
internal data type exists at the SQL level, the <literal>STORAGE</> option
|
|
|
|
of the <command>CREATE OPERATOR CLASS</> command can be used.
|
2010-12-04 05:49:06 +01:00
|
|
|
The optional eighth method is <function>distance</>, which is needed
|
|
|
|
if the operator class wishes to support ordered scans (nearest-neighbor
|
|
|
|
searches).
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>consistent</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
2009-06-12 21:48:53 +02:00
|
|
|
Given an index entry <literal>p</> and a query value <literal>q</>,
|
|
|
|
this function determines whether the index entry is
|
|
|
|
<quote>consistent</> with the query; that is, could the predicate
|
|
|
|
<quote><replaceable>indexed_column</>
|
|
|
|
<replaceable>indexable_operator</> <literal>q</></quote> be true for
|
|
|
|
any row represented by the index entry? For a leaf index entry this is
|
|
|
|
equivalent to testing the indexable condition, while for an internal
|
|
|
|
tree node this determines whether it is necessary to scan the subtree
|
|
|
|
of the index represented by the tree node. When the result is
|
|
|
|
<literal>true</>, a <literal>recheck</> flag must also be returned.
|
|
|
|
This indicates whether the predicate is certainly true or only possibly
|
|
|
|
true. If <literal>recheck</> = <literal>false</> then the index has
|
|
|
|
tested the predicate condition exactly, whereas if <literal>recheck</>
|
|
|
|
= <literal>true</> the row is only a candidate match. In that case the
|
|
|
|
system will automatically evaluate the
|
|
|
|
<replaceable>indexable_operator</> against the actual row value to see
|
|
|
|
if it is really a match. This convention allows
|
|
|
|
<acronym>GiST</acronym> to support both lossless and lossy index
|
|
|
|
structures.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_consistent(internal, data_type, smallint, oid, internal)
|
|
|
|
RETURNS bool
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_consistent(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_consistent);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_consistent(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GISTENTRY *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
|
|
|
|
data_type *query = PG_GETARG_DATA_TYPE_P(1);
|
|
|
|
StrategyNumber strategy = (StrategyNumber) PG_GETARG_UINT16(2);
|
|
|
|
/* Oid subtype = PG_GETARG_OID(3); */
|
|
|
|
bool *recheck = (bool *) PG_GETARG_POINTER(4);
|
|
|
|
data_type *key = DatumGetDataType(entry->key);
|
|
|
|
bool retval;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* determine return value as a function of strategy, key and query.
|
|
|
|
*
|
|
|
|
* Use GIST_LEAF(entry) to know where you're called in the index tree,
|
|
|
|
* which comes handy when supporting the = operator for example (you could
|
|
|
|
* check for non empty union() in non-leaf nodes and equality in leaf
|
|
|
|
* nodes).
|
|
|
|
*/
|
|
|
|
|
|
|
|
*recheck = true; /* or false if check is exact */
|
|
|
|
|
|
|
|
PG_RETURN_BOOL(retval);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Here, <varname>key</> is an element in the index and <varname>query</>
|
|
|
|
the value being looked up in the index. The <literal>StrategyNumber</>
|
|
|
|
parameter indicates which operator of your operator class is being
|
|
|
|
applied — it matches one of the operator numbers in the
|
|
|
|
<command>CREATE OPERATOR CLASS</> command. Depending on what operators
|
|
|
|
you have included in the class, the data type of <varname>query</> could
|
|
|
|
vary with the operator, but the above skeleton assumes it doesn't.
|
|
|
|
</para>
|
|
|
|
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>union</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
This method consolidates information in the tree. Given a set of
|
2009-06-12 21:48:53 +02:00
|
|
|
entries, this function generates a new index entry that represents
|
|
|
|
all the given entries.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_union(internal, internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_union(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_union);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_union(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GistEntryVector *entryvec = (GistEntryVector *) PG_GETARG_POINTER(0);
|
|
|
|
GISTENTRY *ent = entryvec->vector;
|
|
|
|
data_type *out,
|
|
|
|
*tmp,
|
|
|
|
*old;
|
|
|
|
int numranges,
|
|
|
|
i = 0;
|
|
|
|
|
|
|
|
numranges = entryvec->n;
|
|
|
|
tmp = DatumGetDataType(ent[0].key);
|
|
|
|
out = tmp;
|
|
|
|
|
|
|
|
if (numranges == 1)
|
|
|
|
{
|
|
|
|
out = data_type_deep_copy(tmp);
|
|
|
|
|
|
|
|
PG_RETURN_DATA_TYPE_P(out);
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 1; i < numranges; i++)
|
|
|
|
{
|
|
|
|
old = out;
|
|
|
|
tmp = DatumGetDataType(ent[i].key);
|
|
|
|
out = my_union_implementation(out, tmp);
|
|
|
|
}
|
|
|
|
|
|
|
|
PG_RETURN_DATA_TYPE_P(out);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
As you can see, in this skeleton we're dealing with a data type
|
|
|
|
where <literal>union(X, Y, Z) = union(union(X, Y), Z)</>. It's easy
|
|
|
|
enough to support data types where this is not the case, by
|
|
|
|
implementing the proper union algorithm in this
|
|
|
|
<acronym>GiST</> support method.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <function>union</> implementation function should return a
|
|
|
|
pointer to newly <function>palloc()</>ed memory. You can't just
|
|
|
|
return whatever the input is.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>compress</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Converts the data item into a format suitable for physical storage in
|
|
|
|
an index page.
|
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_compress(internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_compress(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_compress);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_compress(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GISTENTRY *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
|
|
|
|
GISTENTRY *retval;
|
|
|
|
|
|
|
|
if (entry->leafkey)
|
|
|
|
{
|
|
|
|
/* replace entry->key with a compressed version */
|
|
|
|
compressed_data_type *compressed_data = palloc(sizeof(compressed_data_type));
|
|
|
|
|
|
|
|
/* fill *compressed_data from entry->key ... */
|
|
|
|
|
|
|
|
retval = palloc(sizeof(GISTENTRY));
|
|
|
|
gistentryinit(*retval, PointerGetDatum(compressed_data),
|
|
|
|
entry->rel, entry->page, entry->offset, FALSE);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* typically we needn't do anything with non-leaf entries */
|
|
|
|
retval = entry;
|
|
|
|
}
|
|
|
|
|
|
|
|
PG_RETURN_POINTER(retval);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
You have to adapt <replaceable>compressed_data_type</> to the specific
|
|
|
|
type you're converting to in order to compress your leaf nodes, of
|
|
|
|
course.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Depending on your needs, you could also need to care about
|
|
|
|
compressing <literal>NULL</> values in there, storing for example
|
|
|
|
<literal>(Datum) 0</> like <literal>gist_circle_compress</> does.
|
|
|
|
</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>decompress</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
The reverse of the <function>compress</function> method. Converts the
|
|
|
|
index representation of the data item into a format that can be
|
|
|
|
manipulated by the database.
|
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_decompress(internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_decompress(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_decompress);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_decompress(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
PG_RETURN_POINTER(PG_GETARG_POINTER(0));
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
The above skeleton is suitable for the case where no decompression
|
|
|
|
is needed.
|
|
|
|
</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>penalty</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Returns a value indicating the <quote>cost</quote> of inserting the new
|
2009-06-12 21:48:53 +02:00
|
|
|
entry into a particular branch of the tree. Items will be inserted
|
2003-10-31 23:41:21 +01:00
|
|
|
down the path of least <function>penalty</function> in the tree.
|
2011-05-31 23:53:45 +02:00
|
|
|
Values returned by <function>penalty</function> should be non-negative.
|
|
|
|
If a negative value is returned, it will be treated as zero.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
2009-06-12 21:48:53 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_penalty(internal, internal, internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT; -- in some cases penalty functions need not be strict
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_penalty(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_penalty);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_penalty(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GISTENTRY *origentry = (GISTENTRY *) PG_GETARG_POINTER(0);
|
|
|
|
GISTENTRY *newentry = (GISTENTRY *) PG_GETARG_POINTER(1);
|
|
|
|
float *penalty = (float *) PG_GETARG_POINTER(2);
|
|
|
|
data_type *orig = DatumGetDataType(origentry->key);
|
|
|
|
data_type *new = DatumGetDataType(newentry->key);
|
|
|
|
|
|
|
|
*penalty = my_penalty_implementation(orig, new);
|
|
|
|
PG_RETURN_POINTER(penalty);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <function>penalty</> function is crucial to good performance of
|
|
|
|
the index. It'll get used at insertion time to determine which branch
|
|
|
|
to follow when choosing where to add the new entry in the tree. At
|
|
|
|
query time, the more balanced the index, the quicker the lookup.
|
|
|
|
</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>picksplit</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
2009-06-12 21:48:53 +02:00
|
|
|
When an index page split is necessary, this function decides which
|
|
|
|
entries on the page are to stay on the old page, and which are to move
|
|
|
|
to the new page.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_picksplit(internal, internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_picksplit(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_picksplit);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_picksplit(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GistEntryVector *entryvec = (GistEntryVector *) PG_GETARG_POINTER(0);
|
|
|
|
OffsetNumber maxoff = entryvec->n - 1;
|
|
|
|
GISTENTRY *ent = entryvec->vector;
|
|
|
|
GIST_SPLITVEC *v = (GIST_SPLITVEC *) PG_GETARG_POINTER(1);
|
|
|
|
int i,
|
|
|
|
nbytes;
|
|
|
|
OffsetNumber *left,
|
|
|
|
*right;
|
|
|
|
data_type *tmp_union;
|
|
|
|
data_type *unionL;
|
|
|
|
data_type *unionR;
|
|
|
|
GISTENTRY **raw_entryvec;
|
|
|
|
|
|
|
|
maxoff = entryvec->n - 1;
|
|
|
|
nbytes = (maxoff + 1) * sizeof(OffsetNumber);
|
|
|
|
|
|
|
|
v->spl_left = (OffsetNumber *) palloc(nbytes);
|
|
|
|
left = v->spl_left;
|
|
|
|
v->spl_nleft = 0;
|
|
|
|
|
|
|
|
v->spl_right = (OffsetNumber *) palloc(nbytes);
|
|
|
|
right = v->spl_right;
|
|
|
|
v->spl_nright = 0;
|
|
|
|
|
|
|
|
unionL = NULL;
|
|
|
|
unionR = NULL;
|
|
|
|
|
|
|
|
/* Initialize the raw entry vector. */
|
|
|
|
raw_entryvec = (GISTENTRY **) malloc(entryvec->n * sizeof(void *));
|
|
|
|
for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
|
|
|
|
raw_entryvec[i] = &(entryvec->vector[i]);
|
|
|
|
|
|
|
|
for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
|
|
|
|
{
|
|
|
|
int real_index = raw_entryvec[i] - entryvec->vector;
|
|
|
|
|
|
|
|
tmp_union = DatumGetDataType(entryvec->vector[real_index].key);
|
|
|
|
Assert(tmp_union != NULL);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Choose where to put the index entries and update unionL and unionR
|
|
|
|
* accordingly. Append the entries to either v_spl_left or
|
|
|
|
* v_spl_right, and care about the counters.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (my_choice_is_left(unionL, curl, unionR, curr))
|
|
|
|
{
|
|
|
|
if (unionL == NULL)
|
|
|
|
unionL = tmp_union;
|
|
|
|
else
|
|
|
|
unionL = my_union_implementation(unionL, tmp_union);
|
|
|
|
|
|
|
|
*left = real_index;
|
|
|
|
++left;
|
|
|
|
++(v->spl_nleft);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Same on the right
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
v->spl_ldatum = DataTypeGetDatum(unionL);
|
|
|
|
v->spl_rdatum = DataTypeGetDatum(unionR);
|
|
|
|
PG_RETURN_POINTER(v);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Like <function>penalty</>, the <function>picksplit</> function
|
|
|
|
is crucial to good performance of the index. Designing suitable
|
|
|
|
<function>penalty</> and <function>picksplit</> implementations
|
|
|
|
is where the challenge of implementing well-performing
|
|
|
|
<acronym>GiST</> indexes lies.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2009-06-12 21:48:53 +02:00
|
|
|
<term><function>same</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>
|
2009-06-12 21:48:53 +02:00
|
|
|
Returns true if two index entries are identical, false otherwise.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_same(internal, internal, internal)
|
|
|
|
RETURNS internal
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_same(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_same);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_same(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
prefix_range *v1 = PG_GETARG_PREFIX_RANGE_P(0);
|
|
|
|
prefix_range *v2 = PG_GETARG_PREFIX_RANGE_P(1);
|
|
|
|
bool *result = (bool *) PG_GETARG_POINTER(2);
|
|
|
|
|
|
|
|
*result = my_eq(v1, v2);
|
|
|
|
PG_RETURN_POINTER(result);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
For historical reasons, the <function>same</> function doesn't
|
2010-08-17 06:37:21 +02:00
|
|
|
just return a Boolean result; instead it has to store the flag
|
2009-06-12 21:48:53 +02:00
|
|
|
at the location indicated by the third argument.
|
2003-10-31 23:41:21 +01:00
|
|
|
</para>
|
|
|
|
</listitem>
|
2010-12-04 05:49:06 +01:00
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
|
|
|
<term><function>distance</></term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Given an index entry <literal>p</> and a query value <literal>q</>,
|
|
|
|
this function determines the index entry's
|
|
|
|
<quote>distance</> from the query value. This function must be
|
|
|
|
supplied if the operator class contains any ordering operators.
|
|
|
|
A query using the ordering operator will be implemented by returning
|
|
|
|
index entries with the smallest <quote>distance</> values first,
|
|
|
|
so the results must be consistent with the operator's semantics.
|
|
|
|
For a leaf index entry the result just represents the distance to
|
|
|
|
the index entry; for an internal tree node, the result must be the
|
|
|
|
smallest distance that any child entry could have.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <acronym>SQL</> declaration of the function must look like this:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
CREATE OR REPLACE FUNCTION my_distance(internal, data_type, smallint, oid)
|
|
|
|
RETURNS float8
|
|
|
|
AS 'MODULE_PATHNAME'
|
|
|
|
LANGUAGE C STRICT;
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
And the matching code in the C module could then follow this skeleton:
|
|
|
|
|
|
|
|
<programlisting>
|
|
|
|
Datum my_distance(PG_FUNCTION_ARGS);
|
|
|
|
PG_FUNCTION_INFO_V1(my_distance);
|
|
|
|
|
|
|
|
Datum
|
|
|
|
my_distance(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
GISTENTRY *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
|
|
|
|
data_type *query = PG_GETARG_DATA_TYPE_P(1);
|
|
|
|
StrategyNumber strategy = (StrategyNumber) PG_GETARG_UINT16(2);
|
|
|
|
/* Oid subtype = PG_GETARG_OID(3); */
|
|
|
|
data_type *key = DatumGetDataType(entry->key);
|
|
|
|
double retval;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* determine return value as a function of strategy, key and query.
|
|
|
|
*/
|
|
|
|
|
|
|
|
PG_RETURN_FLOAT8(retval);
|
|
|
|
}
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
The arguments to the <function>distance</> function are identical to
|
|
|
|
the arguments of the <function>consistent</> function, except that no
|
|
|
|
recheck flag is used. The distance to a leaf index entry must always
|
|
|
|
be determined exactly, since there is no way to re-order the tuples
|
|
|
|
once they are returned. Some approximation is allowed when determining
|
|
|
|
the distance to an internal tree node, so long as the result is never
|
|
|
|
greater than any child's actual distance. Thus, for example, distance
|
|
|
|
to a bounding box is usually sufficient in geometric applications. The
|
|
|
|
result value can be any finite <type>float8</> value. (Infinity and
|
|
|
|
minus infinity are used internally to handle cases such as nulls, so it
|
|
|
|
is not recommended that <function>distance</> functions return these
|
|
|
|
values.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
</listitem>
|
2003-10-31 23:41:21 +01:00
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
</variablelist>
|
|
|
|
|
2011-10-01 01:48:57 +02:00
|
|
|
<para>
|
|
|
|
All the GiST support methods are normally called in short-lived memory
|
|
|
|
contexts; that is, <varname>CurrentMemoryContext</> will get reset after
|
|
|
|
each tuple is processed. It is therefore not very important to worry about
|
|
|
|
pfree'ing everything you palloc. However, in some cases it's useful for a
|
|
|
|
support method to cache data across repeated calls. To do that, allocate
|
|
|
|
the longer-lived data in <literal>fcinfo->flinfo->fn_mcxt</>, and
|
|
|
|
keep a pointer to it in <literal>fcinfo->flinfo->fn_extra</>. Such
|
|
|
|
data will survive for the life of the index operation (e.g., a single GiST
|
|
|
|
index scan, index build, or index tuple insertion). Be careful to pfree
|
|
|
|
the previous value when replacing a <literal>fn_extra</> value, or the leak
|
|
|
|
will accumulate for the duration of the operation.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="gist-implementation">
|
|
|
|
<title>Implementation</title>
|
|
|
|
|
2011-09-08 16:51:23 +02:00
|
|
|
<sect2 id="gist-buffering-build">
|
|
|
|
<title>GiST buffering build</title>
|
|
|
|
<para>
|
|
|
|
Building large GiST indexes by simply inserting all the tuples tends to be
|
|
|
|
slow, because if the index tuples are scattered across the index and the
|
|
|
|
index is large enough to not fit in cache, the insertions need to perform
|
2011-10-01 01:48:57 +02:00
|
|
|
a lot of random I/O. Beginning in version 9.2, PostgreSQL supports a more
|
|
|
|
efficient method to build GiST indexes based on buffering, which can
|
|
|
|
dramatically reduce the number of random I/Os needed for non-ordered data
|
2012-06-07 23:06:20 +02:00
|
|
|
sets. For well-ordered data sets the benefit is smaller or non-existent,
|
2011-10-01 01:48:57 +02:00
|
|
|
because only a small number of pages receive new tuples at a time, and
|
|
|
|
those pages fit in cache even if the index as whole does not.
|
2011-09-08 16:51:23 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
However, buffering index build needs to call the <function>penalty</>
|
|
|
|
function more often, which consumes some extra CPU resources. Also, the
|
|
|
|
buffers used in the buffering build need temporary disk space, up to
|
2011-10-01 01:48:57 +02:00
|
|
|
the size of the resulting index. Buffering can also influence the quality
|
|
|
|
of the resulting index, in both positive and negative directions. That
|
2011-09-08 16:51:23 +02:00
|
|
|
influence depends on various factors, like the distribution of the input
|
2011-10-01 01:48:57 +02:00
|
|
|
data and the operator class implementation.
|
2011-09-08 16:51:23 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2011-10-01 01:48:57 +02:00
|
|
|
By default, a GiST index build switches to the buffering method when the
|
2011-09-08 16:51:23 +02:00
|
|
|
index size reaches <xref linkend="guc-effective-cache-size">. It can
|
|
|
|
be manually turned on or off by the <literal>BUFFERING</literal> parameter
|
2011-10-01 01:48:57 +02:00
|
|
|
to the CREATE INDEX command. The default behavior is good for most cases,
|
2011-09-08 16:51:23 +02:00
|
|
|
but turning buffering off might speed up the build somewhat if the input
|
|
|
|
data is ordered.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect2>
|
2003-10-31 23:41:21 +01:00
|
|
|
</sect1>
|
|
|
|
|
2005-10-21 03:41:28 +02:00
|
|
|
<sect1 id="gist-examples">
|
2003-10-31 23:41:21 +01:00
|
|
|
<title>Examples</title>
|
|
|
|
|
|
|
|
<para>
|
2005-10-21 03:41:28 +02:00
|
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
|
|
|
several examples of index methods implemented using
|
2007-11-14 00:36:26 +01:00
|
|
|
<acronym>GiST</acronym>. The core system currently provides text search
|
|
|
|
support (indexing for <type>tsvector</> and <type>tsquery</>) as well as
|
|
|
|
R-Tree equivalent functionality for some of the built-in geometric data types
|
2005-10-21 03:41:28 +02:00
|
|
|
(see <filename>src/backend/access/gist/gistproc.c</>). The following
|
|
|
|
<filename>contrib</> modules also contain <acronym>GiST</acronym>
|
2009-06-12 21:48:53 +02:00
|
|
|
operator classes:
|
|
|
|
|
2003-10-31 23:41:21 +01:00
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>btree_gist</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
2010-08-17 06:37:21 +02:00
|
|
|
<para>B-tree equivalent functionality for several data types</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>cube</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
2006-10-23 20:10:32 +02:00
|
|
|
<para>Indexing for multidimensional cubes</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2007-11-14 00:36:26 +01:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>hstore</></term>
|
2007-11-14 00:36:26 +01:00
|
|
|
<listitem>
|
|
|
|
<para>Module for storing (key, value) pairs</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
2003-10-31 23:41:21 +01:00
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>intarray</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
|
|
|
<para>RD-Tree for one-dimensional array of int4 values</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>ltree</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
2005-10-21 03:41:28 +02:00
|
|
|
<para>Indexing for tree-like structures</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>pg_trgm</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
2005-10-21 03:41:28 +02:00
|
|
|
<para>Text similarity using trigram matching</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
|
|
|
|
<varlistentry>
|
2010-08-17 06:37:21 +02:00
|
|
|
<term><filename>seg</></term>
|
2003-10-31 23:41:21 +01:00
|
|
|
<listitem>
|
2005-10-21 03:41:28 +02:00
|
|
|
<para>Indexing for <quote>float ranges</quote></para>
|
2003-10-31 23:41:21 +01:00
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
2010-08-17 06:37:21 +02:00
|
|
|
</para>
|
2003-10-31 23:41:21 +01:00
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
</chapter>
|