From d2ddee63b43b27d6c6af169342af10db19bd3a1a Mon Sep 17 00:00:00 2001 From: Tom Lane Date: Tue, 23 Aug 2016 12:10:25 -0400 Subject: [PATCH] Improve SP-GiST opclass API to better support unlabeled nodes. Previously, the spgSplitTuple action could only create a new upper tuple containing a single labeled node. This made it useless for opclasses that prefer to work with fixed sets of nodes (labeled or otherwise), which meant that restrictive prefixes could not be used with such node definitions. Change the output field set for the choose() method to allow it to specify any valid node set for the new upper tuple, and to specify which of these nodes to place the modified lower tuple in. In addition to its primary use for fixed node sets, this feature could allow existing opclasses that use variable node sets to skip a separate spgAddNode action when splitting a tuple, by setting up the node needed for the incoming value as part of the spgSplitTuple action. However, care would have to be taken to add the extra node only when it would not make the tuple bigger than before. (spgAddNode can enlarge the tuple, spgSplitTuple can't.) This is a prerequisite for an upcoming SP-GiST inet opclass, but is being committed separately to increase the visibility of the API change. In passing, improve the documentation about the traverse-values feature that was added by commit ccd6eb49a. Emre Hasegeli, with cosmetic adjustments and documentation rework by me Discussion: --- doc/src/sgml/spgist.sgml | 115 +++++++++++++----------- src/backend/access/spgist/spgdoinsert.c | 39 ++++++-- src/backend/access/spgist/spgtextproc.c | 12 ++- src/include/access/spgist.h | 12 ++- 4 files changed, 115 insertions(+), 63 deletions(-) diff --git a/doc/src/sgml/spgist.sgml b/doc/src/sgml/spgist.sgml index f40c790612..dfa62adb55 100644 --- a/doc/src/sgml/spgist.sgml +++ b/doc/src/sgml/spgist.sgml @@ -114,7 +114,7 @@ box_ops - box + box << &< @@ -183,11 +183,14 @@ Inner tuples are more complex, since they are branching points in the search tree. Each inner tuple contains a set of one or more nodes, which represent groups of similar leaf values. - A node contains a downlink that leads to either another, lower-level inner - tuple, or a short list of leaf tuples that all lie on the same index page. - Each node has a label that describes it; for example, + A node contains a downlink that leads either to another, lower-level inner + tuple, or to a short list of leaf tuples that all lie on the same index page. + Each node normally has a label that describes it; for example, in a radix tree the node label could be the next character of the string - value. Optionally, an inner tuple can have a prefix value + value. (Alternatively, an operator class can omit the node labels, if it + works with a fixed set of nodes for all inner tuples; + see .) + Optionally, an inner tuple can have a prefix value that describes all its members. In a radix tree this could be the common prefix of the represented strings. The prefix value is not necessarily really a prefix, but can be any data needed by the operator class; @@ -202,7 +205,8 @@ tuple, so the SP-GiST core provides the possibility for operator classes to manage level counting while descending the tree. There is also support for incrementally reconstructing the represented - value when that is needed. + value when that is needed, and for passing down additional data (called + traverse values) during a tree descent. @@ -343,10 +347,13 @@ typedef struct spgChooseOut } addNode; struct /* results for spgSplitTuple */ { - /* Info to form new inner tuple with one node */ + /* Info to form new upper-level inner tuple with one child tuple */ bool prefixHasPrefix; /* tuple should have a prefix? */ Datum prefixPrefixDatum; /* if so, its value */ - Datum nodeLabel; /* node's label */ + int prefixNNodes; /* number of nodes */ + Datum *prefixNodeLabels; /* their labels (or NULL for + * no labels) */ + int childNodeN; /* which node gets child tuple */ /* Info to form new lower-level inner tuple with all old nodes */ bool postfixHasPrefix; /* tuple should have a prefix? */ @@ -416,29 +423,33 @@ typedef struct spgChooseOut set resultType to spgSplitTuple. This action moves all the existing nodes into a new lower-level inner tuple, and replaces the existing inner tuple with a tuple - having a single node that links to the new lower-level inner tuple. + having a single downlink pointing to the new lower-level inner tuple. Set prefixHasPrefix to indicate whether the new upper tuple should have a prefix, and if so set prefixPrefixDatum to the prefix value. This new prefix value must be sufficiently less restrictive than the original - to accept the new value to be indexed, and it should be no longer - than the original prefix. - Set nodeLabel to the label to be used for the - node that will point to the new lower-level inner tuple. + to accept the new value to be indexed. + Set prefixNNodes to the number of nodes needed in the + new tuple, and set prefixNodeLabels to a palloc'd array + holding their labels, or to NULL if node labels are not required. + Note that the total size of the new upper tuple must be no more + than the total size of the tuple it is replacing; this constrains + the lengths of the new prefix and new labels. + Set childNodeN to the index (from zero) of the node + that will downlink to the new lower-level inner tuple. Set postfixHasPrefix to indicate whether the new lower-level inner tuple should have a prefix, and if so set postfixPrefixDatum to the prefix value. The - combination of these two prefixes and the additional label must - have the same meaning as the original prefix, because there is - no opportunity to alter the node labels that are moved to the new - lower-level tuple, nor to change any child index entries. + combination of these two prefixes and the downlink node's label + (if any) must have the same meaning as the original prefix, because + there is no opportunity to alter the node labels that are moved to + the new lower-level tuple, nor to change any child index entries. After the node has been split, the choose function will be called again with the replacement inner tuple. - That call will usually result in an spgAddNode result, - since presumably the node label added in the split step will not - match the new value; so after that, there will be a third call - that finally returns spgMatchNode and allows the - insertion to descend to the leaf level. + That call may return an spgAddNode result, if no suitable + node was created by the spgSplitTuple action. Eventually + choose must return spgMatchNode to + allow the insertion to descend to the next level. @@ -492,9 +503,8 @@ typedef struct spgPickSplitOut prefixDatum to the prefix value. Set nNodes to indicate the number of nodes that the new inner tuple will contain, and - set nodeLabels to an array of their label values. - (If the nodes do not require labels, set nodeLabels - to NULL; see for details.) + set nodeLabels to an array of their label values, + or to NULL if node labels are not required. Set mapTuplesToNodes to an array that gives the index (from zero) of the node that each leaf tuple should be assigned to. Set leafTupleDatums to an array of the values to @@ -561,7 +571,7 @@ typedef struct spgInnerConsistentIn Datum reconstructedValue; /* value reconstructed at parent */ void *traversalValue; /* opclass-specific traverse value */ - MemoryContext traversalMemoryContext; + MemoryContext traversalMemoryContext; /* put new traverse values here */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */ @@ -580,7 +590,6 @@ typedef struct spgInnerConsistentOut int *levelAdds; /* increment level by this much for each */ Datum *reconstructedValues; /* associated reconstructed values */ void **traversalValues; /* opclass-specific traverse values */ - } spgInnerConsistentOut; @@ -599,6 +608,11 @@ typedef struct spgInnerConsistentOut parent tuple; it is (Datum) 0 at the root level or if the inner_consistent function did not provide a value at the parent level. + traversalValue is a pointer to any traverse data + passed down from the previous call of inner_consistent + on the parent index tuple, or NULL at the root level. + traversalMemoryContext is the memory context in which + to store output traverse values (see below). level is the current inner tuple's level, starting at zero for the root level. returnData is true if reconstructed data is @@ -615,9 +629,6 @@ typedef struct spgInnerConsistentOut inner tuple, and nodeLabels is an array of their label values, or NULL if the nodes do not have labels. - traversalValue is a pointer to data that - inner_consistent gets when called on child nodes from an - outer call of inner_consistent on parent nodes. @@ -633,17 +644,19 @@ typedef struct spgInnerConsistentOut reconstructedValues to an array of the values reconstructed for each child node to be visited; otherwise, leave reconstructedValues as NULL. + If it is desired to pass down additional out-of-band information + (traverse values) to lower levels of the tree search, + set traversalValues to an array of the appropriate + traverse values, one for each child node to be visited; otherwise, + leave traversalValues as NULL. Note that the inner_consistent function is responsible for palloc'ing the - nodeNumbers, levelAdds and - reconstructedValues arrays. - Sometimes accumulating some information is needed, while - descending from parent to child node was happened. In this case - traversalValues array keeps pointers to - specific data you need to accumulate for every child node. - Memory for traversalValues should be allocated in - the default context, but each element of it should be allocated in - traversalMemoryContext. + nodeNumbers, levelAdds, + reconstructedValues, and + traversalValues arrays in the current memory context. + However, any output traverse values pointed to by + the traversalValues array should be allocated + in traversalMemoryContext. @@ -670,8 +683,8 @@ typedef struct spgLeafConsistentIn ScanKey scankeys; /* array of operators and comparison values */ int nkeys; /* length of array */ - void *traversalValue; /* opclass-specific traverse value */ Datum reconstructedValue; /* value reconstructed at parent */ + void *traversalValue; /* opclass-specific traverse value */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */ @@ -700,6 +713,9 @@ typedef struct spgLeafConsistentOut parent tuple; it is (Datum) 0 at the root level or if the inner_consistent function did not provide a value at the parent level. + traversalValue is a pointer to any traverse data + passed down from the previous call of inner_consistent + on the parent index tuple, or NULL at the root level. level is the current leaf tuple's level, starting at zero for the root level. returnData is true if reconstructed data is @@ -797,7 +813,10 @@ typedef struct spgLeafConsistentOut point. In such a case the code typically works with the nodes by number, and there is no need for explicit node labels. To suppress node labels (and thereby save some space), the picksplit - function can return NULL for the nodeLabels array. + function can return NULL for the nodeLabels array, + and likewise the choose function can return NULL for + the prefixNodeLabels array during + a spgSplitTuple action. This will in turn result in nodeLabels being NULL during subsequent calls to choose and inner_consistent. In principle, node labels could be used for some inner tuples and omitted @@ -807,10 +826,7 @@ typedef struct spgLeafConsistentOut When working with an inner tuple having unlabeled nodes, it is an error for choose to return spgAddNode, since the set - of nodes is supposed to be fixed in such cases. Also, there is no - provision for generating an unlabeled node in spgSplitTuple - actions, since it is expected that an spgAddNode action will - be needed as well. + of nodes is supposed to be fixed in such cases. @@ -859,11 +875,10 @@ typedef struct spgLeafConsistentOut The PostgreSQL source distribution includes - several examples of index operator classes for - SP-GiST. The core system currently provides radix - trees over text columns and two types of trees over points: quad-tree and - k-d tree. Look into src/backend/access/spgist/ to see the - code. + several examples of index operator classes for SP-GiST, + as described in . Look + into src/backend/access/spgist/ + and src/backend/utils/adt/ to see the code. diff --git a/src/backend/access/spgist/spgdoinsert.c b/src/backend/access/spgist/spgdoinsert.c index f090ca528b..6fc04b224d 100644 --- a/src/backend/access/spgist/spgdoinsert.c +++ b/src/backend/access/spgist/spgdoinsert.c @@ -1705,17 +1705,40 @@ spgSplitNodeAction(Relation index, SpGistState *state, /* Should not be applied to nulls */ Assert(!SpGistPageStoresNulls(current->page)); + /* Check opclass gave us sane values */ + if (out->result.splitTuple.prefixNNodes <= 0 || + out->result.splitTuple.prefixNNodes > SGITMAXNNODES) + elog(ERROR, "invalid number of prefix nodes: %d", + out->result.splitTuple.prefixNNodes); + if (out->result.splitTuple.childNodeN < 0 || + out->result.splitTuple.childNodeN >= + out->result.splitTuple.prefixNNodes) + elog(ERROR, "invalid child node number: %d", + out->result.splitTuple.childNodeN); + /* - * Construct new prefix tuple, containing a single node with the specified - * label. (We'll update the node's downlink to point to the new postfix - * tuple, below.) + * Construct new prefix tuple with requested number of nodes. We'll fill + * in the childNodeN'th node's downlink below. */ - node = spgFormNodeTuple(state, out->result.splitTuple.nodeLabel, false); + nodes = (SpGistNodeTuple *) palloc(sizeof(SpGistNodeTuple) * + out->result.splitTuple.prefixNNodes); + + for (i = 0; i < out->result.splitTuple.prefixNNodes; i++) + { + Datum label = (Datum) 0; + bool labelisnull; + + labelisnull = (out->result.splitTuple.prefixNodeLabels == NULL); + if (!labelisnull) + label = out->result.splitTuple.prefixNodeLabels[i]; + nodes[i] = spgFormNodeTuple(state, label, labelisnull); + } prefixTuple = spgFormInnerTuple(state, out->result.splitTuple.prefixHasPrefix, out->result.splitTuple.prefixPrefixDatum, - 1, &node); + out->result.splitTuple.prefixNNodes, + nodes); /* it must fit in the space that innerTuple now occupies */ if (prefixTuple->size > innerTuple->size) @@ -1807,10 +1830,12 @@ spgSplitNodeAction(Relation index, SpGistState *state, * the postfix tuple first.) We have to update the local copy of the * prefixTuple too, because that's what will be written to WAL. */ - spgUpdateNodeLink(prefixTuple, 0, postfixBlkno, postfixOffset); + spgUpdateNodeLink(prefixTuple, out->result.splitTuple.childNodeN, + postfixBlkno, postfixOffset); prefixTuple = (SpGistInnerTuple) PageGetItem(current->page, PageGetItemId(current->page, current->offnum)); - spgUpdateNodeLink(prefixTuple, 0, postfixBlkno, postfixOffset); + spgUpdateNodeLink(prefixTuple, out->result.splitTuple.childNodeN, + postfixBlkno, postfixOffset); MarkBufferDirty(current->buffer); diff --git a/src/backend/access/spgist/spgtextproc.c b/src/backend/access/spgist/spgtextproc.c index e0d8f30ef1..852a9b00fa 100644 --- a/src/backend/access/spgist/spgtextproc.c +++ b/src/backend/access/spgist/spgtextproc.c @@ -212,9 +212,14 @@ spg_text_choose(PG_FUNCTION_ARGS) out->result.splitTuple.prefixPrefixDatum = formTextDatum(prefixStr, commonLen); } - out->result.splitTuple.nodeLabel = + out->result.splitTuple.prefixNNodes = 1; + out->result.splitTuple.prefixNodeLabels = + (Datum *) palloc(sizeof(Datum)); + out->result.splitTuple.prefixNodeLabels[0] = Int16GetDatum(*(unsigned char *) (prefixStr + commonLen)); + out->result.splitTuple.childNodeN = 0; + if (prefixSize - commonLen == 1) { out->result.splitTuple.postfixHasPrefix = false; @@ -280,7 +285,10 @@ spg_text_choose(PG_FUNCTION_ARGS) out->resultType = spgSplitTuple; out->result.splitTuple.prefixHasPrefix = in->hasPrefix; out->result.splitTuple.prefixPrefixDatum = in->prefixDatum; - out->result.splitTuple.nodeLabel = Int16GetDatum(-2); + out->result.splitTuple.prefixNNodes = 1; + out->result.splitTuple.prefixNodeLabels = (Datum *) palloc(sizeof(Datum)); + out->result.splitTuple.prefixNodeLabels[0] = Int16GetDatum(-2); + out->result.splitTuple.childNodeN = 0; out->result.splitTuple.postfixHasPrefix = false; } else diff --git a/src/include/access/spgist.h b/src/include/access/spgist.h index f39a2d6938..a953a5a401 100644 --- a/src/include/access/spgist.h +++ b/src/include/access/spgist.h @@ -90,10 +90,13 @@ typedef struct spgChooseOut } addNode; struct /* results for spgSplitTuple */ { - /* Info to form new inner tuple with one node */ + /* Info to form new upper-level inner tuple with one child tuple */ bool prefixHasPrefix; /* tuple should have a prefix? */ Datum prefixPrefixDatum; /* if so, its value */ - Datum nodeLabel; /* node's label */ + int prefixNNodes; /* number of nodes */ + Datum *prefixNodeLabels; /* their labels (or NULL for + * no labels) */ + int childNodeN; /* which node gets child tuple */ /* Info to form new lower-level inner tuple with all old nodes */ bool postfixHasPrefix; /* tuple should have a prefix? */ @@ -134,7 +137,8 @@ typedef struct spgInnerConsistentIn Datum reconstructedValue; /* value reconstructed at parent */ void *traversalValue; /* opclass-specific traverse value */ - MemoryContext traversalMemoryContext; + MemoryContext traversalMemoryContext; /* put new traverse values + * here */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */ @@ -163,8 +167,8 @@ typedef struct spgLeafConsistentIn ScanKey scankeys; /* array of operators and comparison values */ int nkeys; /* length of array */ - void *traversalValue; /* opclass-specific traverse value */ Datum reconstructedValue; /* value reconstructed at parent */ + void *traversalValue; /* opclass-specific traverse value */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */