2010-09-20 22:08:53 +02:00
|
|
|
src/backend/access/gist/README
|
2008-03-20 18:55:15 +01:00
|
|
|
|
|
|
|
GiST Indexing
|
|
|
|
=============
|
2005-09-15 18:39:15 +02:00
|
|
|
|
|
|
|
This directory contains an implementation of GiST indexing for Postgres.
|
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
GiST stands for Generalized Search Tree. It was introduced in the seminal paper
|
|
|
|
"Generalized Search Trees for Database Systems", 1995, Joseph M. Hellerstein,
|
|
|
|
Jeffrey F. Naughton, Avi Pfeffer:
|
2005-09-15 18:39:15 +02:00
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
http://www.sai.msu.su/~megera/postgres/gist/papers/gist.ps
|
|
|
|
|
|
|
|
and implemented by J. Hellerstein and P. Aoki in an early version of
|
|
|
|
PostgreSQL (more details are available from The GiST Indexing Project
|
|
|
|
at Berkeley at http://gist.cs.berkeley.edu/). As a "university"
|
|
|
|
project it had a limited number of features and was in rare use.
|
|
|
|
|
|
|
|
The current implementation of GiST supports:
|
2005-09-15 18:39:15 +02:00
|
|
|
|
|
|
|
* Variable length keys
|
|
|
|
* Composite keys (multi-key)
|
|
|
|
* provides NULL-safe interface to GiST core
|
|
|
|
* Concurrency
|
|
|
|
* Recovery support via WAL logging
|
|
|
|
|
2005-09-16 16:40:54 +02:00
|
|
|
The support for concurrency implemented in PostgreSQL was developed based on
|
|
|
|
the paper "Access Methods for Next-Generation Database Systems" by
|
|
|
|
Marcel Kornaker:
|
2005-09-15 18:39:15 +02:00
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
http://www.sai.msu.su/~megera/postgres/gist/papers/concurrency/access-methods-for-next-generation.pdf.gz
|
2005-09-15 18:39:15 +02:00
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
The original algorithms were modified in several ways:
|
|
|
|
|
|
|
|
* They should be adapted to PostgreSQL conventions. For example, the SEARCH
|
2005-09-15 18:39:15 +02:00
|
|
|
algorithm was considerably changed, because in PostgreSQL function search
|
|
|
|
should return one tuple (next), not all tuples at once. Also, it should
|
|
|
|
release page locks between calls.
|
2005-09-16 16:40:54 +02:00
|
|
|
* Since we added support for variable length keys, it's not possible to
|
|
|
|
guarantee enough free space for all keys on pages after splitting. User
|
|
|
|
defined function picksplit doesn't have information about size of tuples
|
|
|
|
(each tuple may contain several keys as in multicolumn index while picksplit
|
|
|
|
could work with only one key) and pages.
|
2005-09-15 19:44:27 +02:00
|
|
|
* We modified original INSERT algorithm for performance reason. In particular,
|
|
|
|
it is now a single-pass algorithm.
|
|
|
|
* Since the papers were theoretical, some details were omitted and we
|
|
|
|
have to find out ourself how to solve some specific problems.
|
2005-09-15 18:39:15 +02:00
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
Because of the above reasons, we have to revised interaction of GiST
|
|
|
|
core and PostgreSQL WAL system. Moreover, we encountered (and solved)
|
|
|
|
a problem of uncompleted insertions when recovering after crash, which
|
|
|
|
was not touched in the paper.
|
2005-09-15 18:39:15 +02:00
|
|
|
|
2008-03-20 18:55:15 +01:00
|
|
|
Search Algorithm
|
|
|
|
----------------
|
2005-09-15 19:44:27 +02:00
|
|
|
|
|
|
|
Function gettuple finds a tuple which satisfies the search
|
|
|
|
predicate. It store their state and returns next tuple under
|
|
|
|
subsequent calls. Stack contains page, its LSN and LSN of parent page
|
|
|
|
and currentposition is saved between calls.
|
2005-09-15 18:39:15 +02:00
|
|
|
|
|
|
|
gettuple(search-pred)
|
|
|
|
if ( firsttime )
|
|
|
|
push(stack, [root, 0, 0]) // page, LSN, parentLSN
|
|
|
|
currentposition=0
|
|
|
|
end
|
|
|
|
ptr = top of stack
|
|
|
|
while(true)
|
|
|
|
latch( ptr->page, S-mode )
|
|
|
|
if ( ptr->page->lsn != ptr->lsn )
|
|
|
|
ptr->lsn = ptr->page->lsn
|
|
|
|
currentposition=0
|
|
|
|
if ( ptr->parentlsn < ptr->page->nsn )
|
|
|
|
add to stack rightlink
|
|
|
|
else
|
|
|
|
currentposition++
|
|
|
|
end
|
|
|
|
|
|
|
|
while(true)
|
|
|
|
currentposition = find_first_match( currentposition )
|
|
|
|
if ( currentposition is invalid )
|
|
|
|
unlatch( ptr->page )
|
|
|
|
pop stack
|
|
|
|
ptr = top of stack
|
|
|
|
if (ptr is NULL)
|
|
|
|
return NULL
|
|
|
|
break loop
|
|
|
|
else if ( ptr->page is leaf )
|
|
|
|
unlatch( ptr->page )
|
|
|
|
return tuple
|
|
|
|
else
|
|
|
|
add to stack child page
|
|
|
|
end
|
|
|
|
currentposition++
|
|
|
|
end
|
|
|
|
end
|
|
|
|
|
|
|
|
|
2008-03-20 18:55:15 +01:00
|
|
|
Insert Algorithm
|
|
|
|
----------------
|
2005-09-15 18:39:15 +02:00
|
|
|
|
|
|
|
INSERT guarantees that the GiST tree remains balanced. User defined key method
|
|
|
|
Penalty is used for choosing a subtree to insert; method PickSplit is used for
|
|
|
|
the node splitting algorithm; method Union is used for propagating changes
|
|
|
|
upward to maintain the tree properties.
|
|
|
|
|
2005-09-15 19:44:27 +02:00
|
|
|
NOTICE: We modified original INSERT algorithm for performance reason. In
|
|
|
|
particularly, it is now a single-pass algorithm.
|
2005-09-15 18:39:15 +02:00
|
|
|
|
|
|
|
Function findLeaf is used to identify subtree for insertion. Page, in which
|
|
|
|
insertion is proceeded, is locked as well as its parent page. Functions
|
|
|
|
findParent and findPath are used to find parent pages, which could be changed
|
2010-04-14 22:17:26 +02:00
|
|
|
because of concurrent access. Function pageSplit is recurrent and could split
|
2005-09-15 18:39:15 +02:00
|
|
|
page by more than 2 pages, which could be necessary if keys have different
|
|
|
|
lengths or more than one key are inserted (in such situation, user defined
|
|
|
|
function pickSplit cannot guarantee free space on page).
|
|
|
|
|
|
|
|
findLeaf(new-key)
|
|
|
|
push(stack, [root, 0]) //page, LSN
|
|
|
|
while(true)
|
|
|
|
ptr = top of stack
|
|
|
|
latch( ptr->page, S-mode )
|
|
|
|
ptr->lsn = ptr->page->lsn
|
|
|
|
if ( exists ptr->parent AND ptr->parent->lsn < ptr->page->nsn )
|
|
|
|
unlatch( ptr->page )
|
|
|
|
pop stack
|
|
|
|
else if ( ptr->page is not leaf )
|
|
|
|
push( stack, [get_best_child(ptr->page, new-key), 0] )
|
|
|
|
unlatch( ptr->page )
|
|
|
|
else
|
|
|
|
unlatch( ptr->page )
|
|
|
|
latch( ptr->page, X-mode )
|
|
|
|
if ( ptr->page is not leaf )
|
|
|
|
//the only root page can become a non-leaf
|
|
|
|
unlatch( ptr->page )
|
|
|
|
else if ( ptr->parent->lsn < ptr->page->nsn )
|
|
|
|
unlatch( ptr->page )
|
|
|
|
pop stack
|
|
|
|
else
|
|
|
|
return stack
|
|
|
|
end
|
|
|
|
end
|
|
|
|
end
|
|
|
|
|
|
|
|
findPath( stack item )
|
|
|
|
push stack, [root, 0, 0] // page, LSN, parent
|
|
|
|
while( stack )
|
|
|
|
ptr = top of stack
|
|
|
|
latch( ptr->page, S-mode )
|
|
|
|
if ( ptr->parent->page->lsn < ptr->page->nsn )
|
|
|
|
push stack, [ ptr->page->rightlink, 0, ptr->parent ]
|
|
|
|
end
|
|
|
|
for( each tuple on page )
|
|
|
|
if ( tuple->pagepointer == item->page )
|
|
|
|
return stack
|
|
|
|
else
|
|
|
|
add to stack at the end [tuple->pagepointer,0, ptr]
|
|
|
|
end
|
|
|
|
end
|
|
|
|
unlatch( ptr->page )
|
|
|
|
pop stack
|
|
|
|
end
|
|
|
|
|
|
|
|
findParent( stack item )
|
|
|
|
parent = item->parent
|
|
|
|
latch( parent->page, X-mode )
|
|
|
|
if ( parent->page->lsn != parent->lsn )
|
|
|
|
while(true)
|
|
|
|
search parent tuple on parent->page, if found the return
|
|
|
|
rightlink = parent->page->rightlink
|
|
|
|
unlatch( parent->page )
|
|
|
|
if ( rightlink is incorrect )
|
|
|
|
break loop
|
|
|
|
end
|
|
|
|
parent->page = rightlink
|
|
|
|
latch( parent->page, X-mode )
|
|
|
|
end
|
|
|
|
newstack = findPath( item->parent )
|
|
|
|
replace part of stack to new one
|
|
|
|
return findParent( item )
|
|
|
|
end
|
|
|
|
|
|
|
|
pageSplit(page, allkeys)
|
|
|
|
(lkeys, rkeys) = pickSplit( allkeys )
|
|
|
|
if ( page is root )
|
|
|
|
lpage = new page
|
|
|
|
else
|
|
|
|
lpage = page
|
|
|
|
rpage = new page
|
|
|
|
if ( no space left on rpage )
|
|
|
|
newkeys = pageSplit( rpage, rkeys )
|
|
|
|
else
|
|
|
|
push newkeys, union(rkeys)
|
|
|
|
end
|
|
|
|
if ( no space left on lpage )
|
|
|
|
push newkeys, pageSplit( lpage, lkeys )
|
|
|
|
else
|
|
|
|
push newkeys, union(lkeys)
|
|
|
|
end
|
|
|
|
return newkeys
|
|
|
|
|
|
|
|
|
|
|
|
placetopage(page, keysarray)
|
|
|
|
if ( no space left on page )
|
|
|
|
keysarray = pageSplit(page, [ extract_keys(page), keysarray])
|
|
|
|
last page in chain gets old NSN,
|
2005-09-16 16:40:54 +02:00
|
|
|
original and others - new NSN equals to LSN
|
2005-09-15 18:39:15 +02:00
|
|
|
if ( page is root )
|
|
|
|
make new root with keysarray
|
|
|
|
end
|
|
|
|
else
|
|
|
|
put keysarray on page
|
|
|
|
if ( length of keysarray > 1 )
|
|
|
|
keysarray = [ union(keysarray) ]
|
|
|
|
end
|
|
|
|
end
|
|
|
|
|
|
|
|
insert(new-key)
|
|
|
|
stack = findLeaf(new-key)
|
|
|
|
keysarray = [new-key]
|
|
|
|
ptr = top of stack
|
|
|
|
while(true)
|
|
|
|
findParent( ptr ) //findParent latches parent page
|
|
|
|
keysarray = placetopage(ptr->page, keysarray)
|
|
|
|
unlatch( ptr->page )
|
|
|
|
pop stack;
|
|
|
|
ptr = top of stack
|
|
|
|
if (length of keysarray == 1)
|
|
|
|
newboundingkey = union(oldboundingkey, keysarray)
|
|
|
|
if (newboundingkey == oldboundingkey)
|
|
|
|
unlatch ptr->page
|
|
|
|
break loop
|
|
|
|
end
|
|
|
|
end
|
|
|
|
end
|
|
|
|
|
|
|
|
Authors:
|
|
|
|
Teodor Sigaev <teodor@sigaev.ru>
|
|
|
|
Oleg Bartunov <oleg@sai.msu.su>
|