postgresql/src/backend/access/gist/README

$PostgreSQL: pgsql/src/backend/access/gist/README,v 1.1 2005/09/15 16:39:15 teodor Exp $

This directory contains an implementation of GiST indexing for Postgres.

GiST is stands for Generalized Search Tree. It was introduced in seminal paper
"Generalized Search Trees for Database Systems", 1995,Joseph M. Hellerstein,
Jeffrey F. Naughton,Avi Pfeffer (http://www.sai.msu.su/~megera/postgres/gist/papers/gist.ps) and implemented by J. Hellerstein and P.Aoki in early version of 
PostgreSQL ( more details is available from The GiST Indexing Project at 
Berkeley at http://gist.cs.berkeley.edu/). As an "university" project it had a 
limited number of features and was in rare use. 

Current implementation of GiST supports:

  * Variable length keys
  * Composite keys (multi-key)
  * provides NULL-safe interface to GiST core
  * Concurrency
  * Recovery support via WAL logging

Concurrence algoritms implemented in PostgreSQL were developed following paper 
"Access Methods for Next-Generation Database Systems" by Marcel Kornaker (http://www.sai.msu.su/~megera/postgres/gist/papers/concurrency/access-methods-for-next-generation.pdf.gz).

Original algorithms were modified by following reasons:

* They should be adapted to PostgreSQL conventions. For example, SEARCH 
  algorithm was considerably changed, because in PostgreSQL function search 
  should return one tuple (next), not all tuples at once. Also, it should 
  release page locks between calls.
* since we added support of variable length keys, it's not possible to guarantee
  enough free space for all keys on pages after splitting. User defined function
  picksplit doesn't have information about size of tuples (each tuple may 
  contain several keys as in multicolumn index while picksplit could work with 
  only one key ) and pages.
* We modified original INSERT algorithm for perfomance reason. In particularly,
  it's single-pass algorithm.
* Since the paper were theoretical, some details were omited and we have to find
  out ourself how to solve some specific problems.

Because of above reasons, we have to revised interaction of GiST core and 
PostgreSQL WAL system. Moreover, we encountered (and solved) a problem of 
uncompleted insertions when recovering after crash, which was not touched in 
the paper.

SEARCH ALGORITHM
Function gettuple finds tuple, which satisfy search predicate. It store their 
state and returns next tuple under subsequent calls. Stack contains page, 
its LSN and LSN of parent page and currentposition is saved between calls.

gettuple(search-pred)
	if ( firsttime )
		push(stack, [root, 0, 0]) // page, LSN, parentLSN
		currentposition=0
	end
	ptr = top of stack
	while(true)
		latch( ptr->page, S-mode )
		if ( ptr->page->lsn != ptr->lsn ) 
			ptr->lsn = ptr->page->lsn
			currentposition=0
			if ( ptr->parentlsn < ptr->page->nsn )
				add to stack rightlink
		else
			currentposition++
		end

		while(true)
			currentposition = find_first_match( currentposition )
			if ( currentposition is invalid )
				unlatch( ptr->page )
				pop stack
				ptr = top of stack
				if (ptr is NULL)
					return NULL
				break loop
			else if ( ptr->page is leaf )
				unlatch( ptr->page )
				return tuple
			else 
				add to stack child page
			end
			currentposition++
		end
	end


INSERT ALGORITHM

INSERT guarantees that the GiST tree remains balanced. User defined key method 
Penalty is used for choosing a subtree to insert; method PickSplit is used for 
the node splitting algorithm; method Union is used for propagating changes 
upward to maintain the tree properties.

NOTICE: We modified original INSERT algorithm for perfomance reason. In 
particularly, it's single-pass algorithm.

Function findLeaf is used to identify subtree for insertion. Page, in which 
insertion is proceeded, is locked as well as its parent page. Functions 
findParent and findPath are used to find parent pages, which could be changed 
because of concurrent access. Function pageSplit is reccurrent and could split 
page by more than 2 pages, which could be necessary if keys have different 
lengths or more than one key are inserted (in such situation, user defined 
function pickSplit cannot guarantee free space on page).

findLeaf(new-key)
	push(stack, [root, 0]) //page, LSN
	while(true)
		ptr = top of stack
		latch( ptr->page, S-mode )
		ptr->lsn = ptr->page->lsn
		if ( exists ptr->parent AND ptr->parent->lsn < ptr->page->nsn )
			unlatch( ptr->page )
			pop stack
		else if ( ptr->page is not leaf )
			push( stack, [get_best_child(ptr->page, new-key), 0] )
			unlatch( ptr->page )
		else
			unlatch( ptr->page )
			latch( ptr->page, X-mode )
			if ( ptr->page is not leaf )
				//the only root page can become a non-leaf
				unlatch( ptr->page )
			else if ( ptr->parent->lsn < ptr->page->nsn )
				unlatch( ptr->page )
				pop stack
			else
				return stack
			end
		end
	end

findPath( stack item )
	push stack, [root, 0, 0] // page, LSN, parent 
	while( stack )
		ptr = top of stack
		latch( ptr->page, S-mode )
		if ( ptr->parent->page->lsn < ptr->page->nsn )
			push stack, [ ptr->page->rightlink, 0, ptr->parent ]
		end
		for( each tuple on page )
			if ( tuple->pagepointer == item->page )
				return stack	
			else
				add to stack at the end [tuple->pagepointer,0, ptr]
			end
		end
		unlatch( ptr->page )
		pop stack
	end
	
findParent( stack item )
	parent = item->parent
	latch( parent->page, X-mode )
	if ( parent->page->lsn != parent->lsn )
		while(true) 
			search parent tuple on parent->page, if found the return
			rightlink = parent->page->rightlink
			unlatch( parent->page )
			if ( rightlink is incorrect )
				break loop
			end
			parent->page = rightlink
			latch( parent->page, X-mode )
		end
		newstack = findPath( item->parent )
		replace part of stack to new one
		return findParent( item )
	end

pageSplit(page, allkeys)
	(lkeys, rkeys) = pickSplit( allkeys )
	if ( page is root )
		lpage = new page
	else
		lpage = page
	rpage = new page
	if ( no space left on rpage )
		newkeys = pageSplit( rpage, rkeys )
	else
		push newkeys, union(rkeys)
	end
	if ( no space left on lpage )
		push newkeys, pageSplit( lpage, lkeys )
	else
		push newkeys, union(lkeys)
	end
	return newkeys


placetopage(page, keysarray)
	if ( no space left on page )
		keysarray = pageSplit(page, [ extract_keys(page), keysarray])
		last page in chain gets old NSN,
		original and others - new NSN from current LSN
		if ( page is root )
			make new root with keysarray
		end
	else
		put keysarray on page
		if ( length of keysarray > 1 )
			keysarray = [ union(keysarray) ]
		end
	end
	
insert(new-key)
	stack = findLeaf(new-key)
	keysarray = [new-key]
	ptr = top of stack
	while(true)
		findParent( ptr ) //findParent latches parent page
		keysarray = placetopage(ptr->page, keysarray)
		unlatch( ptr->page )
		pop stack;
		ptr = top of stack
		if (length of keysarray == 1)
			newboundingkey = union(oldboundingkey, keysarray)
			if (newboundingkey == oldboundingkey)
				unlatch ptr->page
				break loop
			end
		end
	end

Authors:
	Teodor Sigaev	<teodor@sigaev.ru>
	Oleg Bartunov   <oleg@sai.msu.su>