rdelaage/postgresql
1
0
Fork 0
mirror of https://git.postgresql.org/git/postgresql.git synced 2024-07-16 13:41:09 +02:00
Code Issues Packages Projects Releases Wiki Activity
67c9e4442f
postgresql/src/include/access/gin.h

650 lines
19 KiB
C
Raw Normal View History

GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/*--------------------------------------------------------------------------
* gin.h
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
* header file for postgres inverted index access method implementation.
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*
Update copyright for the year 2010.
2010-01-02 17:58:17 +01:00
* Copyright (c) 2006-2010, PostgreSQL Global Development Group
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
*
Remove cvs keywords from all files.
2010-09-20 22:08:53 +02:00
* src/include/access/gin.h
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*--------------------------------------------------------------------------
*/
#ifndef GIN_H
#define GIN_H
Improve our #include situation by moving pointer types away from the corresponding struct definitions. This allows other headers to avoid including certain highly-loaded headers such as rel.h and relscan.h, instead using just relcache.h, heapam.h or genam.h, which are more lightweight and thus cause less unnecessary dependencies.
2008-06-19 02:46:06 +02:00
#include "access/genam.h"
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#include "access/itup.h"
Change xlog.h to xlogdefs.h in bufpage.h, and fix fallout.
2008-06-07 00:35:22 +02:00
#include "access/xlog.h"
Generic implementation of red-black binary tree. It's planned to use in several places, but for now only GIN uses it during index creation. Using self-balanced tree greatly speeds up index creation in corner cases with preordered data.
2010-02-11 15:29:50 +01:00
#include "utils/rbtree.h"
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#include "fmgr.h"
/*
* amproc indexes for inverted indexes.
*/
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define GIN_COMPARE_PROC 1
#define GIN_EXTRACTVALUE_PROC 2
#define GIN_EXTRACTQUERY_PROC 3
#define GIN_CONSISTENT_PROC 4
Extend GIN to support partial-match searches, and extend tsquery to support prefix matching using this facility. Teodor Sigaev and Oleg Bartunov
2008-05-16 18:31:02 +02:00
#define GIN_COMPARE_PARTIAL_PROC 5
#define GINNProcs 5
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/*
* Page opaque data in a inverted index page.
Minor tweaking of index special-space definitions so that the various index types can be reliably distinguished by examining the special space on an index page. Per my earlier proposal, plus the realization that there's no need for btree's vacuum cycle ID to cycle through every possible 16-bit value. Restricting its range a little costs nearly nothing and eliminates the possibility of collisions. Memo to self: remember to make bitmap indexes play along with this scheme, assuming that patch ever gets accepted.
2007-04-10 00:04:08 +02:00
*
* Note: GIN does not include a page ID word as do the other index types.
* This is OK because the opaque data is only 8 bytes and so can be reliably
* distinguished by size. Revisit this if the size ever increases.
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*/
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinPageOpaqueData
{
Minor tweaking of index special-space definitions so that the various index types can be reliably distinguished by examining the special space on an index page. Per my earlier proposal, plus the realization that there's no need for btree's vacuum cycle ID to cycle through every possible 16-bit value. Restricting its range a little costs nearly nothing and eliminates the possibility of collisions. Memo to self: remember to make bitmap indexes play along with this scheme, assuming that patch ever gets accepted.
2007-04-10 00:04:08 +02:00
BlockNumber rightlink; /* next page if any */
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
OffsetNumber maxoff; /* number entries on GIN_DATA page; number of
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
* heap ItemPointer on GIN_DATA|GIN_LEAF page
* and number of records on GIN_DATA &
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
* ~GIN_LEAF page. On GIN_LIST page, number of
* heap tuples. */
Minor tweaking of index special-space definitions so that the various index types can be reliably distinguished by examining the special space on an index page. Per my earlier proposal, plus the realization that there's no need for btree's vacuum cycle ID to cycle through every possible 16-bit value. Restricting its range a little costs nearly nothing and eliminates the possibility of collisions. Memo to self: remember to make bitmap indexes play along with this scheme, assuming that patch ever gets accepted.
2007-04-10 00:04:08 +02:00
uint16 flags; /* see bit definitions below */
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinPageOpaqueData;
typedef GinPageOpaqueData *GinPageOpaque;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define GIN_DATA (1 << 0)
#define GIN_LEAF (1 << 1)
#define GIN_DELETED (1 << 2)
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define GIN_META (1 << 3)
#define GIN_LIST (1 << 4)
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
#define GIN_LIST_FULLROW (1 << 5) /* makes sense only on GIN_LIST page */
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/* Page numbers of fixed-location pages */
#define GIN_METAPAGE_BLKNO (0)
#define GIN_ROOT_BLKNO (1)
typedef struct GinMetaPageData
{
/*
* Pointers to head and tail of pending list, which consists of GIN_LIST
* pages. These store fast-inserted entries that haven't yet been moved
* into the regular GIN structure.
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
BlockNumber head;
BlockNumber tail;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/*
* Free space in bytes in the pending list's tail page.
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
uint32 tailFreeSize;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* We store both number of pages and number of heap tuples that are in the
* pending list.
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
BlockNumber nPendingPages;
int64 nPendingHeapTuples;
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
/*
* Statistics for planner use (accurate as of last VACUUM)
*/
BlockNumber nTotalPages;
BlockNumber nEntryPages;
BlockNumber nDataPages;
int64 nEntries;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
} GinMetaPageData;
#define GinPageGetMeta(p) \
((GinMetaPageData *) PageGetContents(p))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/*
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
* Works on page
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*/
#define GinPageGetOpaque(page) ( (GinPageOpaque) PageGetSpecialPointer(page) )
#define GinPageIsLeaf(page) ( GinPageGetOpaque(page)->flags & GIN_LEAF )
#define GinPageSetLeaf(page) ( GinPageGetOpaque(page)->flags |= GIN_LEAF )
#define GinPageSetNonLeaf(page) ( GinPageGetOpaque(page)->flags &= ~GIN_LEAF )
#define GinPageIsData(page) ( GinPageGetOpaque(page)->flags & GIN_DATA )
#define GinPageSetData(page) ( GinPageGetOpaque(page)->flags |= GIN_DATA )
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
#define GinPageIsList(page) ( GinPageGetOpaque(page)->flags & GIN_LIST )
#define GinPageSetList(page) ( GinPageGetOpaque(page)->flags |= GIN_LIST )
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define GinPageHasFullRow(page) ( GinPageGetOpaque(page)->flags & GIN_LIST_FULLROW )
#define GinPageSetFullRow(page) ( GinPageGetOpaque(page)->flags |= GIN_LIST_FULLROW )
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#define GinPageIsDeleted(page) ( GinPageGetOpaque(page)->flags & GIN_DELETED)
#define GinPageSetDeleted(page) ( GinPageGetOpaque(page)->flags |= GIN_DELETED)
#define GinPageSetNonDeleted(page) ( GinPageGetOpaque(page)->flags &= ~GIN_DELETED)
#define GinPageRightMost(page) ( GinPageGetOpaque(page)->rightlink == InvalidBlockNumber)
/*
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
* We use our own ItemPointerGet(BlockNumber|GetOffsetNumber)
* to avoid Asserts, since sometimes the ip_posid isn't "valid"
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*/
#define GinItemPointerGetBlockNumber(pointer) \
BlockIdGetBlockNumber(&(pointer)->ip_blkid)
#define GinItemPointerGetOffsetNumber(pointer) \
((pointer)->ip_posid)
Fix an additional set of problems in GIN's handling of lossy page pointers. Although the key-combining code claimed to work correctly if its input contained both lossy and exact pointers for a single page in a single TID stream, in fact this did not work, and could not work without pretty fundamental redesign. Modify keyGetItem so that it will not return such a stream, by handling lossy-pointer cases a bit more explicitly than we did before. Per followup investigation of a gripe from Artur Dabrowski. An example of a query that failed given his data set is select count(*) from search_tab where (to_tsvector('german', keywords ) @@ to_tsquery('german', 'ee:* | dd:*')) and (to_tsvector('german', keywords ) @@ to_tsquery('german', 'aa:*')); Back-patch to 8.4 where the lossy pointer code was introduced.
2010-08-01 21:16:39 +02:00
/*
* Special-case item pointer values needed by the GIN search logic.
* MIN: sorts less than any valid item pointer
* MAX: sorts greater than any valid item pointer
* LOSSY PAGE: indicates a whole heap page, sorts after normal item
* pointers for that page
* Note that these are all distinguishable from an "invalid" item pointer
* (which is InvalidBlockNumber/0) as well as from all normal item
* pointers (which have item numbers in the range 1..MaxHeapTuplesPerPage).
*/
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define ItemPointerSetMin(p) \
ItemPointerSet((p), (BlockNumber)0, (OffsetNumber)0)
#define ItemPointerIsMin(p) \
GIN's ItemPointerIsMin, ItemPointerIsMax, and ItemPointerIsLossyPage macros should use GinItemPointerGetBlockNumber/GinItemPointerGetOffsetNumber, not ItemPointerGetBlockNumber/ItemPointerGetOffsetNumber, because the latter will Assert() on ip_posid == 0, ie a "Min" pointer. (Thus, ItemPointerIsMin has never worked at all, but it seems unused at present.) I'm not certain that the case can occur in normal functioning, but it's blowing up on me while investigating Tatsuo-san's data corruption problem. In any case it seems like a problem waiting to bite someone. Back-patch just in case this really is a problem for somebody in the field.
2009-06-05 20:50:47 +02:00
(GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0 && \
GinItemPointerGetBlockNumber(p) == (BlockNumber)0)
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define ItemPointerSetMax(p) \
ItemPointerSet((p), InvalidBlockNumber, (OffsetNumber)0xffff)
#define ItemPointerIsMax(p) \
GIN's ItemPointerIsMin, ItemPointerIsMax, and ItemPointerIsLossyPage macros should use GinItemPointerGetBlockNumber/GinItemPointerGetOffsetNumber, not ItemPointerGetBlockNumber/ItemPointerGetOffsetNumber, because the latter will Assert() on ip_posid == 0, ie a "Min" pointer. (Thus, ItemPointerIsMin has never worked at all, but it seems unused at present.) I'm not certain that the case can occur in normal functioning, but it's blowing up on me while investigating Tatsuo-san's data corruption problem. In any case it seems like a problem waiting to bite someone. Back-patch just in case this really is a problem for somebody in the field.
2009-06-05 20:50:47 +02:00
(GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0xffff && \
GinItemPointerGetBlockNumber(p) == InvalidBlockNumber)
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define ItemPointerSetLossyPage(p, b) \
ItemPointerSet((p), (b), (OffsetNumber)0xffff)
#define ItemPointerIsLossyPage(p) \
GIN's ItemPointerIsMin, ItemPointerIsMax, and ItemPointerIsLossyPage macros should use GinItemPointerGetBlockNumber/GinItemPointerGetOffsetNumber, not ItemPointerGetBlockNumber/ItemPointerGetOffsetNumber, because the latter will Assert() on ip_posid == 0, ie a "Min" pointer. (Thus, ItemPointerIsMin has never worked at all, but it seems unused at present.) I'm not certain that the case can occur in normal functioning, but it's blowing up on me while investigating Tatsuo-san's data corruption problem. In any case it seems like a problem waiting to bite someone. Back-patch just in case this really is a problem for somebody in the field.
2009-06-05 20:50:47 +02:00
(GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0xffff && \
GinItemPointerGetBlockNumber(p) != InvalidBlockNumber)
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
Minor tweaking of index special-space definitions so that the various index types can be reliably distinguished by examining the special space on an index page. Per my earlier proposal, plus the realization that there's no need for btree's vacuum cycle ID to cycle through every possible 16-bit value. Restricting its range a little costs nearly nothing and eliminates the possibility of collisions. Memo to self: remember to make bitmap indexes play along with this scheme, assuming that patch ever gets accepted.
2007-04-10 00:04:08 +02:00
typedef struct
{
BlockIdData child_blkno; /* use it instead of BlockNumber to save space
* on page */
ItemPointerData key;
} PostingItem;
#define PostingItemGetBlockNumber(pointer) \
BlockIdGetBlockNumber(&(pointer)->child_blkno)
#define PostingItemSetBlockNumber(pointer, blockNumber) \
BlockIdSet(&((pointer)->child_blkno), (blockNumber))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/*
Minor tweaking of index special-space definitions so that the various index types can be reliably distinguished by examining the special space on an index page. Per my earlier proposal, plus the realization that there's no need for btree's vacuum cycle ID to cycle through every possible 16-bit value. Restricting its range a little costs nearly nothing and eliminates the possibility of collisions. Memo to self: remember to make bitmap indexes play along with this scheme, assuming that patch ever gets accepted.
2007-04-10 00:04:08 +02:00
* Support work on IndexTuple on leaf pages
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
*/
#define GinGetNPosting(itup) GinItemPointerGetOffsetNumber(&(itup)->t_tid)
#define GinSetNPosting(itup,n) ItemPointerSetOffsetNumber(&(itup)->t_tid,(n))
#define GIN_TREE_POSTING ((OffsetNumber)0xffff)
#define GinIsPostingTree(itup) ( GinGetNPosting(itup)==GIN_TREE_POSTING )
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define GinSetPostingTree(itup, blkno) ( GinSetNPosting((itup),GIN_TREE_POSTING ), ItemPointerSetBlockNumber(&(itup)->t_tid, blkno) )
#define GinGetPostingTree(itup) GinItemPointerGetBlockNumber(&(itup)->t_tid)
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define GinGetOrigSizePosting(itup) GinItemPointerGetBlockNumber(&(itup)->t_tid)
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#define GinSetOrigSizePosting(itup,n) ItemPointerSetBlockNumber(&(itup)->t_tid,(n))
#define GinGetPosting(itup) ( (ItemPointer)(( ((char*)(itup)) + SHORTALIGN(GinGetOrigSizePosting(itup)) )) )
#define GinMaxItemSize \
Make sure that GIN fast-insert and regular code paths enforce the same tuple size limit. Improve the error message for index-tuple-too-large so that it includes the actual size, the limit, and the index name. Sync with the btree occurrences of the same error. Back-patch to 8.4 because it appears that the out-of-sync problem is occurring in the field. Teodor and Tom
2009-10-02 23:14:04 +02:00
MAXALIGN_DOWN(((BLCKSZ - SizeOfPageHeaderData - \
MAXALIGN(sizeof(GinPageOpaqueData))) / 3 - sizeof(ItemIdData)))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/*
* Data (posting tree) pages
*/
Change the PageGetContents() macro to guarantee its result is maxalign'd, thereby forestalling any problems with alignment of the data structure placed there. Since SizeOfPageHeaderData is maxalign'd anyway in 8.3 and HEAD, this does not actually change anything right now, but it is foreseeable that the header size will change again someday. I had to fix a couple of places that were assuming that the content offset is just SizeOfPageHeaderData rather than MAXALIGN(SizeOfPageHeaderData). Per discussion of Zdenek's page-macros patch.
2008-07-13 23:50:04 +02:00
#define GinDataPageGetRightBound(page) ((ItemPointer) PageGetContents(page))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#define GinDataPageGetData(page) \
Change the PageGetContents() macro to guarantee its result is maxalign'd, thereby forestalling any problems with alignment of the data structure placed there. Since SizeOfPageHeaderData is maxalign'd anyway in 8.3 and HEAD, this does not actually change anything right now, but it is foreseeable that the header size will change again someday. I had to fix a couple of places that were assuming that the content offset is just SizeOfPageHeaderData rather than MAXALIGN(SizeOfPageHeaderData). Per discussion of Zdenek's page-macros patch.
2008-07-13 23:50:04 +02:00
(PageGetContents(page) + MAXALIGN(sizeof(ItemPointerData)))
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
#define GinSizeOfItem(page) \
Change the PageGetContents() macro to guarantee its result is maxalign'd, thereby forestalling any problems with alignment of the data structure placed there. Since SizeOfPageHeaderData is maxalign'd anyway in 8.3 and HEAD, this does not actually change anything right now, but it is foreseeable that the header size will change again someday. I had to fix a couple of places that were assuming that the content offset is just SizeOfPageHeaderData rather than MAXALIGN(SizeOfPageHeaderData). Per discussion of Zdenek's page-macros patch.
2008-07-13 23:50:04 +02:00
(GinPageIsLeaf(page) ? sizeof(ItemPointerData) : sizeof(PostingItem))
#define GinDataPageGetItem(page,i) \
(GinDataPageGetData(page) + ((i)-1) * GinSizeOfItem(page))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#define GinDataPageGetFreeSpace(page) \
Change the PageGetContents() macro to guarantee its result is maxalign'd, thereby forestalling any problems with alignment of the data structure placed there. Since SizeOfPageHeaderData is maxalign'd anyway in 8.3 and HEAD, this does not actually change anything right now, but it is foreseeable that the header size will change again someday. I had to fix a couple of places that were assuming that the content offset is just SizeOfPageHeaderData rather than MAXALIGN(SizeOfPageHeaderData). Per discussion of Zdenek's page-macros patch.
2008-07-13 23:50:04 +02:00
(BLCKSZ - MAXALIGN(SizeOfPageHeaderData) \
- MAXALIGN(sizeof(ItemPointerData)) \
- GinPageGetOpaque(page)->maxoff * GinSizeOfItem(page) \
- MAXALIGN(sizeof(GinPageOpaqueData)))
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/*
* List pages
*/
#define GinListPageSize \
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
( BLCKSZ - SizeOfPageHeaderData - MAXALIGN(sizeof(GinPageOpaqueData)) )
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/*
* Storage type for GIN's reloptions
*/
typedef struct GinOptions
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int32 vl_len_; /* varlena header (do not touch directly!) */
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
bool useFastUpdate; /* use fast updates? */
} GinOptions;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
#define GIN_DEFAULT_USE_FASTUPDATE true
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define GinGetUseFastUpdate(relation) \
((relation)->rd_options ? \
((GinOptions *) (relation)->rd_options)->useFastUpdate : GIN_DEFAULT_USE_FASTUPDATE)
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define GIN_UNLOCK BUFFER_LOCK_UNLOCK
#define GIN_SHARE BUFFER_LOCK_SHARE
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
#define GIN_EXCLUSIVE BUFFER_LOCK_EXCLUSIVE
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinState
{
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
FmgrInfo compareFn[INDEX_MAX_KEYS];
FmgrInfo extractValueFn[INDEX_MAX_KEYS];
FmgrInfo extractQueryFn[INDEX_MAX_KEYS];
FmgrInfo consistentFn[INDEX_MAX_KEYS];
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
FmgrInfo comparePartialFn[INDEX_MAX_KEYS]; /* optional method */
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
bool canPartialMatch[INDEX_MAX_KEYS]; /* can opclass perform
* partial match (prefix
* search)? */
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
TupleDesc tupdesc[INDEX_MAX_KEYS];
TupleDesc origTupdesc;
bool oneCol;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinState;
/* XLog stuff */
#define XLOG_GIN_CREATE_INDEX 0x00
#define XLOG_GIN_CREATE_PTREE 0x10
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct ginxlogCreatePostingTree
{
RelFileNode node;
BlockNumber blkno;
uint32 nitem;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* follows list of heap's ItemPointer */
} ginxlogCreatePostingTree;
#define XLOG_GIN_INSERT 0x20
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct ginxlogInsert
{
RelFileNode node;
BlockNumber blkno;
BlockNumber updateBlkno;
OffsetNumber offset;
bool isDelete;
bool isData;
bool isLeaf;
OffsetNumber nitem;
/*
* follows: tuples or ItemPointerData or PostingItem or list of
* ItemPointerData
*/
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} ginxlogInsert;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define XLOG_GIN_SPLIT 0x30
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct ginxlogSplit
{
RelFileNode node;
BlockNumber lblkno;
BlockNumber rootBlkno;
BlockNumber rblkno;
BlockNumber rrlink;
OffsetNumber separator;
OffsetNumber nitem;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
bool isData;
bool isLeaf;
bool isRootSplit;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
BlockNumber leftChildBlkno;
BlockNumber updateBlkno;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
ItemPointerData rightbound; /* used only in posting tree */
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* follows: list of tuple or ItemPointerData or PostingItem */
} ginxlogSplit;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define XLOG_GIN_VACUUM_PAGE 0x40
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct ginxlogVacuumPage
{
RelFileNode node;
BlockNumber blkno;
OffsetNumber nitem;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* follows content of page */
} ginxlogVacuumPage;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
#define XLOG_GIN_DELETE_PAGE 0x50
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct ginxlogDeletePage
{
RelFileNode node;
BlockNumber blkno;
BlockNumber parentBlkno;
OffsetNumber parentOffset;
BlockNumber leftBlkno;
BlockNumber rightLink;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} ginxlogDeletePage;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
#define XLOG_GIN_UPDATE_META_PAGE 0x60
typedef struct ginxlogUpdateMeta
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
RelFileNode node;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
GinMetaPageData metadata;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
BlockNumber prevTail;
BlockNumber newRightlink;
int32 ntuples; /* if ntuples > 0 then metadata.tail was
* updated with that many tuples; else new sub
* list was inserted */
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/* array of inserted tuples follows */
} ginxlogUpdateMeta;
#define XLOG_GIN_INSERT_LISTPAGE 0x70
typedef struct ginxlogInsertListPage
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
RelFileNode node;
BlockNumber blkno;
BlockNumber rightlink;
int32 ntuples;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/* array of inserted tuples follows */
} ginxlogInsertListPage;
#define XLOG_GIN_DELETE_LISTPAGE 0x80
#define GIN_NDELETE_AT_ONCE 16
typedef struct ginxlogDeleteListPages
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
RelFileNode node;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
GinMetaPageData metadata;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int32 ndeleted;
BlockNumber toDelete[GIN_NDELETE_AT_ONCE];
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
} ginxlogDeleteListPages;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginutil.c */
Code review for FILLFACTOR patch. Change WITH grammar as per earlier discussion (including making def_arg allow reserved words), add missed opt_definition for UNIQUE case. Put the reloptions support code in a less random place (I chose to make a new file access/common/reloptions.c). Eliminate header inclusion creep. Make the index options functions safely user-callable (seems like client apps might like to be able to test validity of options before trying to make an index). Reduce overhead for normal case with no options by allowing rd_options to be NULL. Fix some unmaintainably klugy code, including getting rid of Natts_pg_class_fixed at long last. Some stylistic cleanup too, and pay attention to keeping comments in sync with code. Documentation still needs work, though I did fix the omissions in catalogs.sgml and indexam.sgml.
2006-07-04 00:45:41 +02:00
extern Datum ginoptions(PG_FUNCTION_ARGS);
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
extern void initGinState(GinState *state, Relation index);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
extern Buffer GinNewBuffer(Relation index);
extern void GinInitBuffer(Buffer b, uint32 f);
extern void GinInitPage(Page page, uint32 f, Size pageSize);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
extern void GinInitMetabuffer(Buffer b);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern int ginCompareEntries(GinState *ginstate, OffsetNumber attnum, Datum a, Datum b);
extern int ginCompareAttEntries(GinState *ginstate, OffsetNumber attnum_a, Datum a,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
OffsetNumber attnum_b, Datum b);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern Datum *ginExtractEntriesS(GinState *ginstate, OffsetNumber attnum, Datum value,
pgindent run for 8.3.
2007-11-15 22:14:46 +01:00
int32 *nentries, bool *needUnique);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern Datum *ginExtractEntriesSU(GinState *ginstate, OffsetNumber attnum, Datum value, int32 *nentries);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
extern Datum gin_index_getattr(GinState *ginstate, IndexTuple tuple);
extern OffsetNumber gintuple_get_attrnum(GinState *ginstate, IndexTuple tuple);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
/*
* GinStatsData represents stats data for planner use
*/
typedef struct GinStatsData
{
BlockNumber nPendingPages;
BlockNumber nTotalPages;
BlockNumber nEntryPages;
BlockNumber nDataPages;
int64 nEntries;
} GinStatsData;
extern void ginGetStats(Relation index, GinStatsData *stats);
extern void ginUpdateStats(Relation index, const GinStatsData *stats);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* gininsert.c */
extern Datum ginbuild(PG_FUNCTION_ARGS);
Support unlogged tables. The contents of an unlogged table are WAL-logged; thus, they are not available on standby servers and are truncated whenever the database system enters recovery. Indexes on unlogged tables are also unlogged. Unlogged GiST indexes are not currently supported.
2010-12-29 12:48:53 +01:00
extern Datum ginbuildempty(PG_FUNCTION_ARGS);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
extern Datum gininsert(PG_FUNCTION_ARGS);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
extern void ginEntryInsert(Relation index, GinState *ginstate,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
OffsetNumber attnum, Datum value,
ItemPointerData *items, uint32 nitem,
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
GinStatsData *buildStats);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginxlog.c */
extern void gin_redo(XLogRecPtr lsn, XLogRecord *record);
extern void gin_desc(StringInfo buf, uint8 xl_info, char *rec);
extern void gin_xlog_startup(void);
extern void gin_xlog_cleanup(void);
Make recovery from WAL be restartable, by executing a checkpoint-like operation every so often. This improves the usefulness of PITR log shipping for hot standby: formerly, if the standby server crashed, it was necessary to restart it from the last base backup and replay all the WAL since then. Now it will only need to reread about the same amount of WAL as the master server would. The behavior might also come in handy during a long PITR replay sequence. Simon Riggs, with some editorialization by Tom Lane.
2006-08-07 18:57:57 +02:00
extern bool gin_safe_restartpoint(void);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginbtree.c */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinBtreeStack
{
BlockNumber blkno;
Buffer buffer;
OffsetNumber off;
/* predictNumber contains prediction number of pages on current level */
uint32 predictNumber;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
struct GinBtreeStack *parent;
} GinBtreeStack;
typedef struct GinBtreeData *GinBtree;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinBtreeData
{
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* search methods */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
BlockNumber (*findChildPage) (GinBtree, GinBtreeStack *);
bool (*isMoveRight) (GinBtree, Page);
bool (*findItem) (GinBtree, GinBtreeStack *);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* insert methods */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
OffsetNumber (*findChildPtr) (GinBtree, Page, BlockNumber, OffsetNumber);
BlockNumber (*getLeftMostPage) (GinBtree, Page);
bool (*isEnoughSpace) (GinBtree, Buffer, OffsetNumber);
void (*placeToPage) (GinBtree, Buffer, OffsetNumber, XLogRecData **);
Page (*splitPage) (GinBtree, Buffer, Buffer, OffsetNumber, XLogRecData **);
void (*fillRoot) (GinBtree, Buffer, Buffer, Buffer);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
bool isData;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
bool searchMode;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
Relation index;
GinState *ginstate;
bool fullScan;
bool isBuild;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
BlockNumber rightblkno;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* Entry options */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
OffsetNumber entryAttnum;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
Datum entryValue;
IndexTuple entry;
bool isDelete;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* Data (posting tree) option */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
ItemPointerData *items;
uint32 nitem;
uint32 curitem;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
PostingItem pitem;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinBtreeData;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
extern GinBtreeStack *ginPrepareFindLeafPage(GinBtree btree, BlockNumber blkno);
extern GinBtreeStack *ginFindLeafPage(GinBtree btree, GinBtreeStack *stack);
extern void freeGinBtreeStack(GinBtreeStack *stack);
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
extern void ginInsertValue(GinBtree btree, GinBtreeStack *stack,
GinStatsData *buildStats);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern void ginFindParents(GinBtree btree, GinBtreeStack *stack, BlockNumber rootBlkno);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginentrypage.c */
Make sure that GIN fast-insert and regular code paths enforce the same tuple size limit. Improve the error message for index-tuple-too-large so that it includes the actual size, the limit, and the index name. Sync with the btree occurrences of the same error. Back-patch to 8.4 because it appears that the out-of-sync problem is occurring in the field. Teodor and Tom
2009-10-02 23:14:04 +02:00
extern IndexTuple GinFormTuple(Relation index, GinState *ginstate,
OffsetNumber attnum, Datum key,
ItemPointerData *ipd, uint32 nipd, bool errorTooBig);
Fix a serious bug introduced into GIN in 8.4: now that MergeItemPointers() is supposed to remove duplicate heap TIDs, we have to be sure to reduce the tuple size and posting-item count accordingly in addItemPointersToTuple(). Failing to do so resulted in the effective injection of garbage TIDs into the index contents, ie, whatever happened to be in the memory palloc'd for the new tuple. I'm not sure that this fully explains the index corruption reported by Tatsuo Ishii, but the test case I'm using no longer fails.
2009-06-06 04:39:40 +02:00
extern void GinShortenTuple(IndexTuple itup, uint32 nipd);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern void ginPrepareEntryScan(GinBtree btree, Relation index, OffsetNumber attnum,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Datum value, GinState *ginstate);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern void ginEntryFillRoot(GinBtree btree, Buffer root, Buffer lbuf, Buffer rbuf);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
extern IndexTuple ginPageGetLinkItup(Buffer buf);
/* gindatapage.c */
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern int ginCompareItemPointers(ItemPointer a, ItemPointer b);
extern uint32 ginMergeItemPointers(ItemPointerData *dst,
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
ItemPointerData *a, uint32 na,
Install a search tree depth limit in GIN bulk-insert operations, to prevent them from degrading badly when the input is sorted or nearly so. In this scenario the tree is unbalanced to the point of becoming a mere linked list, so insertions become O(N^2). The easiest and most safely back-patchable solution is to stop growing the tree sooner, ie limit the growth of N. We might later consider a rebalancing tree algorithm, but it's not clear that the benefit would be worth the cost and complexity. Per report from Sergey Burladyan and an earlier complaint from Heikki. Back-patch to 8.2; older versions didn't have GIN indexes.
2009-03-24 23:06:03 +01:00
ItemPointerData *b, uint32 nb);
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
extern void GinDataPageAddItem(Page page, void *data, OffsetNumber offset);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern void GinPageDeletePostingItem(Page page, OffsetNumber offset);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct
{
GinBtreeData btree;
GinBtreeStack *stack;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinPostingTreeScan;
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern GinPostingTreeScan *ginPrepareScanPostingTree(Relation index,
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
BlockNumber rootBlkno, bool searchMode);
Improve GIN indexscan cost estimation. The better estimate requires more statistics than we previously stored: in particular, counts of "entry" versus "data" pages within the index, as well as knowledge of the number of distinct key values. We collect this information during initial index build and update it during VACUUM, storing the info in new fields on the index metapage. No initdb is required because these fields will read as zeroes in a pre-existing index, and the new gincostestimate code is coded to behave (reasonably) sanely if they are zeroes. Teodor Sigaev, reviewed by Jan Urbanski, Tom Lane, and Itagaki Takahiro.
2010-10-18 02:52:32 +02:00
extern void ginInsertItemPointer(GinPostingTreeScan *gdi,
ItemPointerData *items, uint32 nitem,
GinStatsData *buildStats);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern Buffer ginScanBeginPostingTree(GinPostingTreeScan *gdi);
extern void ginDataFillRoot(GinBtree btree, Buffer root, Buffer lbuf, Buffer rbuf);
extern void ginPrepareDataScan(GinBtree btree, Relation index);
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginscan.c */
typedef struct GinScanEntryData *GinScanEntry;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinScanEntryData
{
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* link to the equals entry in current scan key */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
GinScanEntry master;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
/*
* link to values reported to consistentFn, points to
* GinScanKey->entryRes[i]
*/
bool *pval;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
/* entry, got from extractQueryFn */
Datum entry;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
OffsetNumber attnum;
Pointer extra_data;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Fix using too many LWLocks bug, reported by Craig Ringer <craig@postnewspapers.com.au>. It was my mistake, I missed limitation of number of held locks, now GIN doesn't use continiuous locks, but still hold buffers pinned to prevent interference with vacuum's deletion algorithm. Backpatch is needed.
2008-04-22 19:52:43 +02:00
/* Current page in posting tree */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
Buffer buffer;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Fix using too many LWLocks bug, reported by Craig Ringer <craig@postnewspapers.com.au>. It was my mistake, I missed limitation of number of held locks, now GIN doesn't use continiuous locks, but still hold buffers pinned to prevent interference with vacuum's deletion algorithm. Backpatch is needed.
2008-04-22 19:52:43 +02:00
/* current ItemPointer to heap */
ItemPointerData curItem;
Extend GIN to support partial-match searches, and extend tsquery to support prefix matching using this facility. Teodor Sigaev and Oleg Bartunov
2008-05-16 18:31:02 +02:00
/* partial match support */
bool isPartialMatch;
TIDBitmap *partialMatch;
Revise the TIDBitmap API to support multiple concurrent iterations over a bitmap. This is extracted from Greg Stark's posix_fadvise patch; it seems worth committing separately, since it's potentially useful independently of posix_fadvise.
2009-01-10 22:08:36 +01:00
TBMIterator *partialMatchIterator;
Extend GIN to support partial-match searches, and extend tsquery to support prefix matching using this facility. Teodor Sigaev and Oleg Bartunov
2008-05-16 18:31:02 +02:00
TBMIterateResult *partialMatchResult;
StrategyNumber strategy;
Fix using too many LWLocks bug, reported by Craig Ringer <craig@postnewspapers.com.au>. It was my mistake, I missed limitation of number of held locks, now GIN doesn't use continiuous locks, but still hold buffers pinned to prevent interference with vacuum's deletion algorithm. Backpatch is needed.
2008-04-22 19:52:43 +02:00
/* used for Posting list and one page in Posting tree */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
ItemPointerData *list;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
uint32 nlist;
OffsetNumber offset;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
bool isFinished;
bool reduceResult;
uint32 predictNumberResult;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinScanEntryData;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinScanKeyData
{
/* Number of entries in query (got by extractQueryFn) */
uint32 nentries;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* array of ItemPointer result, reported to consistentFn */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
bool *entryRes;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
/* array of scans per entry */
GinScanEntry scanEntry;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Pointer *extra_data;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* for calling consistentFn(GinScanKey->entryRes, strategy, query) */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
StrategyNumber strategy;
Datum query;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
OffsetNumber attnum;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
ItemPointerData curItem;
Rewrite the key-combination logic in GIN's keyGetItem() and scanGetItem() routines to make them behave better in the presence of "lossy" index pointers. The previous coding was outright incorrect for some cases, as recently reported by Artur Dabrowski: scanGetItem would fail to return index entries in cases where one index key had multiple exact pointers on the same page as another key had a lossy pointer. Also, keyGetItem was extremely inefficient for cases where a single index key generates multiple "entry" streams, such as an @@ operator with a multiple-clause tsquery. The presence of a lossy page pointer in any one stream defeated its ability to use the opclass consistentFn, resulting in probing many heap pages that didn't really need to be visited. In Artur's example case, a query like WHERE tsvector @@ to_tsquery('a & b') was about 50X slower than the theoretically equivalent WHERE tsvector @@ to_tsquery('a') AND tsvector @@ to_tsquery('b') The way that I chose to fix this was to have GIN call the consistentFn twice with both TRUE and FALSE values for the in-doubt entry stream, returning a hit if either call produces TRUE, but not if they both return FALSE. The code handles this for the case of a single in-doubt entry stream, but punts (falling back to the stupid behavior) if there's more than one lossy reference to the same page. The idea could be scaled up to deal with multiple lossy references, but I think that would probably be wasted complexity. At least to judge by Artur's example, such cases don't occur often enough to be worth trying to optimize. Back-patch to 8.4. 8.3 did not have lossy GIN index pointers, so not subject to these problems.
2010-07-31 02:30:54 +02:00
bool recheckCurItem;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
bool firstCall;
bool isFinished;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinScanKeyData;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef GinScanKeyData *GinScanKey;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct GinScanOpaqueData
{
MemoryContext tempCtx;
GinState ginstate;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
GinScanKey keys;
uint32 nkeys;
pgindent run for 8.3.
2007-11-15 22:14:46 +01:00
bool isVoidRes; /* true if ginstate.extractQueryFn guarantees
* that nothing will be found */
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} GinScanOpaqueData;
typedef GinScanOpaqueData *GinScanOpaque;
extern Datum ginbeginscan(PG_FUNCTION_ARGS);
extern Datum ginendscan(PG_FUNCTION_ARGS);
extern Datum ginrescan(PG_FUNCTION_ARGS);
extern Datum ginmarkpos(PG_FUNCTION_ARGS);
extern Datum ginrestrpos(PG_FUNCTION_ARGS);
Fix a passel of inappropriately-named global functions in GIN. The GIN code has absolutely no business exporting GIN-specific functions with names as generic as compareItemPointers() or newScanKey(); that's just trouble waiting to happen. I got annoyed about this again just now and decided to fix it. This commit ensures that all global symbols defined in access/gin/ have names including "gin" or "Gin". There were a couple of cases, like names involving "PostingItem", where arguably the names were already sufficiently nongeneric; but I figured as long as I was risking creating merge problems for unapplied GIN patches I might as well impose a uniform policy. I didn't touch any static symbol names. There might be some places where it'd be appropriate to rename some static functions to match siblings that are exported, but I'll leave that for another time.
2010-10-18 03:43:26 +02:00
extern void ginNewScanKey(IndexScanDesc scan);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginget.c */
Rename DLLIMPORT macro to PGDLLIMPORT to avoid conflict with third party includes (like tcl) that define DLLIMPORT.
2007-07-25 14:22:54 +02:00
extern PGDLLIMPORT int GinFuzzySearchLimit;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Replace "amgetmulti" AM functions with "amgetbitmap", in which the whole indexscan always occurs in one call, and the results are returned in a TIDBitmap instead of a limited-size array of TIDs. This should improve speed a little by reducing AM entry/exit overhead, and it is necessary infrastructure if we are ever to support bitmap indexes. In an only slightly related change, add support for TIDBitmaps to preserve (somewhat lossily) the knowledge that particular TIDs reported by an index need to have their quals rechecked when the heap is visited. This facility is not really used yet; we'll need to extend the forced-recheck feature to plain indexscans before it's useful, and that hasn't been coded yet. The intent is to use it to clean up 8.3's horrid @@@ kluge for text search with weighted queries. There might be other uses in future, but that one alone is sufficient reason. Heikki Linnakangas, with some adjustments by me.
2008-04-11 00:25:26 +02:00
extern Datum gingetbitmap(PG_FUNCTION_ARGS);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
/* ginvacuum.c */
extern Datum ginbulkdelete(PG_FUNCTION_ARGS);
extern Datum ginvacuumcleanup(PG_FUNCTION_ARGS);
/* ginarrayproc.c */
extern Datum ginarrayextract(PG_FUNCTION_ARGS);
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
extern Datum ginqueryarrayextract(PG_FUNCTION_ARGS);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
extern Datum ginarrayconsistent(PG_FUNCTION_ARGS);
/* ginbulk.c */
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct EntryAccumulator
{
Rewrite the rbtree routines so that an RBNode is the first field of the struct representing a tree entry, rather than being a separately allocated piece of storage. This API is at least as clean as the old one (if not more so --- there were some bizarre choices in there) and it permits a very substantial memory savings, on the order of 2X in ginbulk.c's usage. Also, fix minor memory leaks in code called by ginEntryInsert, in particular in ginInsertValue and entryFillRoot, as well as ginEntryInsert itself. These leaks resulted in the GIN index build context continuing to bloat even after we'd filled it to maintenance_work_mem and started to dump data out to the index. In combination these fixes restore the GIN index build code to honoring the maintenance_work_mem limit about as well as it did in 8.4. Speed seems on par with 8.4 too, maybe even a bit faster, for a non-pathological case in which HEAD was formerly slower. Back-patch to 9.0 so we don't have a performance regression from 8.4.
2010-08-01 04:12:42 +02:00
RBNode rbnode;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Datum value;
uint32 length;
uint32 number;
Generic implementation of red-black binary tree. It's planned to use in several places, but for now only GIN uses it during index creation. Using self-balanced tree greatly speeds up index creation in corner cases with preordered data.
2010-02-11 15:29:50 +01:00
OffsetNumber attnum;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
bool shouldSort;
Generic implementation of red-black binary tree. It's planned to use in several places, but for now only GIN uses it during index creation. Using self-balanced tree greatly speeds up index creation in corner cases with preordered data.
2010-02-11 15:29:50 +01:00
ItemPointerData *list;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} EntryAccumulator;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
typedef struct
{
GinState *ginstate;
GIN index build's allocatedMemory counter needs to be long, not uint32. Else, in a 64-bit machine with maintenance_work_mem set to above 4Gb, the counter overflows and we never recognize having reached the maintenance_work_mem limit. I believe this explains out-of-memory failure recently reported by Sean Davis. This is a bug, so backpatch to 8.2.
2007-11-16 22:50:06 +01:00
long allocatedMemory;
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
uint32 length;
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
EntryAccumulator *entryallocator;
Generic implementation of red-black binary tree. It's planned to use in several places, but for now only GIN uses it during index creation. Using self-balanced tree greatly speeds up index creation in corner cases with preordered data.
2010-02-11 15:29:50 +01:00
RBTree *tree;
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
} BuildAccumulator;
extern void ginInitBA(BuildAccumulator *accum);
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
extern void ginInsertRecordBA(BuildAccumulator *accum,
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
ItemPointer heapptr,
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
OffsetNumber attnum, Datum *entries, int32 nentry);
Rewrite the rbtree routines so that an RBNode is the first field of the struct representing a tree entry, rather than being a separately allocated piece of storage. This API is at least as clean as the old one (if not more so --- there were some bizarre choices in there) and it permits a very substantial memory savings, on the order of 2X in ginbulk.c's usage. Also, fix minor memory leaks in code called by ginEntryInsert, in particular in ginInsertValue and entryFillRoot, as well as ginEntryInsert itself. These leaks resulted in the GIN index build context continuing to bloat even after we'd filled it to maintenance_work_mem and started to dump data out to the index. In combination these fixes restore the GIN index build code to honoring the maintenance_work_mem limit about as well as it did in 8.4. Speed seems on par with 8.4 too, maybe even a bit faster, for a non-pathological case in which HEAD was formerly slower. Back-patch to 9.0 so we don't have a performance regression from 8.4.
2010-08-01 04:12:42 +02:00
extern void ginBeginBAScan(BuildAccumulator *accum);
Multi-column GIN indexes. Teodor Sigaev
2008-07-11 23:06:29 +02:00
extern ItemPointerData *ginGetEntry(BuildAccumulator *accum, OffsetNumber *attnum, Datum *entry, uint32 *n);
GIN: Generalized Inverted iNdex. text[], int4[], Tsearch2 support for GIN.
2006-05-02 13:28:56 +02:00
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/* ginfast.c */
typedef struct GinTupleCollector
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
IndexTuple *tuples;
uint32 ntuples;
uint32 lentuples;
uint32 sumsize;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
} GinTupleCollector;
extern void ginHeapTupleFastInsert(Relation index, GinState *ginstate,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
GinTupleCollector *collector);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
extern uint32 ginHeapTupleFastCollect(Relation index, GinState *ginstate,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
GinTupleCollector *collector,
OffsetNumber attnum, Datum value, ItemPointer item);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
extern void ginInsertCleanup(Relation index, GinState *ginstate,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
bool vac_delay, IndexBulkDeleteResult *stats);
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
#endif /* GIN_H */
Reference in New Issue Copy Permalink