1996-07-09 08:22:35 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* nbtsearch.c
|
2003-02-22 01:45:05 +01:00
|
|
|
* Search code for postgres btrees.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
*
|
2024-01-04 02:49:05 +01:00
|
|
|
* Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
|
2000-01-26 06:58:53 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/backend/access/nbtree/nbtsearch.c
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
1999-07-16 01:04:24 +02:00
|
|
|
#include "postgres.h"
|
1996-07-09 08:22:35 +02:00
|
|
|
|
1999-07-16 01:04:24 +02:00
|
|
|
#include "access/nbtree.h"
|
2008-06-19 02:46:06 +02:00
|
|
|
#include "access/relscan.h"
|
2023-07-03 06:16:27 +02:00
|
|
|
#include "access/xact.h"
|
2009-05-05 21:36:32 +02:00
|
|
|
#include "miscadmin.h"
|
2005-10-06 04:29:23 +02:00
|
|
|
#include "pgstat.h"
|
Implement genuine serializable isolation level.
Until now, our Serializable mode has in fact been what's called Snapshot
Isolation, which allows some anomalies that could not occur in any
serialized ordering of the transactions. This patch fixes that using a
method called Serializable Snapshot Isolation, based on research papers by
Michael J. Cahill (see README-SSI for full references). In Serializable
Snapshot Isolation, transactions run like they do in Snapshot Isolation,
but a predicate lock manager observes the reads and writes performed and
aborts transactions if it detects that an anomaly might occur. This method
produces some false positives, ie. it sometimes aborts transactions even
though there is no anomaly.
To track reads we implement predicate locking, see storage/lmgr/predicate.c.
Whenever a tuple is read, a predicate lock is acquired on the tuple. Shared
memory is finite, so when a transaction takes many tuple-level locks on a
page, the locks are promoted to a single page-level lock, and further to a
single relation level lock if necessary. To lock key values with no matching
tuple, a sequential scan always takes a relation-level lock, and an index
scan acquires a page-level lock that covers the search key, whether or not
there are any matching keys at the moment.
A predicate lock doesn't conflict with any regular locks or with another
predicate locks in the normal sense. They're only used by the predicate lock
manager to detect the danger of anomalies. Only serializable transactions
participate in predicate locking, so there should be no extra overhead for
for other transactions.
Predicate locks can't be released at commit, but must be remembered until
all the transactions that overlapped with it have completed. That means that
we need to remember an unbounded amount of predicate locks, so we apply a
lossy but conservative method of tracking locks for committed transactions.
If we run short of shared memory, we overflow to a new "pg_serial" SLRU
pool.
We don't currently allow Serializable transactions in Hot Standby mode.
That would be hard, because even read-only transactions can cause anomalies
that wouldn't otherwise occur.
Serializable isolation mode now means the new fully serializable level.
Repeatable Read gives you the old Snapshot Isolation level that we have
always had.
Kevin Grittner and Dan Ports, reviewed by Jeff Davis, Heikki Linnakangas and
Anssi Kääriäinen
2011-02-07 22:46:51 +01:00
|
|
|
#include "storage/predicate.h"
|
2003-11-12 22:15:59 +01:00
|
|
|
#include "utils/lsyscache.h"
|
2008-06-19 02:46:06 +02:00
|
|
|
#include "utils/rel.h"
|
1996-10-23 09:42:13 +02:00
|
|
|
|
1996-11-03 13:35:27 +01:00
|
|
|
|
2019-03-19 17:59:05 +01:00
|
|
|
static void _bt_drop_lock_and_maybe_pin(IndexScanDesc scan, BTScanPos sp);
|
2019-03-20 17:30:57 +01:00
|
|
|
static OffsetNumber _bt_binsrch(Relation rel, BTScanInsert key, Buffer buf);
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
static int _bt_binsrch_posting(BTScanInsert key, Page page,
|
|
|
|
OffsetNumber offnum);
|
2006-05-07 03:21:30 +02:00
|
|
|
static bool _bt_readpage(IndexScanDesc scan, ScanDirection dir,
|
2023-12-27 13:21:49 +01:00
|
|
|
OffsetNumber offnum, bool firstPage);
|
2011-10-09 06:21:08 +02:00
|
|
|
static void _bt_saveitem(BTScanOpaque so, int itemIndex,
|
|
|
|
OffsetNumber offnum, IndexTuple itup);
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
static int _bt_setuppostingitems(BTScanOpaque so, int itemIndex,
|
|
|
|
OffsetNumber offnum, ItemPointer heapTid,
|
|
|
|
IndexTuple itup);
|
|
|
|
static inline void _bt_savepostingitem(BTScanOpaque so, int itemIndex,
|
|
|
|
OffsetNumber offnum,
|
|
|
|
ItemPointer heapTid, int tupleOffset);
|
2006-05-07 03:21:30 +02:00
|
|
|
static bool _bt_steppage(IndexScanDesc scan, ScanDirection dir);
|
2017-02-15 13:41:14 +01:00
|
|
|
static bool _bt_readnextpage(IndexScanDesc scan, BlockNumber blkno, ScanDirection dir);
|
|
|
|
static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
|
|
|
|
ScanDirection dir);
|
2023-09-08 07:12:12 +02:00
|
|
|
static Buffer _bt_walk_left(Relation rel, Buffer buf);
|
2002-05-21 01:51:44 +02:00
|
|
|
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
|
2017-02-15 13:41:14 +01:00
|
|
|
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
|
2015-03-25 20:24:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_drop_lock_and_maybe_pin()
|
|
|
|
*
|
2021-12-09 02:24:45 +01:00
|
|
|
* Unlock the buffer; and if it is safe to release the pin, do that, too.
|
2015-03-25 20:24:43 +01:00
|
|
|
* This will prevent vacuum from stalling in a blocked state trying to read a
|
2021-12-09 02:24:45 +01:00
|
|
|
* page when a cursor is sitting on it.
|
2015-03-25 20:24:43 +01:00
|
|
|
*
|
2021-12-09 02:24:45 +01:00
|
|
|
* See nbtree/README section on making concurrent TID recycling safe.
|
2015-03-25 20:24:43 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
_bt_drop_lock_and_maybe_pin(IndexScanDesc scan, BTScanPos sp)
|
|
|
|
{
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_unlockbuf(scan->indexRelation, sp->buf);
|
2015-03-25 20:24:43 +01:00
|
|
|
|
|
|
|
if (IsMVCCSnapshot(scan->xs_snapshot) &&
|
|
|
|
RelationNeedsWAL(scan->indexRelation) &&
|
|
|
|
!scan->xs_want_itup)
|
|
|
|
{
|
|
|
|
ReleaseBuffer(sp->buf);
|
|
|
|
sp->buf = InvalidBuffer;
|
|
|
|
}
|
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
|
|
|
|
/*
|
2000-07-21 08:42:39 +02:00
|
|
|
* _bt_search() -- Search the tree for a particular scankey,
|
|
|
|
* or more precisely for the first leaf page it could be on.
|
|
|
|
*
|
2019-03-20 17:30:57 +01:00
|
|
|
* The passed scankey is an insertion-type scankey (see nbtree/README),
|
2006-01-17 01:09:01 +01:00
|
|
|
* but it can omit the rightmost column(s) of the index.
|
|
|
|
*
|
2020-07-02 23:54:55 +02:00
|
|
|
* Return value is a stack of parent-page pointers (i.e. there is no entry for
|
|
|
|
* the leaf level/page). *bufP is set to the address of the leaf-page buffer,
|
|
|
|
* which is locked and pinned. No locks are held on the parent pages,
|
|
|
|
* however!
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2018-07-27 23:31:40 +02:00
|
|
|
* The returned buffer is locked according to access parameter. Additionally,
|
|
|
|
* access = BT_WRITE will allow an empty root page to be created and returned.
|
|
|
|
* When access = BT_READ, an empty index will result in *bufP being set to
|
|
|
|
* InvalidBuffer. Also, in BT_WRITE mode, any incomplete splits encountered
|
|
|
|
* during the search will be finished.
|
2023-06-10 23:08:25 +02:00
|
|
|
*
|
|
|
|
* heaprel must be provided by callers that pass access = BT_WRITE, since we
|
|
|
|
* might need to allocate a new root page for caller -- see _bt_allocbuf.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
BTStack
|
2023-04-02 05:12:26 +02:00
|
|
|
_bt_search(Relation rel, Relation heaprel, BTScanInsert key, Buffer *bufP,
|
2023-09-08 07:12:12 +02:00
|
|
|
int access)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
BTStack stack_in = NULL;
|
2018-07-27 23:31:40 +02:00
|
|
|
int page_access = BT_READ;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2023-06-10 23:08:25 +02:00
|
|
|
/* heaprel must be set whenever _bt_allocbuf is reachable */
|
|
|
|
Assert(access == BT_READ || access == BT_WRITE);
|
|
|
|
Assert(access == BT_READ || heaprel != NULL);
|
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* Get the root page to start with */
|
2023-04-02 05:12:26 +02:00
|
|
|
*bufP = _bt_getroot(rel, heaprel, access);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* If index is empty and access = BT_READ, no root page is created. */
|
|
|
|
if (!BufferIsValid(*bufP))
|
|
|
|
return (BTStack) NULL;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* Loop iterates once per level descended in the tree */
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
OffsetNumber offnum;
|
|
|
|
ItemId itemid;
|
|
|
|
IndexTuple itup;
|
2020-07-02 23:54:55 +02:00
|
|
|
BlockNumber child;
|
2000-07-21 08:42:39 +02:00
|
|
|
BTStack new_stack;
|
|
|
|
|
2003-07-30 00:18:38 +02:00
|
|
|
/*
|
|
|
|
* Race -- the page we just grabbed may have split since we read its
|
2020-07-02 23:54:55 +02:00
|
|
|
* downlink in its parent page (or the metapage). If it has, we may
|
|
|
|
* need to move right to its new sibling. Do that.
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
*
|
|
|
|
* In write-mode, allow _bt_moveright to finish any incomplete splits
|
|
|
|
* along the way. Strictly speaking, we'd only need to finish an
|
|
|
|
* incomplete split on the leaf page we're about to insert to, not on
|
2020-07-02 23:54:55 +02:00
|
|
|
* any of the upper levels (internal pages with incomplete splits are
|
|
|
|
* also taken care of in _bt_getstackbuf). But this is a good
|
|
|
|
* opportunity to finish splits of internal pages too.
|
2003-07-30 00:18:38 +02:00
|
|
|
*/
|
2023-04-02 05:12:26 +02:00
|
|
|
*bufP = _bt_moveright(rel, heaprel, key, *bufP, (access == BT_WRITE),
|
2023-09-08 07:12:12 +02:00
|
|
|
stack_in, page_access);
|
2003-07-30 00:18:38 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* if this is a leaf page, we're done */
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(*bufP);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2000-07-21 08:42:39 +02:00
|
|
|
if (P_ISLEAF(opaque))
|
|
|
|
break;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/*
|
2020-07-02 23:54:55 +02:00
|
|
|
* Find the appropriate pivot tuple on this page. Its downlink points
|
|
|
|
* to the child page that we're about to descend to.
|
2000-07-21 08:42:39 +02:00
|
|
|
*/
|
2019-03-20 17:30:57 +01:00
|
|
|
offnum = _bt_binsrch(rel, key, *bufP);
|
2000-07-21 08:42:39 +02:00
|
|
|
itemid = PageGetItemId(page, offnum);
|
2006-01-26 00:04:21 +01:00
|
|
|
itup = (IndexTuple) PageGetItem(page, itemid);
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
Assert(BTreeTupleIsPivot(itup) || !key->heapkeyspace);
|
2020-07-02 23:54:55 +02:00
|
|
|
child = BTreeTupleGetDownLink(itup);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/*
|
2020-07-02 23:54:55 +02:00
|
|
|
* We need to save the location of the pivot tuple we chose in a new
|
|
|
|
* stack entry for this page/level. If caller ends up splitting a
|
|
|
|
* page one level down, it usually ends up inserting a new pivot
|
|
|
|
* tuple/downlink immediately after the location recorded here.
|
2000-07-21 08:42:39 +02:00
|
|
|
*/
|
|
|
|
new_stack = (BTStack) palloc(sizeof(BTStackData));
|
2020-07-02 23:54:55 +02:00
|
|
|
new_stack->bts_blkno = BufferGetBlockNumber(*bufP);
|
2000-07-21 08:42:39 +02:00
|
|
|
new_stack->bts_offset = offnum;
|
|
|
|
new_stack->bts_parent = stack_in;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2018-07-27 23:31:40 +02:00
|
|
|
/*
|
|
|
|
* Page level 1 is lowest non-leaf page level prior to leaves. So, if
|
|
|
|
* we're on the level 1 and asked to lock leaf page in write mode,
|
|
|
|
* then lock next page in write mode, because it must be a leaf.
|
|
|
|
*/
|
2021-02-25 03:41:34 +01:00
|
|
|
if (opaque->btpo_level == 1 && access == BT_WRITE)
|
2018-07-27 23:31:40 +02:00
|
|
|
page_access = BT_WRITE;
|
|
|
|
|
2020-07-02 23:54:55 +02:00
|
|
|
/* drop the read lock on the page, then acquire one on its child */
|
|
|
|
*bufP = _bt_relandgetbuf(rel, *bufP, child, page_access);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* okay, all set to move down a level */
|
|
|
|
stack_in = new_stack;
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2018-07-27 23:31:40 +02:00
|
|
|
/*
|
|
|
|
* If we're asked to lock leaf in write mode, but didn't manage to, then
|
2019-03-29 20:29:05 +01:00
|
|
|
* relock. This should only happen when the root page is a leaf page (and
|
|
|
|
* the only page in the index other than the metapage).
|
2018-07-27 23:31:40 +02:00
|
|
|
*/
|
|
|
|
if (access == BT_WRITE && page_access == BT_READ)
|
|
|
|
{
|
|
|
|
/* trade in our read lock for a write lock */
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_unlockbuf(rel, *bufP);
|
|
|
|
_bt_lockbuf(rel, *bufP, BT_WRITE);
|
2018-07-27 23:31:40 +02:00
|
|
|
|
|
|
|
/*
|
2020-07-22 00:50:58 +02:00
|
|
|
* Race -- the leaf page may have split after we dropped the read lock
|
|
|
|
* but before we acquired a write lock. If it has, we may need to
|
|
|
|
* move right to its new sibling. Do that.
|
2018-07-27 23:31:40 +02:00
|
|
|
*/
|
2023-09-08 07:12:12 +02:00
|
|
|
*bufP = _bt_moveright(rel, heaprel, key, *bufP, true, stack_in, BT_WRITE);
|
2018-07-27 23:31:40 +02:00
|
|
|
}
|
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
return stack_in;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_moveright() -- move right in the btree if necessary.
|
|
|
|
*
|
2003-12-21 02:23:06 +01:00
|
|
|
* When we follow a pointer to reach a page, it is possible that
|
|
|
|
* the page has changed in the meanwhile. If this happens, we're
|
|
|
|
* guaranteed that the page has "split right" -- that is, that any
|
|
|
|
* data that appeared on the page originally is either on the page
|
|
|
|
* or strictly to the right of it.
|
|
|
|
*
|
|
|
|
* This routine decides whether or not we need to move right in the
|
2019-03-20 17:30:57 +01:00
|
|
|
* tree by examining the high key entry on the page. If that entry is
|
|
|
|
* strictly less than the scankey, or <= the scankey in the
|
|
|
|
* key.nextkey=true case, then we followed the wrong link and we need
|
|
|
|
* to move right.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2019-03-20 17:30:57 +01:00
|
|
|
* The passed insertion-type scankey can omit the rightmost column(s) of the
|
|
|
|
* index. (see nbtree/README)
|
2006-01-17 01:09:01 +01:00
|
|
|
*
|
2019-03-20 17:30:57 +01:00
|
|
|
* When key.nextkey is false (the usual case), we are looking for the first
|
|
|
|
* item >= key. When key.nextkey is true, we are looking for the first item
|
|
|
|
* strictly greater than key.
|
2006-01-17 01:09:01 +01:00
|
|
|
*
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
* If forupdate is true, we will attempt to finish any incomplete splits
|
|
|
|
* that we encounter. This is required when locking a target page for an
|
2023-06-10 23:08:25 +02:00
|
|
|
* insertion, because we don't allow inserting on a page before the split is
|
|
|
|
* completed. 'heaprel' and 'stack' are only used if forupdate is true.
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
*
|
2003-12-21 02:23:06 +01:00
|
|
|
* On entry, we have the buffer pinned and a lock of the type specified by
|
|
|
|
* 'access'. If we move right, we release the buffer and lock and acquire
|
|
|
|
* the same on the right sibling. Return value is the buffer we stop at.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
Buffer
|
|
|
|
_bt_moveright(Relation rel,
|
2023-04-02 05:12:26 +02:00
|
|
|
Relation heaprel,
|
2019-03-20 17:30:57 +01:00
|
|
|
BTScanInsert key,
|
1996-07-09 08:22:35 +02:00
|
|
|
Buffer buf,
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
bool forupdate,
|
|
|
|
BTStack stack,
|
2023-09-08 07:12:12 +02:00
|
|
|
int access)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
2003-12-21 02:23:06 +01:00
|
|
|
int32 cmpval;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2023-06-10 23:08:25 +02:00
|
|
|
Assert(!forupdate || heaprel != NULL);
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2003-12-21 02:23:06 +01:00
|
|
|
* When nextkey = false (normal case): if the scan key that brought us to
|
|
|
|
* this page is > the high key stored on the page, then the page has split
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
* and we need to move right. (pg_upgrade'd !heapkeyspace indexes could
|
|
|
|
* have some duplicates to the right as well as the left, but that's
|
|
|
|
* something that's only ever dealt with on the leaf level, after
|
|
|
|
* _bt_search has found an initial leaf page.)
|
2003-12-21 02:23:06 +01:00
|
|
|
*
|
|
|
|
* When nextkey = true: move right if the scan key is >= page's high key.
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
* (Note that key.scantid cannot be set in this case.)
|
2003-12-21 02:23:06 +01:00
|
|
|
*
|
|
|
|
* The page could even have split more than once, so scan as far as
|
|
|
|
* needed.
|
2003-02-22 01:45:05 +01:00
|
|
|
*
|
|
|
|
* We also have to move right if we followed a link that brought us to a
|
|
|
|
* dead page.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2019-03-20 17:30:57 +01:00
|
|
|
cmpval = key->nextkey ? 0 : 1;
|
2003-12-21 02:23:06 +01:00
|
|
|
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
for (;;)
|
1997-04-16 03:48:29 +02:00
|
|
|
{
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
|
|
|
|
if (P_RIGHTMOST(opaque))
|
|
|
|
break;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Finish any incomplete splits we encounter along the way.
|
|
|
|
*/
|
|
|
|
if (forupdate && P_INCOMPLETE_SPLIT(opaque))
|
|
|
|
{
|
|
|
|
BlockNumber blkno = BufferGetBlockNumber(buf);
|
|
|
|
|
|
|
|
/* upgrade our lock if necessary */
|
|
|
|
if (access == BT_READ)
|
|
|
|
{
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_unlockbuf(rel, buf);
|
|
|
|
_bt_lockbuf(rel, buf, BT_WRITE);
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
if (P_INCOMPLETE_SPLIT(opaque))
|
2023-04-02 05:12:26 +02:00
|
|
|
_bt_finish_split(rel, heaprel, buf, stack);
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
else
|
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
|
|
|
|
/* re-acquire the lock in the right mode, and re-check */
|
2023-06-10 23:08:25 +02:00
|
|
|
buf = _bt_getbuf(rel, blkno, access);
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2019-03-20 17:30:57 +01:00
|
|
|
if (P_IGNORE(opaque) || _bt_compare(rel, key, page, P_HIKEY) >= cmpval)
|
Make the handling of interrupted B-tree page splits more robust.
Splitting a page consists of two separate steps: splitting the child page,
and inserting the downlink for the new right page to the parent. Previously,
we handled the case that you crash in between those steps with a cleanup
routine after the WAL recovery had finished, which finished the incomplete
split. However, that doesn't help if the page split is interrupted but the
database doesn't crash, so that you don't perform WAL recovery. That could
happen for example if you run out of disk space.
Remove the end-of-recovery cleanup step. Instead, when a page is split, the
left page is marked with a new INCOMPLETE_SPLIT flag, and when the downlink
is inserted to the parent, the flag is cleared again. If an insertion sees
a page with the flag set, it knows that the split was interrupted for some
reason, and inserts the missing downlink before proceeding.
I used the same approach to fix GIN and GiST split algorithms earlier. This
was the last WAL cleanup routine, so we could get rid of that whole
machinery now, but I'll leave that for a separate patch.
Reviewed by Peter Geoghegan.
2014-03-18 19:12:58 +01:00
|
|
|
{
|
|
|
|
/* step right one page */
|
|
|
|
buf = _bt_relandgetbuf(rel, buf, opaque->btpo_next, access);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
break;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2000-07-21 08:42:39 +02:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
if (P_IGNORE(opaque))
|
2007-12-31 05:52:05 +01:00
|
|
|
elog(ERROR, "fell off the end of index \"%s\"",
|
2003-02-22 01:45:05 +01:00
|
|
|
RelationGetRelationName(rel));
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2000-07-21 08:42:39 +02:00
|
|
|
* _bt_binsrch() -- Do a binary search for a key on a particular page.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* On an internal (non-leaf) page, _bt_binsrch() returns the OffsetNumber
|
2003-12-21 02:23:06 +01:00
|
|
|
* of the last key < given scankey, or last key <= given scankey if nextkey
|
|
|
|
* is true. (Since _bt_compare treats the first data key of such a page as
|
|
|
|
* minus infinity, there will be at least one key < scankey, so the result
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* always points at one of the keys on the page.)
|
|
|
|
*
|
|
|
|
* On a leaf page, _bt_binsrch() returns the final result of the initial
|
|
|
|
* positioning process that started with _bt_first's call to _bt_search.
|
|
|
|
* We're returning a non-pivot tuple offset, so things are a little different.
|
|
|
|
* It is possible that we'll return an offset that's either past the last
|
|
|
|
* non-pivot slot, or (in the case of a backward scan) before the first slot.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* This procedure is not responsible for walking right, it just examines
|
|
|
|
* the given page. _bt_binsrch() has no lock or refcount side effects
|
|
|
|
* on the buffer.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2019-03-20 17:30:57 +01:00
|
|
|
static OffsetNumber
|
1996-07-09 08:22:35 +02:00
|
|
|
_bt_binsrch(Relation rel,
|
2019-03-20 17:30:57 +01:00
|
|
|
BTScanInsert key,
|
|
|
|
Buffer buf)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
OffsetNumber low,
|
|
|
|
high;
|
2003-12-21 02:23:06 +01:00
|
|
|
int32 result,
|
|
|
|
cmpval;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2019-09-09 20:41:19 +02:00
|
|
|
/* Requesting nextkey semantics while using scantid seems nonsensical */
|
|
|
|
Assert(!key->nextkey || key->scantid == NULL);
|
|
|
|
/* scantid-set callers must use _bt_binsrch_insert() on leaf pages */
|
|
|
|
Assert(!P_ISLEAF(opaque) || key->scantid == NULL);
|
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
low = P_FIRSTDATAKEY(opaque);
|
1997-05-30 20:35:40 +02:00
|
|
|
high = PageGetMaxOffsetNumber(page);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
|
|
|
/*
|
1999-07-17 00:17:06 +02:00
|
|
|
* If there are no keys on the page, return the first available slot. Note
|
|
|
|
* this covers two cases: the page is really empty (no keys), or it
|
|
|
|
* contains only a high key. The latter case is possible after vacuuming.
|
2000-07-21 08:42:39 +02:00
|
|
|
* This can never happen on an internal page, however, since they are
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* never empty (an internal page must have at least one child).
|
1997-05-30 20:35:40 +02:00
|
|
|
*/
|
2019-03-20 17:30:57 +01:00
|
|
|
if (unlikely(high < low))
|
1997-04-16 03:48:29 +02:00
|
|
|
return low;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
/*
|
2003-12-21 02:23:06 +01:00
|
|
|
* Binary search to find the first key on the page >= scan key, or first
|
|
|
|
* key > scankey when nextkey is true.
|
|
|
|
*
|
|
|
|
* For nextkey=false (cmpval=1), the loop invariant is: all slots before
|
|
|
|
* 'low' are < scan key, all slots at or after 'high' are >= scan key.
|
|
|
|
*
|
|
|
|
* For nextkey=true (cmpval=0), the loop invariant is: all slots before
|
|
|
|
* 'low' are <= scan key, all slots at or after 'high' are > scan key.
|
|
|
|
*
|
|
|
|
* We can fall out when high == low.
|
1999-07-17 00:17:06 +02:00
|
|
|
*/
|
|
|
|
high++; /* establish the loop invariant for high */
|
|
|
|
|
2019-03-20 17:30:57 +01:00
|
|
|
cmpval = key->nextkey ? 0 : 1; /* select comparison value */
|
2003-12-21 02:23:06 +01:00
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
while (high > low)
|
1997-04-16 03:48:29 +02:00
|
|
|
{
|
1999-07-17 00:17:06 +02:00
|
|
|
OffsetNumber mid = low + ((high - low) / 2);
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
/* We have low <= mid < high, so mid points at a real slot */
|
|
|
|
|
2019-03-20 17:30:57 +01:00
|
|
|
result = _bt_compare(rel, key, page, mid);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-12-21 02:23:06 +01:00
|
|
|
if (result >= cmpval)
|
1999-07-17 00:17:06 +02:00
|
|
|
low = mid + 1;
|
1997-09-07 07:04:48 +02:00
|
|
|
else
|
1999-07-17 00:17:06 +02:00
|
|
|
high = mid;
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* At this point we have high == low.
|
1997-09-07 07:04:48 +02:00
|
|
|
*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* On a leaf page we always return the first non-pivot tuple >= scan key
|
|
|
|
* (resp. > scan key) for forward scan callers. For backward scans, it's
|
|
|
|
* always the _last_ non-pivot tuple < scan key (resp. <= scan key).
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
2000-07-21 08:42:39 +02:00
|
|
|
if (P_ISLEAF(opaque))
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* In the backward scan case we're supposed to locate the last
|
|
|
|
* matching tuple on the leaf level -- not the first matching tuple
|
|
|
|
* (the last tuple will be the first one returned by the scan).
|
|
|
|
*
|
|
|
|
* At this point we've located the first non-pivot tuple immediately
|
|
|
|
* after the last matching tuple (which might just be maxoff + 1).
|
|
|
|
* Compensate by stepping back.
|
|
|
|
*/
|
|
|
|
if (key->backward)
|
|
|
|
return OffsetNumberPrev(low);
|
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
return low;
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
/*
|
2003-12-21 02:23:06 +01:00
|
|
|
* On a non-leaf page, return the last key < scan key (resp. <= scan key).
|
|
|
|
* There must be one if _bt_compare() is playing by the rules.
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
*
|
|
|
|
* _bt_compare() will seldom see any exactly-matching pivot tuples, since
|
|
|
|
* a truncated -inf heap TID is usually enough to prevent it altogether.
|
|
|
|
* Even omitted scan key entries are treated as > truncated attributes.
|
|
|
|
*
|
|
|
|
* However, during backward scans _bt_compare() interprets omitted scan
|
|
|
|
* key attributes as == corresponding truncated -inf attributes instead.
|
|
|
|
* This works just like < would work here. Under this scheme, < strategy
|
|
|
|
* backward scans will always directly descend to the correct leaf page.
|
|
|
|
* In particular, they will never incur an "extra" leaf page access with a
|
|
|
|
* scan key that happens to contain the same prefix of values as some
|
|
|
|
* pivot tuple's untruncated prefix. VACUUM relies on this guarantee when
|
|
|
|
* it uses a leaf page high key to "re-find" a page undergoing deletion.
|
1999-07-17 00:17:06 +02:00
|
|
|
*/
|
2000-07-21 08:42:39 +02:00
|
|
|
Assert(low > P_FIRSTDATAKEY(opaque));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1999-07-17 00:17:06 +02:00
|
|
|
return OffsetNumberPrev(low);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2019-03-20 17:30:57 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2019-06-14 02:34:34 +02:00
|
|
|
* _bt_binsrch_insert() -- Cacheable, incremental leaf page binary search.
|
2019-03-20 17:30:57 +01:00
|
|
|
*
|
|
|
|
* Like _bt_binsrch(), but with support for caching the binary search
|
|
|
|
* bounds. Only used during insertion, and only on the leaf page that it
|
|
|
|
* looks like caller will insert tuple on. Exclusive-locked and pinned
|
|
|
|
* leaf page is contained within insertstate.
|
|
|
|
*
|
|
|
|
* Caches the bounds fields in insertstate so that a subsequent call can
|
|
|
|
* reuse the low and strict high bounds of original binary search. Callers
|
|
|
|
* that use these fields directly must be prepared for the case where low
|
|
|
|
* and/or stricthigh are not on the same page (one or both exceed maxoff
|
|
|
|
* for the page). The case where there are no items on the page (high <
|
|
|
|
* low) makes bounds invalid.
|
|
|
|
*
|
|
|
|
* Caller is responsible for invalidating bounds when it modifies the page
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
* before calling here a second time, and for dealing with posting list
|
|
|
|
* tuple matches (callers can use insertstate's postingoff field to
|
|
|
|
* determine which existing heap TID will need to be replaced by a posting
|
|
|
|
* list split).
|
2019-03-20 17:30:57 +01:00
|
|
|
*/
|
|
|
|
OffsetNumber
|
|
|
|
_bt_binsrch_insert(Relation rel, BTInsertState insertstate)
|
|
|
|
{
|
|
|
|
BTScanInsert key = insertstate->itup_key;
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
OffsetNumber low,
|
|
|
|
high,
|
|
|
|
stricthigh;
|
|
|
|
int32 result,
|
|
|
|
cmpval;
|
|
|
|
|
|
|
|
page = BufferGetPage(insertstate->buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2019-03-20 17:30:57 +01:00
|
|
|
|
|
|
|
Assert(P_ISLEAF(opaque));
|
|
|
|
Assert(!key->nextkey);
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
Assert(insertstate->postingoff == 0);
|
2019-03-20 17:30:57 +01:00
|
|
|
|
|
|
|
if (!insertstate->bounds_valid)
|
|
|
|
{
|
|
|
|
/* Start new binary search */
|
|
|
|
low = P_FIRSTDATAKEY(opaque);
|
|
|
|
high = PageGetMaxOffsetNumber(page);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Restore result of previous binary search against same page */
|
|
|
|
low = insertstate->low;
|
|
|
|
high = insertstate->stricthigh;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If there are no keys on the page, return the first available slot */
|
|
|
|
if (unlikely(high < low))
|
|
|
|
{
|
|
|
|
/* Caller can't reuse bounds */
|
|
|
|
insertstate->low = InvalidOffsetNumber;
|
|
|
|
insertstate->stricthigh = InvalidOffsetNumber;
|
|
|
|
insertstate->bounds_valid = false;
|
|
|
|
return low;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Binary search to find the first key on the page >= scan key. (nextkey
|
|
|
|
* is always false when inserting).
|
|
|
|
*
|
|
|
|
* The loop invariant is: all slots before 'low' are < scan key, all slots
|
|
|
|
* at or after 'high' are >= scan key. 'stricthigh' is > scan key, and is
|
|
|
|
* maintained to save additional search effort for caller.
|
|
|
|
*
|
|
|
|
* We can fall out when high == low.
|
|
|
|
*/
|
|
|
|
if (!insertstate->bounds_valid)
|
|
|
|
high++; /* establish the loop invariant for high */
|
|
|
|
stricthigh = high; /* high initially strictly higher */
|
|
|
|
|
|
|
|
cmpval = 1; /* !nextkey comparison value */
|
|
|
|
|
|
|
|
while (high > low)
|
|
|
|
{
|
|
|
|
OffsetNumber mid = low + ((high - low) / 2);
|
|
|
|
|
|
|
|
/* We have low <= mid < high, so mid points at a real slot */
|
|
|
|
|
|
|
|
result = _bt_compare(rel, key, page, mid);
|
|
|
|
|
|
|
|
if (result >= cmpval)
|
|
|
|
low = mid + 1;
|
|
|
|
else
|
|
|
|
{
|
|
|
|
high = mid;
|
|
|
|
if (result != 0)
|
|
|
|
stricthigh = high;
|
|
|
|
}
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If tuple at offset located by binary search is a posting list whose
|
|
|
|
* TID range overlaps with caller's scantid, perform posting list
|
|
|
|
* binary search to set postingoff for caller. Caller must split the
|
|
|
|
* posting list when postingoff is set. This should happen
|
|
|
|
* infrequently.
|
|
|
|
*/
|
|
|
|
if (unlikely(result == 0 && key->scantid != NULL))
|
2021-10-27 21:10:47 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* postingoff should never be set more than once per leaf page
|
|
|
|
* binary search. That would mean that there are duplicate table
|
|
|
|
* TIDs in the index, which is never okay. Check for that here.
|
|
|
|
*/
|
|
|
|
if (insertstate->postingoff != 0)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INDEX_CORRUPTED),
|
|
|
|
errmsg_internal("table tid from new index tuple (%u,%u) cannot find insert offset between offsets %u and %u of block %u in index \"%s\"",
|
|
|
|
ItemPointerGetBlockNumber(key->scantid),
|
|
|
|
ItemPointerGetOffsetNumber(key->scantid),
|
|
|
|
low, stricthigh,
|
|
|
|
BufferGetBlockNumber(insertstate->buf),
|
|
|
|
RelationGetRelationName(rel))));
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
insertstate->postingoff = _bt_binsrch_posting(key, page, mid);
|
2021-10-27 21:10:47 +02:00
|
|
|
}
|
2019-03-20 17:30:57 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* On a leaf page, a binary search always returns the first key >= scan
|
|
|
|
* key (at least in !nextkey case), which could be the last slot + 1. This
|
|
|
|
* is also the lower bound of cached search.
|
|
|
|
*
|
|
|
|
* stricthigh may also be the last slot + 1, which prevents caller from
|
|
|
|
* using bounds directly, but is still useful to us if we're called a
|
|
|
|
* second time with cached bounds (cached low will be < stricthigh when
|
|
|
|
* that happens).
|
|
|
|
*/
|
|
|
|
insertstate->low = low;
|
|
|
|
insertstate->stricthigh = stricthigh;
|
|
|
|
insertstate->bounds_valid = true;
|
|
|
|
|
|
|
|
return low;
|
|
|
|
}
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
/*----------
|
|
|
|
* _bt_binsrch_posting() -- posting list binary search.
|
|
|
|
*
|
|
|
|
* Helper routine for _bt_binsrch_insert().
|
|
|
|
*
|
|
|
|
* Returns offset into posting list where caller's scantid belongs.
|
|
|
|
*----------
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
_bt_binsrch_posting(BTScanInsert key, Page page, OffsetNumber offnum)
|
|
|
|
{
|
|
|
|
IndexTuple itup;
|
|
|
|
ItemId itemid;
|
|
|
|
int low,
|
|
|
|
high,
|
|
|
|
mid,
|
|
|
|
res;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If this isn't a posting tuple, then the index must be corrupt (if it is
|
|
|
|
* an ordinary non-pivot tuple then there must be an existing tuple with a
|
|
|
|
* heap TID that equals inserter's new heap TID/scantid). Defensively
|
|
|
|
* check that tuple is a posting list tuple whose posting list range
|
|
|
|
* includes caller's scantid.
|
|
|
|
*
|
|
|
|
* (This is also needed because contrib/amcheck's rootdescend option needs
|
|
|
|
* to be able to relocate a non-pivot tuple using _bt_binsrch_insert().)
|
|
|
|
*/
|
|
|
|
itemid = PageGetItemId(page, offnum);
|
|
|
|
itup = (IndexTuple) PageGetItem(page, itemid);
|
|
|
|
if (!BTreeTupleIsPosting(itup))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
Assert(key->heapkeyspace && key->allequalimage);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In the event that posting list tuple has LP_DEAD bit set, indicate this
|
|
|
|
* to _bt_binsrch_insert() caller by returning -1, a sentinel value. A
|
|
|
|
* second call to _bt_binsrch_insert() can take place when its caller has
|
|
|
|
* removed the dead item.
|
|
|
|
*/
|
|
|
|
if (ItemIdIsDead(itemid))
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
/* "high" is past end of posting list for loop invariant */
|
|
|
|
low = 0;
|
|
|
|
high = BTreeTupleGetNPosting(itup);
|
|
|
|
Assert(high >= 2);
|
|
|
|
|
|
|
|
while (high > low)
|
|
|
|
{
|
|
|
|
mid = low + ((high - low) / 2);
|
|
|
|
res = ItemPointerCompare(key->scantid,
|
|
|
|
BTreeTupleGetPostingN(itup, mid));
|
|
|
|
|
|
|
|
if (res > 0)
|
|
|
|
low = mid + 1;
|
|
|
|
else if (res < 0)
|
|
|
|
high = mid;
|
|
|
|
else
|
|
|
|
return mid;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Exact match not found */
|
|
|
|
return low;
|
|
|
|
}
|
|
|
|
|
2019-03-20 17:30:57 +01:00
|
|
|
/*----------
|
|
|
|
* _bt_compare() -- Compare insertion-type scankey to tuple on a page.
|
2006-01-17 01:09:01 +01:00
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* page/offnum: location of btree item to be compared to.
|
|
|
|
*
|
1996-07-09 08:22:35 +02:00
|
|
|
* This routine returns:
|
2000-05-30 06:25:00 +02:00
|
|
|
* <0 if scankey < tuple at offnum;
|
1996-07-09 08:22:35 +02:00
|
|
|
* 0 if scankey == tuple at offnum;
|
2000-05-30 06:25:00 +02:00
|
|
|
* >0 if scankey > tuple at offnum.
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
*
|
|
|
|
* NULLs in the keys are treated as sortable values. Therefore
|
|
|
|
* "equality" does not necessarily mean that the item should be returned
|
|
|
|
* to the caller as a matching key. Similarly, an insertion scankey
|
|
|
|
* with its scantid set is treated as equal to a posting tuple whose TID
|
|
|
|
* range overlaps with their scantid. There generally won't be a
|
|
|
|
* matching TID in the posting tuple, which caller must handle
|
|
|
|
* themselves (e.g., by splitting the posting list tuple).
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* CRUCIAL NOTE: on a non-leaf page, the first data key is assumed to be
|
|
|
|
* "minus infinity": this routine will always claim it is less than the
|
2019-03-20 17:30:57 +01:00
|
|
|
* scankey. The actual key value stored is explicitly truncated to 0
|
|
|
|
* attributes (explicitly minus infinity) with version 3+ indexes, but
|
|
|
|
* that isn't relied upon. This allows us to implement the Lehman and
|
|
|
|
* Yao convention that the first down-link pointer is before the first
|
|
|
|
* key. See backend/access/nbtree/README for details.
|
2000-07-21 08:42:39 +02:00
|
|
|
*----------
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2000-07-21 08:42:39 +02:00
|
|
|
int32
|
1996-07-09 08:22:35 +02:00
|
|
|
_bt_compare(Relation rel,
|
2019-03-20 17:30:57 +01:00
|
|
|
BTScanInsert key,
|
2000-07-21 08:42:39 +02:00
|
|
|
Page page,
|
1996-07-09 08:22:35 +02:00
|
|
|
OffsetNumber offnum)
|
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
TupleDesc itupdesc = RelationGetDescr(rel);
|
2022-04-01 06:24:50 +02:00
|
|
|
BTPageOpaque opaque = BTPageGetOpaque(page);
|
1996-07-09 08:22:35 +02:00
|
|
|
IndexTuple itup;
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
ItemPointer heapTid;
|
2019-03-20 17:30:57 +01:00
|
|
|
ScanKey scankey;
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
int ncmpkey;
|
|
|
|
int ntupatts;
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
int32 result;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
Assert(_bt_check_natts(rel, key->heapkeyspace, page, offnum));
|
2019-03-20 17:30:57 +01:00
|
|
|
Assert(key->keysz <= IndexRelationGetNumberOfKeyAttributes(rel));
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
Assert(key->heapkeyspace || key->scantid == NULL);
|
2018-04-07 22:00:39 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2000-07-21 08:42:39 +02:00
|
|
|
* Force result ">" if target item is first data item on an internal page
|
|
|
|
* --- see NOTE above.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2000-07-21 08:42:39 +02:00
|
|
|
if (!P_ISLEAF(opaque) && offnum == P_FIRSTDATAKEY(opaque))
|
1996-07-09 08:22:35 +02:00
|
|
|
return 1;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-01-26 00:04:21 +01:00
|
|
|
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
ntupatts = BTreeTupleGetNAtts(itup, rel);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
1996-12-06 10:41:45 +01:00
|
|
|
* The scan key is set up with the attribute number associated with each
|
|
|
|
* term in the key. It is important that, if the index is multi-key, the
|
|
|
|
* scan contain the first k key attributes, and that they be in order. If
|
|
|
|
* you think about how multi-key ordering works, you'll understand why
|
|
|
|
* this is.
|
|
|
|
*
|
2000-07-21 08:42:39 +02:00
|
|
|
* We don't test for violation of this condition here, however. The
|
|
|
|
* initial setup for the index scan had better have gotten it right (see
|
|
|
|
* _bt_first).
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
ncmpkey = Min(ntupatts, key->keysz);
|
|
|
|
Assert(key->heapkeyspace || ncmpkey == key->keysz);
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
Assert(!BTreeTupleIsPosting(itup) || key->allequalimage);
|
2019-03-20 17:30:57 +01:00
|
|
|
scankey = key->scankeys;
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
for (int i = 1; i <= ncmpkey; i++)
|
1997-03-24 09:48:16 +01:00
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
Datum datum;
|
|
|
|
bool isNull;
|
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
datum = index_getattr(itup, scankey->sk_attno, itupdesc, &isNull);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
if (scankey->sk_flags & SK_ISNULL) /* key is NULL */
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
if (isNull)
|
2000-05-30 06:25:00 +02:00
|
|
|
result = 0; /* NULL "=" NULL */
|
2007-01-09 03:14:16 +01:00
|
|
|
else if (scankey->sk_flags & SK_BT_NULLS_FIRST)
|
|
|
|
result = -1; /* NULL "<" NOT_NULL */
|
1997-03-24 09:48:16 +01:00
|
|
|
else
|
2000-05-30 06:25:00 +02:00
|
|
|
result = 1; /* NULL ">" NOT_NULL */
|
1997-03-24 09:48:16 +01:00
|
|
|
}
|
2000-07-21 08:42:39 +02:00
|
|
|
else if (isNull) /* key is NOT_NULL and item is NULL */
|
1997-03-24 09:48:16 +01:00
|
|
|
{
|
2007-01-09 03:14:16 +01:00
|
|
|
if (scankey->sk_flags & SK_BT_NULLS_FIRST)
|
|
|
|
result = 1; /* NOT_NULL ">" NULL */
|
|
|
|
else
|
|
|
|
result = -1; /* NOT_NULL "<" NULL */
|
1997-03-24 09:48:16 +01:00
|
|
|
}
|
|
|
|
else
|
2000-07-21 08:42:39 +02:00
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* The sk_func needs to be passed the index value as left arg and
|
|
|
|
* the sk_argument as right arg (they might be of different
|
|
|
|
* types). Since it is convenient for callers to think of
|
|
|
|
* _bt_compare as comparing the scankey to the index item, we have
|
2007-01-09 03:14:16 +01:00
|
|
|
* to flip the sign of the comparison result. (Unless it's a DESC
|
|
|
|
* column, in which case we *don't* flip the sign.)
|
2003-11-12 22:15:59 +01:00
|
|
|
*/
|
2011-04-13 01:19:24 +02:00
|
|
|
result = DatumGetInt32(FunctionCall2Coll(&scankey->sk_func,
|
|
|
|
scankey->sk_collation,
|
|
|
|
datum,
|
|
|
|
scankey->sk_argument));
|
2007-01-09 03:14:16 +01:00
|
|
|
|
|
|
|
if (!(scankey->sk_flags & SK_BT_DESC))
|
Allow btree comparison functions to return INT_MIN.
Historically we forbade datatype-specific comparison functions from
returning INT_MIN, so that it would be safe to invert the sort order
just by negating the comparison result. However, this was never
really safe for comparison functions that directly return the result
of memcmp(), strcmp(), etc, as POSIX doesn't place any such restriction
on those library functions. Buildfarm results show that at least on
recent Linux on s390x, memcmp() actually does return INT_MIN sometimes,
causing sort failures.
The agreed-on answer is to remove this restriction and fix relevant
call sites to not make such an assumption; code such as "res = -res"
should be replaced by "INVERT_COMPARE_RESULT(res)". The same is needed
in a few places that just directly negated the result of memcmp or
strcmp.
To help find places having this problem, I've also added a compile option
to nbtcompare.c that causes some of the commonly used comparators to
return INT_MIN/INT_MAX instead of their usual -1/+1. It'd likely be
a good idea to have at least one buildfarm member running with
"-DSTRESS_SORT_INT_MIN". That's far from a complete test of course,
but it should help to prevent fresh introductions of such bugs.
This is a longstanding portability hazard, so back-patch to all supported
branches.
Discussion: https://postgr.es/m/20180928185215.ffoq2xrq5d3pafna@alap3.anarazel.de
2018-10-05 22:01:29 +02:00
|
|
|
INVERT_COMPARE_RESULT(result);
|
2000-07-21 08:42:39 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* if the keys are unequal, return the difference */
|
|
|
|
if (result != 0)
|
|
|
|
return result;
|
2003-11-12 22:15:59 +01:00
|
|
|
|
|
|
|
scankey++;
|
1996-10-30 07:08:10 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
/*
|
|
|
|
* All non-truncated attributes (other than heap TID) were found to be
|
|
|
|
* equal. Treat truncated attributes as minus infinity when scankey has a
|
|
|
|
* key attribute value that would otherwise be compared directly.
|
|
|
|
*
|
|
|
|
* Note: it doesn't matter if ntupatts includes non-key attributes;
|
|
|
|
* scankey won't, so explicitly excluding non-key attributes isn't
|
|
|
|
* necessary.
|
|
|
|
*/
|
|
|
|
if (key->keysz > ntupatts)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Use the heap TID attribute and scantid to try to break the tie. The
|
|
|
|
* rules are the same as any other key attribute -- only the
|
|
|
|
* representation differs.
|
|
|
|
*/
|
|
|
|
heapTid = BTreeTupleGetHeapTID(itup);
|
|
|
|
if (key->scantid == NULL)
|
|
|
|
{
|
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* Forward scans have a scankey that is considered greater than a
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
* truncated pivot tuple if and when the scankey has equal values for
|
|
|
|
* attributes up to and including the least significant untruncated
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* attribute in tuple. Even attributes that were omitted from the
|
|
|
|
* scan key are considered greater than -inf truncated attributes.
|
|
|
|
* (See _bt_binsrch for an explanation of our backward scan behavior.)
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
*
|
|
|
|
* For example, if an index has the minimum two attributes (single
|
|
|
|
* user key attribute, plus heap TID attribute), and a page's high key
|
|
|
|
* is ('foo', -inf), and scankey is ('foo', <omitted>), the search
|
|
|
|
* will not descend to the page to the left. The search will descend
|
|
|
|
* right instead. The truncated attribute in pivot tuple means that
|
|
|
|
* all non-pivot tuples on the page to the left are strictly < 'foo',
|
|
|
|
* so it isn't necessary to descend left. In other words, search
|
|
|
|
* doesn't have to descend left because it isn't interested in a match
|
|
|
|
* that has a heap TID value of -inf.
|
|
|
|
*
|
|
|
|
* Note: the heap TID part of the test ensures that scankey is being
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* compared to a pivot tuple with one or more truncated -inf key
|
|
|
|
* attributes. The heap TID attribute is the last key attribute in
|
|
|
|
* every index, of course, but other than that it isn't special.
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
if (!key->backward && key->keysz == ntupatts && heapTid == NULL &&
|
|
|
|
key->heapkeyspace)
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* All provided scankey arguments found to be equal */
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Treat truncated heap TID as minus infinity, since scankey has a key
|
|
|
|
* attribute value (scantid) that would otherwise be compared directly
|
|
|
|
*/
|
|
|
|
Assert(key->keysz == IndexRelationGetNumberOfKeyAttributes(rel));
|
|
|
|
if (heapTid == NULL)
|
|
|
|
return 1;
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
/*
|
|
|
|
* Scankey must be treated as equal to a posting list tuple if its scantid
|
|
|
|
* value falls within the range of the posting list. In all other cases
|
|
|
|
* there can only be a single heap TID value, which is compared directly
|
|
|
|
* with scantid.
|
|
|
|
*/
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
Assert(ntupatts >= IndexRelationGetNumberOfKeyAttributes(rel));
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
result = ItemPointerCompare(key->scantid, heapTid);
|
|
|
|
if (result <= 0 || !BTreeTupleIsPosting(itup))
|
|
|
|
return result;
|
|
|
|
else
|
|
|
|
{
|
|
|
|
result = ItemPointerCompare(key->scantid,
|
|
|
|
BTreeTupleGetMaxHeapTID(itup));
|
|
|
|
if (result > 0)
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_first() -- Find the first item in a scan.
|
|
|
|
*
|
2006-01-17 01:09:01 +01:00
|
|
|
* We need to be clever about the direction of scan, the search
|
|
|
|
* conditions, and the tree ordering. We find the first item (or,
|
|
|
|
* if backwards scan, the last item) in the tree that satisfies the
|
2006-05-07 03:21:30 +02:00
|
|
|
* qualifications in the scan key. On success exit, the page containing
|
|
|
|
* the current index tuple is pinned but not locked, and data about
|
2011-10-09 06:21:08 +02:00
|
|
|
* the matching tuple(s) on the page has been loaded into so->currPos.
|
2024-03-11 23:07:10 +01:00
|
|
|
* scan->xs_heaptid is set to the heap TID of the current tuple, and if
|
|
|
|
* requested, scan->xs_itup points to a copy of the index tuple.
|
2006-05-07 03:21:30 +02:00
|
|
|
*
|
2017-08-16 06:22:32 +02:00
|
|
|
* If there are no matching items in the index, we return false, with no
|
2006-05-07 03:21:30 +02:00
|
|
|
* pins or locks held.
|
2006-01-17 01:09:01 +01:00
|
|
|
*
|
|
|
|
* Note that scan->keyData[], and the so->keyData[] scankey built from it,
|
|
|
|
* are both search-type scankeys (see nbtree/README for more about this).
|
|
|
|
* Within this routine, we build a temporary insertion-type scankey to use
|
|
|
|
* in locating the scan start position.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2002-05-21 01:51:44 +02:00
|
|
|
bool
|
1996-07-09 08:22:35 +02:00
|
|
|
_bt_first(IndexScanDesc scan, ScanDirection dir)
|
|
|
|
{
|
2002-05-24 20:57:57 +02:00
|
|
|
Relation rel = scan->indexRelation;
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
1996-07-09 08:22:35 +02:00
|
|
|
Buffer buf;
|
|
|
|
BTStack stack;
|
2000-07-21 08:42:39 +02:00
|
|
|
OffsetNumber offnum;
|
1996-07-09 08:22:35 +02:00
|
|
|
StrategyNumber strat;
|
2019-03-20 17:30:57 +01:00
|
|
|
BTScanInsertData inskey;
|
2005-06-20 00:41:00 +02:00
|
|
|
ScanKey startKeys[INDEX_MAX_KEYS];
|
2011-11-03 00:35:48 +01:00
|
|
|
ScanKeyData notnullkeys[INDEX_MAX_KEYS];
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
int keysz = 0;
|
2003-11-12 22:15:59 +01:00
|
|
|
int i;
|
2020-09-05 19:17:32 +02:00
|
|
|
bool status;
|
1999-09-27 20:20:21 +02:00
|
|
|
StrategyNumber strat_total;
|
2011-10-09 06:21:08 +02:00
|
|
|
BTScanPosItem *currItem;
|
2017-02-15 13:41:14 +01:00
|
|
|
BlockNumber blkno;
|
2000-04-12 19:17:23 +02:00
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
Assert(!BTScanPosIsValid(so->currPos));
|
|
|
|
|
2007-05-27 05:50:39 +02:00
|
|
|
pgstat_count_index_scan(rel);
|
2005-10-06 04:29:23 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2006-01-23 23:31:41 +01:00
|
|
|
* Examine the scan keys and eliminate any redundant keys; also mark the
|
|
|
|
* keys that must be matched to continue the scan.
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
2003-11-12 22:15:59 +01:00
|
|
|
_bt_preprocess_keys(scan);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-25 06:47:59 +02:00
|
|
|
/*
|
2003-11-12 22:15:59 +01:00
|
|
|
* Quit now if _bt_preprocess_keys() discovered that the scan keys can
|
|
|
|
* never be satisfied (eg, x == 1 AND x > 2).
|
2000-07-25 06:47:59 +02:00
|
|
|
*/
|
|
|
|
if (!so->qual_ok)
|
2020-09-17 11:46:46 +02:00
|
|
|
{
|
|
|
|
_bt_parallel_done(scan);
|
2002-05-21 01:51:44 +02:00
|
|
|
return false;
|
2020-09-17 11:46:46 +02:00
|
|
|
}
|
2000-07-25 06:47:59 +02:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
/*
|
|
|
|
* For parallel scans, get the starting page from shared state. If the
|
|
|
|
* scan has not started, proceed to find out first leaf page in the usual
|
|
|
|
* way while keeping other participating processes waiting. If the scan
|
|
|
|
* has already begun, use the page number from the shared structure.
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
*
|
|
|
|
* When a parallel scan has another primitive index scan scheduled, a
|
|
|
|
* parallel worker will seize the scan for that purpose now. This is
|
|
|
|
* similar to the case where the top-level scan hasn't started.
|
2017-02-15 13:41:14 +01:00
|
|
|
*/
|
|
|
|
if (scan->parallel_scan != NULL)
|
|
|
|
{
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
status = _bt_parallel_seize(scan, &blkno, true);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Initialize arrays (when _bt_parallel_seize didn't already set up
|
|
|
|
* the next primitive index scan)
|
|
|
|
*/
|
|
|
|
if (so->numArrayKeys && !so->needPrimScan)
|
|
|
|
_bt_start_array_keys(scan, dir);
|
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
if (!status)
|
|
|
|
return false;
|
|
|
|
else if (blkno == P_NONE)
|
|
|
|
{
|
|
|
|
_bt_parallel_done(scan);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
else if (blkno != InvalidBlockNumber)
|
|
|
|
{
|
|
|
|
if (!_bt_parallel_readpage(scan, blkno, dir))
|
|
|
|
return false;
|
|
|
|
goto readcomplete;
|
|
|
|
}
|
|
|
|
}
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
else if (so->numArrayKeys && !so->needPrimScan)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* First _bt_first call (for current btrescan) without parallelism.
|
|
|
|
*
|
|
|
|
* Initialize arrays, and the corresponding scan keys that were just
|
|
|
|
* output by _bt_preprocess_keys.
|
|
|
|
*/
|
|
|
|
_bt_start_array_keys(scan, dir);
|
|
|
|
}
|
2017-02-15 13:41:14 +01:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*----------
|
2000-07-25 06:47:59 +02:00
|
|
|
* Examine the scan keys to discover where we need to start the scan.
|
2003-11-12 22:15:59 +01:00
|
|
|
*
|
|
|
|
* We want to identify the keys that can be used as starting boundaries;
|
|
|
|
* these are =, >, or >= keys for a forward scan or =, <, <= keys for
|
|
|
|
* a backwards scan. We can use keys for multiple attributes so long as
|
|
|
|
* the prior attributes had only =, >= (resp. =, <=) keys. Once we accept
|
|
|
|
* a > or < boundary or find an attribute with no boundary (which can be
|
|
|
|
* thought of as the same as "> -infinity"), we can't use keys for any
|
|
|
|
* attributes to its right, because it would break our simplistic notion
|
|
|
|
* of what initial positioning strategy to use.
|
|
|
|
*
|
2006-12-29 00:16:39 +01:00
|
|
|
* When the scan keys include cross-type operators, _bt_preprocess_keys
|
2003-11-12 22:15:59 +01:00
|
|
|
* may not be able to eliminate redundant keys; in such cases we will
|
|
|
|
* arbitrarily pick a usable one for each attribute. This is correct
|
|
|
|
* but possibly not optimal behavior. (For example, with keys like
|
|
|
|
* "x >= 4 AND x >= 5" we would elect to scan starting at x=4 when
|
2006-12-29 00:16:39 +01:00
|
|
|
* x=5 would be more efficient.) Since the situation only arises given
|
|
|
|
* a poorly-worded query plus an incomplete opfamily, live with it.
|
2003-11-12 22:15:59 +01:00
|
|
|
*
|
|
|
|
* When both equality and inequality keys appear for a single attribute
|
2006-12-29 00:16:39 +01:00
|
|
|
* (again, only possible when cross-type operators appear), we *must*
|
2003-11-12 22:15:59 +01:00
|
|
|
* select one of the equality keys for the starting point, because
|
|
|
|
* _bt_checkkeys() will stop the scan as soon as an equality qual fails.
|
|
|
|
* For example, if we have keys like "x >= 4 AND x = 10" and we elect to
|
|
|
|
* start at x=4, we will fail and stop before reaching x=10. If multiple
|
|
|
|
* equality quals survive preprocessing, however, it doesn't matter which
|
|
|
|
* one we use --- by definition, they are either redundant or
|
|
|
|
* contradictory.
|
2006-01-17 01:09:01 +01:00
|
|
|
*
|
2011-11-03 00:35:48 +01:00
|
|
|
* Any regular (not SK_SEARCHNULL) key implies a NOT NULL qualifier.
|
|
|
|
* If the index stores nulls at the end of the index we'll be starting
|
|
|
|
* from, and we have no boundary key for the column (which means the key
|
|
|
|
* we deduced NOT NULL from is an inequality key that constrains the other
|
|
|
|
* end of the index), then we cons up an explicit SK_SEARCHNOTNULL key to
|
|
|
|
* use as a boundary key. If we didn't do this, we might find ourselves
|
|
|
|
* traversing a lot of null entries at the start of the scan.
|
|
|
|
*
|
2006-01-25 21:29:24 +01:00
|
|
|
* In this loop, row-comparison keys are treated the same as keys on their
|
|
|
|
* first (leftmost) columns. We'll add on lower-order columns of the row
|
|
|
|
* comparison below, if possible.
|
|
|
|
*
|
2006-01-17 01:09:01 +01:00
|
|
|
* The selected scan keys (at most one per index column) are remembered by
|
|
|
|
* storing their addresses into the local startKeys[] array.
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
*
|
|
|
|
* _bt_checkkeys/_bt_advance_array_keys decide whether and when to start
|
|
|
|
* the next primitive index scan (for scans with array keys) based in part
|
|
|
|
* on an understanding of how it'll enable us to reposition the scan.
|
|
|
|
* They're directly aware of how we'll sometimes cons up an explicit
|
|
|
|
* SK_SEARCHNOTNULL key. They'll even end primitive scans by applying a
|
|
|
|
* symmetric "deduce NOT NULL" rule of their own. This allows top-level
|
|
|
|
* scans to skip large groups of NULLs through repeated deductions about
|
|
|
|
* key strictness (for a required inequality key) and whether NULLs in the
|
|
|
|
* key's index column are stored last or first (relative to non-NULLs).
|
|
|
|
* If you update anything here, _bt_checkkeys/_bt_advance_array_keys might
|
|
|
|
* need to be kept in sync.
|
2003-11-12 22:15:59 +01:00
|
|
|
*----------
|
2000-07-25 06:47:59 +02:00
|
|
|
*/
|
1999-09-27 20:20:21 +02:00
|
|
|
strat_total = BTEqualStrategyNumber;
|
2000-07-25 06:47:59 +02:00
|
|
|
if (so->numberOfKeys > 0)
|
1999-09-27 20:20:21 +02:00
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
AttrNumber curattr;
|
|
|
|
ScanKey chosen;
|
2011-11-03 00:35:48 +01:00
|
|
|
ScanKey impliesNN;
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKey cur;
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* chosen is the so-far-chosen key for the current attribute, if any.
|
|
|
|
* We don't cast the decision in stone until we reach keys for the
|
|
|
|
* next attribute.
|
|
|
|
*/
|
|
|
|
curattr = 1;
|
|
|
|
chosen = NULL;
|
2011-11-03 00:35:48 +01:00
|
|
|
/* Also remember any scankey that implies a NOT NULL constraint */
|
|
|
|
impliesNN = NULL;
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* Loop iterates from 0 to numberOfKeys inclusive; we use the last
|
|
|
|
* pass to handle after-last-key processing. Actual exit from the
|
|
|
|
* loop is at one of the "break" statements below.
|
|
|
|
*/
|
|
|
|
for (cur = so->keyData, i = 0;; cur++, i++)
|
|
|
|
{
|
|
|
|
if (i >= so->numberOfKeys || cur->sk_attno != curattr)
|
1999-09-27 20:20:21 +02:00
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* Done looking at keys for curattr. If we didn't find a
|
2011-11-03 00:35:48 +01:00
|
|
|
* usable boundary key, see if we can deduce a NOT NULL key.
|
|
|
|
*/
|
|
|
|
if (chosen == NULL && impliesNN != NULL &&
|
|
|
|
((impliesNN->sk_flags & SK_BT_NULLS_FIRST) ?
|
|
|
|
ScanDirectionIsForward(dir) :
|
|
|
|
ScanDirectionIsBackward(dir)))
|
|
|
|
{
|
|
|
|
/* Yes, so build the key in notnullkeys[keysCount] */
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
chosen = ¬nullkeys[keysz];
|
2011-11-03 00:35:48 +01:00
|
|
|
ScanKeyEntryInitialize(chosen,
|
|
|
|
(SK_SEARCHNOTNULL | SK_ISNULL |
|
|
|
|
(impliesNN->sk_flags &
|
|
|
|
(SK_BT_DESC | SK_BT_NULLS_FIRST))),
|
|
|
|
curattr,
|
|
|
|
((impliesNN->sk_flags & SK_BT_NULLS_FIRST) ?
|
|
|
|
BTGreaterStrategyNumber :
|
|
|
|
BTLessStrategyNumber),
|
|
|
|
InvalidOid,
|
|
|
|
InvalidOid,
|
|
|
|
InvalidOid,
|
|
|
|
(Datum) 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we still didn't find a usable boundary key, quit; else
|
|
|
|
* save the boundary key pointer in startKeys.
|
2003-11-12 22:15:59 +01:00
|
|
|
*/
|
|
|
|
if (chosen == NULL)
|
|
|
|
break;
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
startKeys[keysz++] = chosen;
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* Adjust strat_total, and quit if we have stored a > or <
|
|
|
|
* key.
|
|
|
|
*/
|
|
|
|
strat = chosen->sk_strategy;
|
|
|
|
if (strat != BTEqualStrategyNumber)
|
|
|
|
{
|
|
|
|
strat_total = strat;
|
|
|
|
if (strat == BTGreaterStrategyNumber ||
|
|
|
|
strat == BTLessStrategyNumber)
|
|
|
|
break;
|
|
|
|
}
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
2005-06-14 01:14:49 +02:00
|
|
|
* Done if that was the last attribute, or if next key is not
|
|
|
|
* in sequence (implying no boundary key is available for the
|
|
|
|
* next attribute).
|
2003-11-12 22:15:59 +01:00
|
|
|
*/
|
2005-06-14 01:14:49 +02:00
|
|
|
if (i >= so->numberOfKeys ||
|
|
|
|
cur->sk_attno != curattr + 1)
|
1999-04-13 19:18:29 +02:00
|
|
|
break;
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
2005-06-14 01:14:49 +02:00
|
|
|
* Reset for next attr.
|
2003-11-12 22:15:59 +01:00
|
|
|
*/
|
|
|
|
curattr = cur->sk_attno;
|
|
|
|
chosen = NULL;
|
2011-11-03 00:35:48 +01:00
|
|
|
impliesNN = NULL;
|
1999-09-27 20:20:21 +02:00
|
|
|
}
|
2003-11-12 22:15:59 +01:00
|
|
|
|
2011-11-03 00:35:48 +01:00
|
|
|
/*
|
|
|
|
* Can we use this key as a starting boundary for this attr?
|
|
|
|
*
|
|
|
|
* If not, does it imply a NOT NULL constraint? (Because
|
|
|
|
* SK_SEARCHNULL keys are always assigned BTEqualStrategyNumber,
|
|
|
|
* *any* inequality key works for that; we need not test.)
|
|
|
|
*/
|
2003-11-12 22:15:59 +01:00
|
|
|
switch (cur->sk_strategy)
|
1999-09-27 20:20:21 +02:00
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
case BTLessStrategyNumber:
|
|
|
|
case BTLessEqualStrategyNumber:
|
2011-11-03 00:35:48 +01:00
|
|
|
if (chosen == NULL)
|
|
|
|
{
|
|
|
|
if (ScanDirectionIsBackward(dir))
|
|
|
|
chosen = cur;
|
|
|
|
else
|
|
|
|
impliesNN = cur;
|
|
|
|
}
|
2003-11-12 22:15:59 +01:00
|
|
|
break;
|
|
|
|
case BTEqualStrategyNumber:
|
|
|
|
/* override any non-equality choice */
|
|
|
|
chosen = cur;
|
|
|
|
break;
|
|
|
|
case BTGreaterEqualStrategyNumber:
|
|
|
|
case BTGreaterStrategyNumber:
|
2011-11-03 00:35:48 +01:00
|
|
|
if (chosen == NULL)
|
|
|
|
{
|
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
chosen = cur;
|
|
|
|
else
|
|
|
|
impliesNN = cur;
|
|
|
|
}
|
1999-04-13 19:18:29 +02:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
|
|
|
* If we found no usable boundary keys, we have to start from one end of
|
|
|
|
* the tree. Walk down that edge to the first or last key, and scan from
|
|
|
|
* there.
|
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
if (keysz == 0)
|
2017-02-15 13:41:14 +01:00
|
|
|
{
|
|
|
|
bool match;
|
|
|
|
|
|
|
|
match = _bt_endpoint(scan, dir);
|
|
|
|
|
|
|
|
if (!match)
|
|
|
|
{
|
|
|
|
/* No match, so mark (parallel) scan finished */
|
|
|
|
_bt_parallel_done(scan);
|
|
|
|
}
|
|
|
|
|
|
|
|
return match;
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
|
|
|
/*
|
2006-01-17 01:09:01 +01:00
|
|
|
* We want to start the scan somewhere within the index. Set up an
|
|
|
|
* insertion scankey we can use to search for the boundary point we
|
2019-03-20 17:30:57 +01:00
|
|
|
* identified above. The insertion scankey is built using the keys
|
|
|
|
* identified by startKeys[]. (Remaining insertion scankey fields are
|
|
|
|
* initialized after initial-positioning strategy is finalized.)
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
Assert(keysz <= INDEX_MAX_KEYS);
|
|
|
|
for (i = 0; i < keysz; i++)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKey cur = startKeys[i];
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2006-01-25 21:29:24 +01:00
|
|
|
Assert(cur->sk_attno == i + 1);
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2006-01-25 21:29:24 +01:00
|
|
|
if (cur->sk_flags & SK_ROW_HEADER)
|
2003-11-12 22:15:59 +01:00
|
|
|
{
|
2006-01-25 21:29:24 +01:00
|
|
|
/*
|
|
|
|
* Row comparison header: look to the first row member instead.
|
|
|
|
*
|
|
|
|
* The member scankeys are already in insertion format (ie, they
|
|
|
|
* have sk_func = 3-way-comparison function), but we have to watch
|
|
|
|
* out for nulls, which _bt_preprocess_keys didn't check. A null
|
|
|
|
* in the first row member makes the condition unmatchable, just
|
|
|
|
* like qual_ok = false.
|
|
|
|
*/
|
2007-01-09 03:14:16 +01:00
|
|
|
ScanKey subkey = (ScanKey) DatumGetPointer(cur->sk_argument);
|
|
|
|
|
|
|
|
Assert(subkey->sk_flags & SK_ROW_MEMBER);
|
|
|
|
if (subkey->sk_flags & SK_ISNULL)
|
2017-02-15 13:41:14 +01:00
|
|
|
{
|
|
|
|
_bt_parallel_done(scan);
|
2006-01-25 21:29:24 +01:00
|
|
|
return false;
|
2017-02-15 13:41:14 +01:00
|
|
|
}
|
2019-03-20 17:30:57 +01:00
|
|
|
memcpy(inskey.scankeys + i, subkey, sizeof(ScanKeyData));
|
2006-10-04 02:30:14 +02:00
|
|
|
|
2006-01-25 21:29:24 +01:00
|
|
|
/*
|
|
|
|
* If the row comparison is the last positioning key we accepted,
|
|
|
|
* try to add additional keys from the lower-order row members.
|
|
|
|
* (If we accepted independent conditions on additional index
|
|
|
|
* columns, we use those instead --- doesn't seem worth trying to
|
|
|
|
* determine which is more restrictive.) Note that this is OK
|
|
|
|
* even if the row comparison is of ">" or "<" type, because the
|
|
|
|
* condition applied to all but the last row member is effectively
|
|
|
|
* ">=" or "<=", and so the extra keys don't break the positioning
|
2007-01-09 03:14:16 +01:00
|
|
|
* scheme. But, by the same token, if we aren't able to use all
|
|
|
|
* the row members, then the part of the row comparison that we
|
|
|
|
* did use has to be treated as just a ">=" or "<=" condition, and
|
|
|
|
* so we'd better adjust strat_total accordingly.
|
2006-01-25 21:29:24 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
if (i == keysz - 1)
|
2006-01-25 21:29:24 +01:00
|
|
|
{
|
2007-01-09 03:14:16 +01:00
|
|
|
bool used_all_subkeys = false;
|
|
|
|
|
|
|
|
Assert(!(subkey->sk_flags & SK_ROW_END));
|
|
|
|
for (;;)
|
2006-01-25 21:29:24 +01:00
|
|
|
{
|
2007-01-09 03:14:16 +01:00
|
|
|
subkey++;
|
|
|
|
Assert(subkey->sk_flags & SK_ROW_MEMBER);
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
if (subkey->sk_attno != keysz + 1)
|
2006-01-25 21:29:24 +01:00
|
|
|
break; /* out-of-sequence, can't use it */
|
2007-01-09 03:14:16 +01:00
|
|
|
if (subkey->sk_strategy != cur->sk_strategy)
|
|
|
|
break; /* wrong direction, can't use it */
|
|
|
|
if (subkey->sk_flags & SK_ISNULL)
|
2006-01-25 21:29:24 +01:00
|
|
|
break; /* can't use null keys */
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
Assert(keysz < INDEX_MAX_KEYS);
|
|
|
|
memcpy(inskey.scankeys + keysz, subkey,
|
2019-03-20 17:30:57 +01:00
|
|
|
sizeof(ScanKeyData));
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
keysz++;
|
2007-01-09 03:14:16 +01:00
|
|
|
if (subkey->sk_flags & SK_ROW_END)
|
|
|
|
{
|
|
|
|
used_all_subkeys = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!used_all_subkeys)
|
|
|
|
{
|
|
|
|
switch (strat_total)
|
|
|
|
{
|
|
|
|
case BTLessStrategyNumber:
|
|
|
|
strat_total = BTLessEqualStrategyNumber;
|
|
|
|
break;
|
|
|
|
case BTGreaterStrategyNumber:
|
|
|
|
strat_total = BTGreaterEqualStrategyNumber;
|
|
|
|
break;
|
|
|
|
}
|
2006-01-25 21:29:24 +01:00
|
|
|
}
|
|
|
|
break; /* done with outer loop */
|
|
|
|
}
|
2003-11-12 22:15:59 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2006-01-25 21:29:24 +01:00
|
|
|
/*
|
|
|
|
* Ordinary comparison key. Transform the search-style scan key
|
|
|
|
* to an insertion scan key by replacing the sk_func with the
|
|
|
|
* appropriate btree comparison function.
|
|
|
|
*
|
2006-12-29 00:16:39 +01:00
|
|
|
* If scankey operator is not a cross-type comparison, we can use
|
|
|
|
* the cached comparison function; otherwise gotta look it up in
|
|
|
|
* the catalogs. (That can't lead to infinite recursion, since no
|
|
|
|
* indexscan initiated by syscache lookup will use cross-data-type
|
|
|
|
* operators.)
|
|
|
|
*
|
|
|
|
* We support the convention that sk_subtype == InvalidOid means
|
|
|
|
* the opclass input type; this is a hack to simplify life for
|
|
|
|
* ScanKeyInit().
|
2006-01-25 21:29:24 +01:00
|
|
|
*/
|
2006-12-23 01:43:13 +01:00
|
|
|
if (cur->sk_subtype == rel->rd_opcintype[i] ||
|
|
|
|
cur->sk_subtype == InvalidOid)
|
2006-01-25 21:29:24 +01:00
|
|
|
{
|
|
|
|
FmgrInfo *procinfo;
|
|
|
|
|
|
|
|
procinfo = index_getprocinfo(rel, cur->sk_attno, BTORDER_PROC);
|
2019-03-20 17:30:57 +01:00
|
|
|
ScanKeyEntryInitializeWithInfo(inskey.scankeys + i,
|
2006-01-25 21:29:24 +01:00
|
|
|
cur->sk_flags,
|
|
|
|
cur->sk_attno,
|
|
|
|
InvalidStrategy,
|
2006-12-23 01:43:13 +01:00
|
|
|
cur->sk_subtype,
|
2011-04-13 01:19:24 +02:00
|
|
|
cur->sk_collation,
|
2006-01-25 21:29:24 +01:00
|
|
|
procinfo,
|
|
|
|
cur->sk_argument);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
RegProcedure cmp_proc;
|
|
|
|
|
2006-12-23 01:43:13 +01:00
|
|
|
cmp_proc = get_opfamily_proc(rel->rd_opfamily[i],
|
|
|
|
rel->rd_opcintype[i],
|
|
|
|
cur->sk_subtype,
|
|
|
|
BTORDER_PROC);
|
|
|
|
if (!RegProcedureIsValid(cmp_proc))
|
|
|
|
elog(ERROR, "missing support function %d(%u,%u) for attribute %d of index \"%s\"",
|
|
|
|
BTORDER_PROC, rel->rd_opcintype[i], cur->sk_subtype,
|
|
|
|
cur->sk_attno, RelationGetRelationName(rel));
|
2019-03-20 17:30:57 +01:00
|
|
|
ScanKeyEntryInitialize(inskey.scankeys + i,
|
2006-01-25 21:29:24 +01:00
|
|
|
cur->sk_flags,
|
|
|
|
cur->sk_attno,
|
|
|
|
InvalidStrategy,
|
|
|
|
cur->sk_subtype,
|
2011-04-13 01:19:24 +02:00
|
|
|
cur->sk_collation,
|
2006-01-25 21:29:24 +01:00
|
|
|
cmp_proc,
|
|
|
|
cur->sk_argument);
|
|
|
|
}
|
2003-11-12 22:15:59 +01:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2003-11-12 22:15:59 +01:00
|
|
|
|
2006-01-17 01:09:01 +01:00
|
|
|
/*----------
|
2003-12-21 18:52:34 +01:00
|
|
|
* Examine the selected initial-positioning strategy to determine exactly
|
|
|
|
* where we need to start the scan, and set flag variables to control the
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* initial descent by _bt_search (and our _bt_binsrch call for the leaf
|
|
|
|
* page _bt_search returns).
|
2006-01-17 01:09:01 +01:00
|
|
|
*----------
|
2003-12-21 02:23:06 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
_bt_metaversion(rel, &inskey.heapkeyspace, &inskey.allequalimage);
|
|
|
|
inskey.anynullkeys = false; /* unused */
|
|
|
|
inskey.scantid = NULL;
|
|
|
|
inskey.keysz = keysz;
|
2003-12-21 02:23:06 +01:00
|
|
|
switch (strat_total)
|
|
|
|
{
|
|
|
|
case BTLessStrategyNumber:
|
2004-08-29 07:07:03 +02:00
|
|
|
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = false;
|
|
|
|
inskey.backward = true;
|
2003-12-21 02:23:06 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
case BTLessEqualStrategyNumber:
|
2004-08-29 07:07:03 +02:00
|
|
|
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = true;
|
|
|
|
inskey.backward = true;
|
2003-12-21 02:23:06 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
case BTEqualStrategyNumber:
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-12-21 02:23:06 +01:00
|
|
|
/*
|
|
|
|
* If a backward scan was specified, need to start with last equal
|
|
|
|
* item not first one.
|
|
|
|
*/
|
|
|
|
if (ScanDirectionIsBackward(dir))
|
2003-12-21 18:52:34 +01:00
|
|
|
{
|
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* This is the same as the <= strategy
|
2003-12-21 18:52:34 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = true;
|
|
|
|
inskey.backward = true;
|
2003-12-21 18:52:34 +01:00
|
|
|
}
|
2003-12-21 02:23:06 +01:00
|
|
|
else
|
2003-12-21 18:52:34 +01:00
|
|
|
{
|
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* This is the same as the >= strategy
|
2003-12-21 18:52:34 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = false;
|
|
|
|
inskey.backward = false;
|
2003-12-21 18:52:34 +01:00
|
|
|
}
|
2003-12-21 02:23:06 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
case BTGreaterEqualStrategyNumber:
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-12-21 18:52:34 +01:00
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* Find first item >= scankey
|
2003-12-21 18:52:34 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = false;
|
|
|
|
inskey.backward = false;
|
2003-12-21 02:23:06 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
case BTGreaterStrategyNumber:
|
2004-08-29 07:07:03 +02:00
|
|
|
|
2003-12-21 18:52:34 +01:00
|
|
|
/*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* Find first item > scankey
|
2003-12-21 18:52:34 +01:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
inskey.nextkey = true;
|
|
|
|
inskey.backward = false;
|
2003-12-21 02:23:06 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
default:
|
|
|
|
/* can't get here, but keep compiler quiet */
|
|
|
|
elog(ERROR, "unrecognized strat_total: %d", (int) strat_total);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
1997-09-07 07:04:48 +02:00
|
|
|
/*
|
2006-01-17 01:09:01 +01:00
|
|
|
* Use the manufactured insertion scan key to descend the tree and
|
|
|
|
* position ourselves on the target leaf page.
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
Assert(ScanDirectionIsBackward(dir) == inskey.backward);
|
2023-09-08 07:12:12 +02:00
|
|
|
stack = _bt_search(rel, NULL, &inskey, &buf, BT_READ);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* don't need to keep the stack around... */
|
|
|
|
_bt_freestack(stack);
|
|
|
|
|
|
|
|
if (!BufferIsValid(buf))
|
1997-05-30 20:35:40 +02:00
|
|
|
{
|
2011-06-15 10:43:05 +02:00
|
|
|
/*
|
2011-06-16 15:16:34 +02:00
|
|
|
* We only get here if the index is completely empty. Lock relation
|
2023-07-03 06:16:27 +02:00
|
|
|
* because nothing finer to lock exists. Without a buffer lock, it's
|
|
|
|
* possible for another transaction to insert data between
|
|
|
|
* _bt_search() and PredicateLockRelation(). We have to try again
|
|
|
|
* after taking the relation-level predicate lock, to close a narrow
|
|
|
|
* window where we wouldn't scan concurrently inserted tuples, but the
|
|
|
|
* writer wouldn't see our predicate lock.
|
2011-06-15 10:43:05 +02:00
|
|
|
*/
|
2023-07-03 06:16:27 +02:00
|
|
|
if (IsolationIsSerializable())
|
|
|
|
{
|
|
|
|
PredicateLockRelation(rel, scan->xs_snapshot);
|
2023-09-08 07:12:12 +02:00
|
|
|
stack = _bt_search(rel, NULL, &inskey, &buf, BT_READ);
|
2023-07-03 06:16:27 +02:00
|
|
|
_bt_freestack(stack);
|
|
|
|
}
|
2017-02-15 13:41:14 +01:00
|
|
|
|
2023-07-03 06:16:27 +02:00
|
|
|
if (!BufferIsValid(buf))
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Mark parallel scan as done, so that all the workers can finish
|
|
|
|
* their scan.
|
|
|
|
*/
|
|
|
|
_bt_parallel_done(scan);
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
return false;
|
|
|
|
}
|
1997-05-30 20:35:40 +02:00
|
|
|
}
|
2023-07-03 06:16:27 +02:00
|
|
|
|
|
|
|
PredicateLockPage(rel, BufferGetBlockNumber(buf), scan->xs_snapshot);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_initialize_more_data(so, dir);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-25 06:47:59 +02:00
|
|
|
/* position to the precise item on the page */
|
2019-03-20 17:30:57 +01:00
|
|
|
offnum = _bt_binsrch(rel, &inskey, buf);
|
2015-03-25 20:24:43 +01:00
|
|
|
Assert(!BTScanPosIsValid(so->currPos));
|
|
|
|
so->currPos.buf = buf;
|
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/*
|
|
|
|
* Now load data from the first page of the scan.
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
*
|
|
|
|
* If inskey.nextkey = false and inskey.backward = false, offnum is
|
|
|
|
* positioned at the first non-pivot tuple >= inskey.scankeys.
|
|
|
|
*
|
|
|
|
* If inskey.nextkey = false and inskey.backward = true, offnum is
|
|
|
|
* positioned at the last non-pivot tuple < inskey.scankeys.
|
|
|
|
*
|
|
|
|
* If inskey.nextkey = true and inskey.backward = false, offnum is
|
|
|
|
* positioned at the first non-pivot tuple > inskey.scankeys.
|
|
|
|
*
|
|
|
|
* If inskey.nextkey = true and inskey.backward = true, offnum is
|
|
|
|
* positioned at the last non-pivot tuple <= inskey.scankeys.
|
|
|
|
*
|
|
|
|
* It's possible that _bt_binsrch returned an offnum that is out of bounds
|
|
|
|
* for the page. For example, when inskey is both < the leaf page's high
|
|
|
|
* key and > all of its non-pivot tuples, offnum will be "maxoff + 1".
|
2006-05-07 03:21:30 +02:00
|
|
|
*/
|
2023-12-27 13:21:49 +01:00
|
|
|
if (!_bt_readpage(scan, dir, offnum, true))
|
1997-05-30 20:35:40 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/*
|
|
|
|
* There's no actually-matching data on this page. Try to advance to
|
|
|
|
* the next page. Return false if there's no matching data at all.
|
|
|
|
*/
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_unlockbuf(scan->indexRelation, so->currPos.buf);
|
2006-05-07 03:21:30 +02:00
|
|
|
if (!_bt_steppage(scan, dir))
|
2003-12-21 18:52:34 +01:00
|
|
|
return false;
|
|
|
|
}
|
2015-03-25 20:24:43 +01:00
|
|
|
else
|
|
|
|
{
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
/* We have at least one item to return as scan's first item */
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_drop_lock_and_maybe_pin(scan, &so->currPos);
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
readcomplete:
|
2006-05-07 03:21:30 +02:00
|
|
|
/* OK, itemIndex says what to return */
|
2011-10-09 06:21:08 +02:00
|
|
|
currItem = &so->currPos.items[so->currPos.itemIndex];
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
scan->xs_heaptid = currItem->heapTid;
|
2011-10-09 06:21:08 +02:00
|
|
|
if (scan->xs_want_itup)
|
|
|
|
scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_next() -- Get the next item in a scan.
|
|
|
|
*
|
2015-03-25 20:24:43 +01:00
|
|
|
* On entry, so->currPos describes the current page, which may be pinned
|
|
|
|
* but is not locked, and so->currPos.itemIndex identifies which item was
|
2006-05-07 03:21:30 +02:00
|
|
|
* previously returned.
|
|
|
|
*
|
2024-03-11 23:07:10 +01:00
|
|
|
* On successful exit, scan->xs_heaptid is set to the TID of the next
|
|
|
|
* heap tuple, and if requested, scan->xs_itup points to a copy of the
|
|
|
|
* index tuple. so->currPos is updated as needed.
|
2006-05-07 03:21:30 +02:00
|
|
|
*
|
|
|
|
* On failure exit (no more tuples), we release pin and set
|
|
|
|
* so->currPos.buf to InvalidBuffer.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
_bt_next(IndexScanDesc scan, ScanDirection dir)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
2011-10-09 06:21:08 +02:00
|
|
|
BTScanPosItem *currItem;
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Advance to next tuple on current page; or if there's no more, try to
|
|
|
|
* step to the next page with data.
|
|
|
|
*/
|
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
{
|
|
|
|
if (++so->currPos.itemIndex > so->currPos.lastItem)
|
|
|
|
{
|
|
|
|
if (!_bt_steppage(scan, dir))
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
2003-12-21 18:52:34 +01:00
|
|
|
else
|
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
if (--so->currPos.itemIndex < so->currPos.firstItem)
|
2003-12-21 18:52:34 +01:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
if (!_bt_steppage(scan, dir))
|
2002-05-21 01:51:44 +02:00
|
|
|
return false;
|
2003-12-21 18:52:34 +01:00
|
|
|
}
|
1997-05-30 20:35:40 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* OK, itemIndex says what to return */
|
2011-10-09 06:21:08 +02:00
|
|
|
currItem = &so->currPos.items[so->currPos.itemIndex];
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
scan->xs_heaptid = currItem->heapTid;
|
2011-10-09 06:21:08 +02:00
|
|
|
if (scan->xs_want_itup)
|
|
|
|
scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_readpage() -- Load data from current index page into so->currPos
|
|
|
|
*
|
|
|
|
* Caller must have pinned and read-locked so->currPos.buf; the buffer's state
|
|
|
|
* is not changed here. Also, currPos.moreLeft and moreRight must be valid;
|
|
|
|
* they are updated as appropriate. All other fields of so->currPos are
|
|
|
|
* initialized from scratch here.
|
|
|
|
*
|
|
|
|
* We scan the current page starting at offnum and moving in the indicated
|
|
|
|
* direction. All items matching the scan keys are loaded into currPos.items.
|
|
|
|
* moreLeft or moreRight (as appropriate) is cleared if _bt_checkkeys reports
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
* that there can be no more matching tuples in the current scan direction
|
|
|
|
* (could just be for the current primitive index scan when scan has arrays).
|
2006-05-07 03:21:30 +02:00
|
|
|
*
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
* _bt_first caller passes us an offnum returned by _bt_binsrch, which might
|
|
|
|
* be an out of bounds offnum such as "maxoff + 1" in certain corner cases.
|
|
|
|
* _bt_checkkeys will stop the scan as soon as an equality qual fails (when
|
|
|
|
* its scan key was marked required), so _bt_first _must_ pass us an offnum
|
|
|
|
* exactly at the beginning of where equal tuples are to be found. When we're
|
|
|
|
* passed an offnum past the end of the page, we might still manage to stop
|
|
|
|
* the scan on this page by calling _bt_checkkeys against the high key.
|
|
|
|
*
|
2017-02-15 13:41:14 +01:00
|
|
|
* In the case of a parallel scan, caller must have called _bt_parallel_seize
|
|
|
|
* prior to calling this function; this function will invoke
|
|
|
|
* _bt_parallel_release before returning.
|
|
|
|
*
|
2006-05-07 03:21:30 +02:00
|
|
|
* Returns true if any matching items found on the page, false if none.
|
|
|
|
*/
|
|
|
|
static bool
|
2023-12-27 13:21:49 +01:00
|
|
|
_bt_readpage(IndexScanDesc scan, ScanDirection dir, OffsetNumber offnum,
|
|
|
|
bool firstPage)
|
2006-05-07 03:21:30 +02:00
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
OffsetNumber minoff;
|
|
|
|
OffsetNumber maxoff;
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
BTReadPageState pstate;
|
|
|
|
bool arrayKeys;
|
|
|
|
int itemIndex,
|
|
|
|
indnatts;
|
2006-05-07 03:21:30 +02:00
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
/*
|
|
|
|
* We must have the buffer pinned and locked, but the usual macro can't be
|
|
|
|
* used here; this function is what makes it good for currPos.
|
|
|
|
*/
|
2006-05-07 03:21:30 +02:00
|
|
|
Assert(BufferIsValid(so->currPos.buf));
|
|
|
|
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(so->currPos.buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2017-02-15 13:41:14 +01:00
|
|
|
|
|
|
|
/* allow next page be processed by parallel worker */
|
|
|
|
if (scan->parallel_scan)
|
|
|
|
{
|
|
|
|
if (ScanDirectionIsForward(dir))
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.prev_scan_page = opaque->btpo_next;
|
2017-02-15 13:41:14 +01:00
|
|
|
else
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.prev_scan_page = BufferGetBlockNumber(so->currPos.buf);
|
|
|
|
|
|
|
|
_bt_parallel_release(scan, pstate.prev_scan_page);
|
2017-02-15 13:41:14 +01:00
|
|
|
}
|
|
|
|
|
2019-03-23 19:01:53 +01:00
|
|
|
indnatts = IndexRelationGetNumberOfAttributes(scan->indexRelation);
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
arrayKeys = so->numArrayKeys != 0;
|
2006-05-07 03:21:30 +02:00
|
|
|
minoff = P_FIRSTDATAKEY(opaque);
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
/* initialize page-level state that we'll pass to _bt_checkkeys */
|
|
|
|
pstate.dir = dir;
|
|
|
|
pstate.minoff = minoff;
|
|
|
|
pstate.maxoff = maxoff;
|
|
|
|
pstate.finaltup = NULL;
|
|
|
|
pstate.page = page;
|
|
|
|
pstate.offnum = InvalidOffsetNumber;
|
|
|
|
pstate.skip = InvalidOffsetNumber;
|
|
|
|
pstate.continuescan = true; /* default assumption */
|
|
|
|
pstate.prechecked = false;
|
|
|
|
pstate.firstmatch = false;
|
|
|
|
pstate.rechecks = 0;
|
|
|
|
pstate.targetdistance = 0;
|
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
/*
|
|
|
|
* We note the buffer's block number so that we can release the pin later.
|
|
|
|
* This allows us to re-read the buffer if it is needed again for hinting.
|
|
|
|
*/
|
|
|
|
so->currPos.currPage = BufferGetBlockNumber(so->currPos.buf);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We save the LSN of the page as we read it, so that we know whether it
|
|
|
|
* safe to apply LP_DEAD hints to the page later. This allows us to drop
|
|
|
|
* the pin for MVCC scans, which allows vacuum to avoid blocking.
|
|
|
|
*/
|
2018-01-09 19:54:39 +01:00
|
|
|
so->currPos.lsn = BufferGetLSNAtomic(so->currPos.buf);
|
2015-03-25 20:24:43 +01:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/*
|
|
|
|
* we must save the page's right-link while scanning it; this tells us
|
|
|
|
* where to step right to after we're done with these items. There is no
|
|
|
|
* corresponding need for the left-link, since splits always go right.
|
|
|
|
*/
|
|
|
|
so->currPos.nextPage = opaque->btpo_next;
|
|
|
|
|
2011-10-09 06:21:08 +02:00
|
|
|
/* initialize tuple workspace to empty */
|
|
|
|
so->currPos.nextTupleOffset = 0;
|
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
/*
|
|
|
|
* Now that the current page has been made consistent, the macro should be
|
|
|
|
* good.
|
|
|
|
*/
|
|
|
|
Assert(BTScanPosIsPinned(so->currPos));
|
|
|
|
|
2023-10-06 09:40:51 +02:00
|
|
|
/*
|
2023-12-27 13:22:02 +01:00
|
|
|
* Prechecking the value of the continuescan flag for the last item on the
|
|
|
|
* page (for backwards scan it will be the first item on a page). If we
|
|
|
|
* observe it to be true, then it should be true for all other items. This
|
|
|
|
* allows us to do significant optimizations in the _bt_checkkeys()
|
|
|
|
* function for all the items on the page.
|
2023-10-06 09:40:51 +02:00
|
|
|
*
|
|
|
|
* With the forward scan, we do this check for the last item on the page
|
|
|
|
* instead of the high key. It's relatively likely that the most
|
|
|
|
* significant column in the high key will be different from the
|
|
|
|
* corresponding value from the last item on the page. So checking with
|
|
|
|
* the last item on the page would give a more precise answer.
|
|
|
|
*
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
* We skip this for the first page read by each (primitive) scan, to avoid
|
|
|
|
* slowing down point queries. They typically don't stand to gain much
|
|
|
|
* when the optimization can be applied, and are more likely to notice the
|
|
|
|
* overhead of the precheck.
|
|
|
|
*
|
|
|
|
* The optimization is unsafe and must be avoided whenever _bt_checkkeys
|
|
|
|
* just set a low-order required array's key to the best available match
|
|
|
|
* for a truncated -inf attribute value from the prior page's high key
|
|
|
|
* (array element 0 is always the best available match in this scenario).
|
|
|
|
* It's quite likely that matches for array element 0 begin on this page,
|
|
|
|
* but the start of matches won't necessarily align with page boundaries.
|
|
|
|
* When the start of matches is somewhere in the middle of this page, it
|
|
|
|
* would be wrong to treat page's final non-pivot tuple as representative.
|
|
|
|
* Doing so might lead us to treat some of the page's earlier tuples as
|
|
|
|
* being part of a group of tuples thought to satisfy the required keys.
|
|
|
|
*
|
|
|
|
* Note: Conversely, in the case where the scan's arrays just advanced
|
|
|
|
* using the prior page's HIKEY _without_ advancement setting scanBehind,
|
|
|
|
* the start of matches must be aligned with page boundaries, which makes
|
|
|
|
* it safe to attempt the optimization here now. It's also safe when the
|
|
|
|
* prior page's HIKEY simply didn't need to advance any required array. In
|
|
|
|
* both cases we can safely assume that the _first_ tuple from this page
|
|
|
|
* must be >= the current set of array keys/equality constraints. And so
|
|
|
|
* if the final tuple is == those same keys (and also satisfies any
|
|
|
|
* required < or <= strategy scan keys) during the precheck, we can safely
|
|
|
|
* assume that this must also be true of all earlier tuples from the page.
|
2023-10-06 09:40:51 +02:00
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (!firstPage && !so->scanBehind && minoff < maxoff)
|
2023-10-06 09:40:51 +02:00
|
|
|
{
|
|
|
|
ItemId iid;
|
|
|
|
IndexTuple itup;
|
|
|
|
|
|
|
|
iid = PageGetItemId(page, ScanDirectionIsForward(dir) ? maxoff : minoff);
|
|
|
|
itup = (IndexTuple) PageGetItem(page, iid);
|
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
/* Call with arrayKeys=false to avoid undesirable side-effects */
|
|
|
|
_bt_checkkeys(scan, &pstate, false, itup, indnatts);
|
|
|
|
pstate.prechecked = pstate.continuescan;
|
|
|
|
pstate.continuescan = true; /* reset */
|
2023-10-06 09:40:51 +02:00
|
|
|
}
|
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
if (ScanDirectionIsForward(dir))
|
1999-04-13 19:18:29 +02:00
|
|
|
{
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
/* SK_SEARCHARRAY forward scans must provide high key up front */
|
|
|
|
if (arrayKeys && !P_RIGHTMOST(opaque))
|
|
|
|
{
|
|
|
|
ItemId iid = PageGetItemId(page, P_HIKEY);
|
|
|
|
|
|
|
|
pstate.finaltup = (IndexTuple) PageGetItem(page, iid);
|
|
|
|
}
|
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* load items[] in ascending order */
|
|
|
|
itemIndex = 0;
|
|
|
|
|
|
|
|
offnum = Max(offnum, minoff);
|
|
|
|
|
|
|
|
while (offnum <= maxoff)
|
|
|
|
{
|
2019-03-23 19:01:53 +01:00
|
|
|
ItemId iid = PageGetItemId(page, offnum);
|
|
|
|
IndexTuple itup;
|
2023-10-06 09:40:51 +02:00
|
|
|
bool passes_quals;
|
2019-03-23 19:01:53 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the scan specifies not to return killed tuples, then we
|
|
|
|
* treat a killed tuple as not passing the qual
|
|
|
|
*/
|
|
|
|
if (scan->ignore_killed_tuples && ItemIdIsDead(iid))
|
|
|
|
{
|
|
|
|
offnum = OffsetNumberNext(offnum);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
itup = (IndexTuple) PageGetItem(page, iid);
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
Assert(!BTreeTupleIsPivot(itup));
|
2019-03-23 19:01:53 +01:00
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.offnum = offnum;
|
|
|
|
passes_quals = _bt_checkkeys(scan, &pstate, arrayKeys,
|
|
|
|
itup, indnatts);
|
2023-10-06 09:40:51 +02:00
|
|
|
|
|
|
|
/*
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
* Check if we need to skip ahead to a later tuple (only possible
|
|
|
|
* when the scan uses array keys)
|
2023-10-06 09:40:51 +02:00
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (arrayKeys && OffsetNumberIsValid(pstate.skip))
|
|
|
|
{
|
|
|
|
Assert(!passes_quals && pstate.continuescan);
|
|
|
|
Assert(offnum < pstate.skip);
|
|
|
|
|
|
|
|
offnum = pstate.skip;
|
|
|
|
pstate.skip = InvalidOffsetNumber;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2023-10-06 09:40:51 +02:00
|
|
|
if (passes_quals)
|
2006-05-07 03:21:30 +02:00
|
|
|
{
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
/* tuple passes all scan key conditions */
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.firstmatch = true;
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
if (!BTreeTupleIsPosting(itup))
|
|
|
|
{
|
|
|
|
/* Remember it */
|
|
|
|
_bt_saveitem(so, itemIndex, offnum, itup);
|
|
|
|
itemIndex++;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
int tupleOffset;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Set up state to return posting list, and remember first
|
|
|
|
* TID
|
|
|
|
*/
|
|
|
|
tupleOffset =
|
|
|
|
_bt_setuppostingitems(so, itemIndex, offnum,
|
|
|
|
BTreeTupleGetPostingN(itup, 0),
|
|
|
|
itup);
|
|
|
|
itemIndex++;
|
|
|
|
/* Remember additional TIDs */
|
|
|
|
for (int i = 1; i < BTreeTupleGetNPosting(itup); i++)
|
|
|
|
{
|
|
|
|
_bt_savepostingitem(so, itemIndex, offnum,
|
|
|
|
BTreeTupleGetPostingN(itup, i),
|
|
|
|
tupleOffset);
|
|
|
|
itemIndex++;
|
|
|
|
}
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
}
|
2019-03-23 19:01:53 +01:00
|
|
|
/* When !continuescan, there can't be any more matches, so stop */
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (!pstate.continuescan)
|
2006-05-07 03:21:30 +02:00
|
|
|
break;
|
|
|
|
|
|
|
|
offnum = OffsetNumberNext(offnum);
|
|
|
|
}
|
|
|
|
|
2019-03-23 19:01:53 +01:00
|
|
|
/*
|
|
|
|
* We don't need to visit page to the right when the high key
|
|
|
|
* indicates that no more matches will be found there.
|
|
|
|
*
|
|
|
|
* Checking the high key like this works out more often than you might
|
|
|
|
* think. Leaf page splits pick a split point between the two most
|
|
|
|
* dissimilar tuples (this is weighed against the need to evenly share
|
|
|
|
* free space). Leaf pages with high key attribute values that can
|
|
|
|
* only appear on non-pivot tuples on the right sibling page are
|
|
|
|
* common.
|
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (pstate.continuescan && !P_RIGHTMOST(opaque))
|
2019-03-23 19:01:53 +01:00
|
|
|
{
|
|
|
|
ItemId iid = PageGetItemId(page, P_HIKEY);
|
|
|
|
IndexTuple itup = (IndexTuple) PageGetItem(page, iid);
|
|
|
|
int truncatt;
|
|
|
|
|
|
|
|
truncatt = BTreeTupleGetNAtts(itup, scan->indexRelation);
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.prechecked = false; /* precheck didn't cover HIKEY */
|
|
|
|
_bt_checkkeys(scan, &pstate, arrayKeys, itup, truncatt);
|
2019-03-23 19:01:53 +01:00
|
|
|
}
|
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (!pstate.continuescan)
|
2019-03-23 19:01:53 +01:00
|
|
|
so->currPos.moreRight = false;
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
Assert(itemIndex <= MaxTIDsPerBTreePage);
|
2006-05-07 03:21:30 +02:00
|
|
|
so->currPos.firstItem = 0;
|
|
|
|
so->currPos.lastItem = itemIndex - 1;
|
|
|
|
so->currPos.itemIndex = 0;
|
1999-04-13 19:18:29 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
else
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
/* SK_SEARCHARRAY backward scans must provide final tuple up front */
|
|
|
|
if (arrayKeys && minoff <= maxoff && !P_LEFTMOST(opaque))
|
|
|
|
{
|
|
|
|
ItemId iid = PageGetItemId(page, minoff);
|
|
|
|
|
|
|
|
pstate.finaltup = (IndexTuple) PageGetItem(page, iid);
|
|
|
|
}
|
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* load items[] in descending order */
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
itemIndex = MaxTIDsPerBTreePage;
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
offnum = Min(offnum, maxoff);
|
|
|
|
|
|
|
|
while (offnum >= minoff)
|
|
|
|
{
|
2019-03-23 19:01:53 +01:00
|
|
|
ItemId iid = PageGetItemId(page, offnum);
|
|
|
|
IndexTuple itup;
|
|
|
|
bool tuple_alive;
|
|
|
|
bool passes_quals;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the scan specifies not to return killed tuples, then we
|
|
|
|
* treat a killed tuple as not passing the qual. Most of the
|
|
|
|
* time, it's a win to not bother examining the tuple's index
|
|
|
|
* keys, but just skip to the next tuple (previous, actually,
|
|
|
|
* since we're scanning backwards). However, if this is the first
|
|
|
|
* tuple on the page, we do check the index keys, to prevent
|
|
|
|
* uselessly advancing to the page to the left. This is similar
|
|
|
|
* to the high key optimization used by forward scans.
|
|
|
|
*/
|
|
|
|
if (scan->ignore_killed_tuples && ItemIdIsDead(iid))
|
|
|
|
{
|
|
|
|
Assert(offnum >= P_FIRSTDATAKEY(opaque));
|
|
|
|
if (offnum > P_FIRSTDATAKEY(opaque))
|
|
|
|
{
|
|
|
|
offnum = OffsetNumberPrev(offnum);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
tuple_alive = false;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
tuple_alive = true;
|
|
|
|
|
|
|
|
itup = (IndexTuple) PageGetItem(page, iid);
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
Assert(!BTreeTupleIsPivot(itup));
|
2019-03-23 19:01:53 +01:00
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.offnum = offnum;
|
|
|
|
passes_quals = _bt_checkkeys(scan, &pstate, arrayKeys,
|
|
|
|
itup, indnatts);
|
2023-10-06 09:40:51 +02:00
|
|
|
|
|
|
|
/*
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
* Check if we need to skip ahead to a later tuple (only possible
|
|
|
|
* when the scan uses array keys)
|
2023-10-06 09:40:51 +02:00
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (arrayKeys && OffsetNumberIsValid(pstate.skip))
|
|
|
|
{
|
|
|
|
Assert(!passes_quals && pstate.continuescan);
|
|
|
|
Assert(offnum > pstate.skip);
|
|
|
|
|
|
|
|
offnum = pstate.skip;
|
|
|
|
pstate.skip = InvalidOffsetNumber;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2019-03-23 19:01:53 +01:00
|
|
|
if (passes_quals && tuple_alive)
|
2006-05-07 03:21:30 +02:00
|
|
|
{
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
/* tuple passes all scan key conditions */
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
pstate.firstmatch = true;
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
if (!BTreeTupleIsPosting(itup))
|
|
|
|
{
|
|
|
|
/* Remember it */
|
|
|
|
itemIndex--;
|
|
|
|
_bt_saveitem(so, itemIndex, offnum, itup);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
int tupleOffset;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Set up state to return posting list, and remember first
|
|
|
|
* TID.
|
|
|
|
*
|
|
|
|
* Note that we deliberately save/return items from
|
|
|
|
* posting lists in ascending heap TID order for backwards
|
|
|
|
* scans. This allows _bt_killitems() to make a
|
|
|
|
* consistent assumption about the order of items
|
|
|
|
* associated with the same posting list tuple.
|
|
|
|
*/
|
|
|
|
itemIndex--;
|
|
|
|
tupleOffset =
|
|
|
|
_bt_setuppostingitems(so, itemIndex, offnum,
|
|
|
|
BTreeTupleGetPostingN(itup, 0),
|
|
|
|
itup);
|
|
|
|
/* Remember additional TIDs */
|
|
|
|
for (int i = 1; i < BTreeTupleGetNPosting(itup); i++)
|
|
|
|
{
|
|
|
|
itemIndex--;
|
|
|
|
_bt_savepostingitem(so, itemIndex, offnum,
|
|
|
|
BTreeTupleGetPostingN(itup, i),
|
|
|
|
tupleOffset);
|
|
|
|
}
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
}
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
if (!pstate.continuescan)
|
2006-05-07 03:21:30 +02:00
|
|
|
{
|
|
|
|
/* there can't be any more matches, so stop */
|
|
|
|
so->currPos.moreLeft = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
offnum = OffsetNumberPrev(offnum);
|
|
|
|
}
|
|
|
|
|
|
|
|
Assert(itemIndex >= 0);
|
|
|
|
so->currPos.firstItem = itemIndex;
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
so->currPos.lastItem = MaxTIDsPerBTreePage - 1;
|
|
|
|
so->currPos.itemIndex = MaxTIDsPerBTreePage - 1;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
return (so->currPos.firstItem <= so->currPos.lastItem);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2011-10-09 06:21:08 +02:00
|
|
|
/* Save an index item into so->currPos.items[itemIndex] */
|
|
|
|
static void
|
|
|
|
_bt_saveitem(BTScanOpaque so, int itemIndex,
|
|
|
|
OffsetNumber offnum, IndexTuple itup)
|
|
|
|
{
|
|
|
|
BTScanPosItem *currItem = &so->currPos.items[itemIndex];
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
Assert(!BTreeTupleIsPivot(itup) && !BTreeTupleIsPosting(itup));
|
|
|
|
|
2011-10-09 06:21:08 +02:00
|
|
|
currItem->heapTid = itup->t_tid;
|
|
|
|
currItem->indexOffset = offnum;
|
|
|
|
if (so->currTuples)
|
|
|
|
{
|
|
|
|
Size itupsz = IndexTupleSize(itup);
|
|
|
|
|
|
|
|
currItem->tupleOffset = so->currPos.nextTupleOffset;
|
|
|
|
memcpy(so->currTuples + so->currPos.nextTupleOffset, itup, itupsz);
|
|
|
|
so->currPos.nextTupleOffset += MAXALIGN(itupsz);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 22:05:30 +01:00
|
|
|
/*
|
|
|
|
* Setup state to save TIDs/items from a single posting list tuple.
|
|
|
|
*
|
|
|
|
* Saves an index item into so->currPos.items[itemIndex] for TID that is
|
|
|
|
* returned to scan first. Second or subsequent TIDs for posting list should
|
|
|
|
* be saved by calling _bt_savepostingitem().
|
|
|
|
*
|
|
|
|
* Returns an offset into tuple storage space that main tuple is stored at if
|
|
|
|
* needed.
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
_bt_setuppostingitems(BTScanOpaque so, int itemIndex, OffsetNumber offnum,
|
|
|
|
ItemPointer heapTid, IndexTuple itup)
|
|
|
|
{
|
|
|
|
BTScanPosItem *currItem = &so->currPos.items[itemIndex];
|
|
|
|
|
|
|
|
Assert(BTreeTupleIsPosting(itup));
|
|
|
|
|
|
|
|
currItem->heapTid = *heapTid;
|
|
|
|
currItem->indexOffset = offnum;
|
|
|
|
if (so->currTuples)
|
|
|
|
{
|
|
|
|
/* Save base IndexTuple (truncate posting list) */
|
|
|
|
IndexTuple base;
|
|
|
|
Size itupsz = BTreeTupleGetPostingOffset(itup);
|
|
|
|
|
|
|
|
itupsz = MAXALIGN(itupsz);
|
|
|
|
currItem->tupleOffset = so->currPos.nextTupleOffset;
|
|
|
|
base = (IndexTuple) (so->currTuples + so->currPos.nextTupleOffset);
|
|
|
|
memcpy(base, itup, itupsz);
|
|
|
|
/* Defensively reduce work area index tuple header size */
|
|
|
|
base->t_info &= ~INDEX_SIZE_MASK;
|
|
|
|
base->t_info |= itupsz;
|
|
|
|
so->currPos.nextTupleOffset += itupsz;
|
|
|
|
|
|
|
|
return currItem->tupleOffset;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Save an index item into so->currPos.items[itemIndex] for current posting
|
|
|
|
* tuple.
|
|
|
|
*
|
|
|
|
* Assumes that _bt_setuppostingitems() has already been called for current
|
|
|
|
* posting list tuple. Caller passes its return value as tupleOffset.
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
_bt_savepostingitem(BTScanOpaque so, int itemIndex, OffsetNumber offnum,
|
|
|
|
ItemPointer heapTid, int tupleOffset)
|
|
|
|
{
|
|
|
|
BTScanPosItem *currItem = &so->currPos.items[itemIndex];
|
|
|
|
|
|
|
|
currItem->heapTid = *heapTid;
|
|
|
|
currItem->indexOffset = offnum;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Have index-only scans return the same base IndexTuple for every TID
|
|
|
|
* that originates from the same posting list
|
|
|
|
*/
|
|
|
|
if (so->currTuples)
|
|
|
|
currItem->tupleOffset = tupleOffset;
|
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2006-05-07 03:21:30 +02:00
|
|
|
* _bt_steppage() -- Step to next page containing valid data for scan
|
|
|
|
*
|
2015-03-25 20:24:43 +01:00
|
|
|
* On entry, if so->currPos.buf is valid the buffer is pinned but not locked;
|
|
|
|
* if pinned, we'll drop the pin before moving to next page. The buffer is
|
|
|
|
* not locked on entry.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2017-02-15 13:41:14 +01:00
|
|
|
* For success on a scan using a non-MVCC snapshot we hold a pin, but not a
|
|
|
|
* read lock, on that page. If we do not hold the pin, we set so->currPos.buf
|
2017-08-16 06:22:32 +02:00
|
|
|
* to InvalidBuffer. We return true to indicate success.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2006-05-07 03:21:30 +02:00
|
|
|
static bool
|
|
|
|
_bt_steppage(IndexScanDesc scan, ScanDirection dir)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
2017-02-15 13:41:14 +01:00
|
|
|
BlockNumber blkno = InvalidBlockNumber;
|
2020-09-05 19:17:32 +02:00
|
|
|
bool status;
|
1999-05-25 18:15:34 +02:00
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
Assert(BTScanPosIsValid(so->currPos));
|
2000-07-21 08:42:39 +02:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Before leaving current page, deal with any killed items */
|
|
|
|
if (so->numKilled > 0)
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_killitems(scan);
|
2006-05-07 03:21:30 +02:00
|
|
|
|
2006-08-24 03:18:34 +02:00
|
|
|
/*
|
|
|
|
* Before we modify currPos, make a copy of the page data if there was a
|
|
|
|
* mark position that needs it.
|
|
|
|
*/
|
|
|
|
if (so->markItemIndex >= 0)
|
|
|
|
{
|
|
|
|
/* bump pin on current buffer for assignment to mark buffer */
|
2015-03-25 20:24:43 +01:00
|
|
|
if (BTScanPosIsPinned(so->currPos))
|
|
|
|
IncrBufferRefCount(so->currPos.buf);
|
2006-08-24 03:18:34 +02:00
|
|
|
memcpy(&so->markPos, &so->currPos,
|
|
|
|
offsetof(BTScanPosData, items[1]) +
|
|
|
|
so->currPos.lastItem * sizeof(BTScanPosItem));
|
2011-10-09 06:21:08 +02:00
|
|
|
if (so->markTuples)
|
|
|
|
memcpy(so->markTuples, so->currTuples,
|
|
|
|
so->currPos.nextTupleOffset);
|
2006-08-24 03:18:34 +02:00
|
|
|
so->markPos.itemIndex = so->markItemIndex;
|
|
|
|
so->markItemIndex = -1;
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we're just about to start the next primitive index scan
|
|
|
|
* (possible with a scan that has arrays keys, and needs to skip to
|
|
|
|
* continue in the current scan direction), moreLeft/moreRight only
|
|
|
|
* indicate the end of the current primitive index scan. They must
|
|
|
|
* never be taken to indicate that the top-level index scan has ended
|
|
|
|
* (that would be wrong).
|
|
|
|
*
|
|
|
|
* We could handle this case by treating the current array keys as
|
|
|
|
* markPos state. But depending on the current array state like this
|
|
|
|
* would add complexity. Instead, we just unset markPos's copy of
|
|
|
|
* moreRight or moreLeft (whichever might be affected), while making
|
|
|
|
* btrestpos reset the scan's arrays to their initial scan positions.
|
|
|
|
* In effect, btrestpos leaves advancing the arrays up to the first
|
|
|
|
* _bt_readpage call (that takes place after it has restored markPos).
|
|
|
|
*/
|
|
|
|
Assert(so->markPos.dir == dir);
|
|
|
|
if (so->needPrimScan)
|
|
|
|
{
|
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
so->markPos.moreRight = true;
|
|
|
|
else
|
|
|
|
so->markPos.moreLeft = true;
|
|
|
|
}
|
2006-08-24 03:18:34 +02:00
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Walk right to the next page with data */
|
2017-02-15 13:41:14 +01:00
|
|
|
if (scan->parallel_scan != NULL)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Seize the scan to get the next block number; if the scan has
|
|
|
|
* ended already, bail out.
|
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
status = _bt_parallel_seize(scan, &blkno, false);
|
2017-02-15 13:41:14 +01:00
|
|
|
if (!status)
|
|
|
|
{
|
|
|
|
/* release the previous buffer, if pinned */
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Not parallel, so use the previously-saved nextPage link. */
|
|
|
|
blkno = so->currPos.nextPage;
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
/* Remember we left a page with data */
|
|
|
|
so->currPos.moreLeft = true;
|
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
/* release the previous buffer, if pinned */
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
2017-02-15 13:41:14 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Remember we left a page with data */
|
|
|
|
so->currPos.moreRight = true;
|
|
|
|
|
|
|
|
if (scan->parallel_scan != NULL)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Seize the scan to get the current block number; if the scan has
|
|
|
|
* ended already, bail out.
|
|
|
|
*/
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
status = _bt_parallel_seize(scan, &blkno, false);
|
2017-02-15 13:41:14 +01:00
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
|
|
|
if (!status)
|
|
|
|
{
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Not parallel, so just use our own notion of the current page */
|
|
|
|
blkno = so->currPos.currPage;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!_bt_readnextpage(scan, blkno, dir))
|
|
|
|
return false;
|
|
|
|
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
/* We have at least one item to return as scan's next item */
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_drop_lock_and_maybe_pin(scan, &so->currPos);
|
2015-03-25 20:24:43 +01:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_readnextpage() -- Read next page containing valid data for scan
|
|
|
|
*
|
|
|
|
* On success exit, so->currPos is updated to contain data from the next
|
2023-12-09 00:37:53 +01:00
|
|
|
* interesting page, and we return true. Caller must release the lock (and
|
|
|
|
* maybe the pin) on the buffer on success exit.
|
2017-02-15 13:41:14 +01:00
|
|
|
*
|
|
|
|
* If there are no more matching records in the given direction, we drop all
|
2017-08-16 06:22:32 +02:00
|
|
|
* locks and pins, set so->currPos.buf to InvalidBuffer, and return false.
|
2017-02-15 13:41:14 +01:00
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
_bt_readnextpage(IndexScanDesc scan, BlockNumber blkno, ScanDirection dir)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
Relation rel;
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
2020-09-05 19:17:32 +02:00
|
|
|
bool status;
|
2017-02-15 13:41:14 +01:00
|
|
|
|
|
|
|
rel = scan->indexRelation;
|
|
|
|
|
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
for (;;)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2017-02-15 13:41:14 +01:00
|
|
|
/*
|
|
|
|
* if we're at end of scan, give up and mark parallel scan as
|
|
|
|
* done, so that all the workers can finish their scan
|
|
|
|
*/
|
2006-05-07 03:21:30 +02:00
|
|
|
if (blkno == P_NONE || !so->currPos.moreRight)
|
2015-03-25 20:24:43 +01:00
|
|
|
{
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_parallel_done(scan);
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosInvalidate(so->currPos);
|
2006-05-07 03:21:30 +02:00
|
|
|
return false;
|
2015-03-25 20:24:43 +01:00
|
|
|
}
|
2009-05-05 21:36:32 +02:00
|
|
|
/* check for interrupts while we're not holding any buffer lock */
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
2006-05-07 03:21:30 +02:00
|
|
|
/* step right one page */
|
2023-06-10 23:08:25 +02:00
|
|
|
so->currPos.buf = _bt_getbuf(rel, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(so->currPos.buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2017-01-31 23:12:54 +01:00
|
|
|
/* check for deleted page */
|
2006-05-07 03:21:30 +02:00
|
|
|
if (!P_IGNORE(opaque))
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2011-06-15 10:43:05 +02:00
|
|
|
PredicateLockPage(rel, blkno, scan->xs_snapshot);
|
2006-05-07 03:21:30 +02:00
|
|
|
/* see if there are any matches on this page */
|
|
|
|
/* note that this will clear moreRight if we can stop */
|
2023-12-27 13:21:49 +01:00
|
|
|
if (_bt_readpage(scan, dir, P_FIRSTDATAKEY(opaque), false))
|
2006-05-07 03:21:30 +02:00
|
|
|
break;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2017-12-13 22:09:00 +01:00
|
|
|
else if (scan->parallel_scan != NULL)
|
|
|
|
{
|
|
|
|
/* allow next page be processed by parallel worker */
|
|
|
|
_bt_parallel_release(scan, opaque->btpo_next);
|
|
|
|
}
|
2015-03-25 20:24:43 +01:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* nope, keep going */
|
2017-02-15 13:41:14 +01:00
|
|
|
if (scan->parallel_scan != NULL)
|
|
|
|
{
|
2018-07-27 07:23:00 +02:00
|
|
|
_bt_relbuf(rel, so->currPos.buf);
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
status = _bt_parallel_seize(scan, &blkno, false);
|
2017-02-15 13:41:14 +01:00
|
|
|
if (!status)
|
|
|
|
{
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
2018-07-27 07:23:00 +02:00
|
|
|
{
|
2017-02-15 13:41:14 +01:00
|
|
|
blkno = opaque->btpo_next;
|
2018-07-27 07:23:00 +02:00
|
|
|
_bt_relbuf(rel, so->currPos.buf);
|
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
else
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2017-02-15 13:41:14 +01:00
|
|
|
/*
|
|
|
|
* Should only happen in parallel cases, when some other backend
|
|
|
|
* advanced the scan.
|
|
|
|
*/
|
|
|
|
if (so->currPos.currPage != blkno)
|
|
|
|
{
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
|
|
|
so->currPos.currPage = blkno;
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Walk left to the next page with data. This is much more complex
|
|
|
|
* than the walk-right case because of the possibility that the page
|
|
|
|
* to our left splits while we are in flight to it, plus the
|
|
|
|
* possibility that the page we were on gets deleted after we leave
|
|
|
|
* it. See nbtree/README for details.
|
2015-03-25 20:24:43 +01:00
|
|
|
*
|
|
|
|
* It might be possible to rearrange this code to have less overhead
|
|
|
|
* in pinning and locking, but that would require capturing the left
|
2023-12-09 00:37:53 +01:00
|
|
|
* sibling block number when the page is initially read, and then
|
|
|
|
* optimistically starting there (rather than pinning the page twice).
|
|
|
|
* It is not clear that this would be worth the complexity.
|
2006-05-07 03:21:30 +02:00
|
|
|
*/
|
2015-03-25 20:24:43 +01:00
|
|
|
if (BTScanPosIsPinned(so->currPos))
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_lockbuf(rel, so->currPos.buf, BT_READ);
|
2015-03-25 20:24:43 +01:00
|
|
|
else
|
2023-06-10 23:08:25 +02:00
|
|
|
so->currPos.buf = _bt_getbuf(rel, so->currPos.currPage, BT_READ);
|
2015-03-25 20:24:43 +01:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
for (;;)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Done if we know there are no matching keys to the left */
|
|
|
|
if (!so->currPos.moreLeft)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
_bt_relbuf(rel, so->currPos.buf);
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_parallel_done(scan);
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosInvalidate(so->currPos);
|
2006-05-07 03:21:30 +02:00
|
|
|
return false;
|
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Step to next physical page */
|
2023-09-08 07:12:12 +02:00
|
|
|
so->currPos.buf = _bt_walk_left(rel, so->currPos.buf);
|
2003-08-04 02:43:34 +02:00
|
|
|
|
2006-05-07 03:21:30 +02:00
|
|
|
/* if we're physically at end of index, return failure */
|
|
|
|
if (so->currPos.buf == InvalidBuffer)
|
2015-03-25 20:24:43 +01:00
|
|
|
{
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_parallel_done(scan);
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosInvalidate(so->currPos);
|
2006-05-07 03:21:30 +02:00
|
|
|
return false;
|
2015-03-25 20:24:43 +01:00
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Okay, we managed to move left to a non-deleted page. Done if
|
|
|
|
* it's not half-dead and contains matching tuples. Else loop back
|
|
|
|
* and do it all again.
|
|
|
|
*/
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(so->currPos.buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2006-05-07 03:21:30 +02:00
|
|
|
if (!P_IGNORE(opaque))
|
|
|
|
{
|
2011-06-15 10:43:05 +02:00
|
|
|
PredicateLockPage(rel, BufferGetBlockNumber(so->currPos.buf), scan->xs_snapshot);
|
2006-05-07 03:21:30 +02:00
|
|
|
/* see if there are any matches on this page */
|
|
|
|
/* note that this will clear moreLeft if we can stop */
|
2023-12-27 13:21:49 +01:00
|
|
|
if (_bt_readpage(scan, dir, PageGetMaxOffsetNumber(page), false))
|
2006-05-07 03:21:30 +02:00
|
|
|
break;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2017-12-13 22:09:00 +01:00
|
|
|
else if (scan->parallel_scan != NULL)
|
|
|
|
{
|
|
|
|
/* allow next page be processed by parallel worker */
|
|
|
|
_bt_parallel_release(scan, BufferGetBlockNumber(so->currPos.buf));
|
|
|
|
}
|
2017-02-15 13:41:14 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* For parallel scans, get the last page scanned as it is quite
|
|
|
|
* possible that by the time we try to seize the scan, some other
|
|
|
|
* worker has already advanced the scan to a different page. We
|
|
|
|
* must continue based on the latest page scanned by any worker.
|
|
|
|
*/
|
|
|
|
if (scan->parallel_scan != NULL)
|
|
|
|
{
|
|
|
|
_bt_relbuf(rel, so->currPos.buf);
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
status = _bt_parallel_seize(scan, &blkno, false);
|
2017-02-15 13:41:14 +01:00
|
|
|
if (!status)
|
|
|
|
{
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
return false;
|
|
|
|
}
|
2023-06-10 23:08:25 +02:00
|
|
|
so->currPos.buf = _bt_getbuf(rel, blkno, BT_READ);
|
2017-02-15 13:41:14 +01:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
}
|
2000-07-21 08:42:39 +02:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_parallel_readpage() -- Read current page containing valid data for scan
|
|
|
|
*
|
2017-08-16 06:22:32 +02:00
|
|
|
* On success, release lock and maybe pin on buffer. We return true to
|
2017-02-15 13:41:14 +01:00
|
|
|
* indicate success.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
_bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno, ScanDirection dir)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
Assert(!so->needPrimScan);
|
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_initialize_more_data(so, dir);
|
|
|
|
|
|
|
|
if (!_bt_readnextpage(scan, blkno, dir))
|
|
|
|
return false;
|
|
|
|
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
/* We have at least one item to return as scan's next item */
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_drop_lock_and_maybe_pin(scan, &so->currPos);
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/*
|
|
|
|
* _bt_walk_left() -- step left one page, if possible
|
|
|
|
*
|
|
|
|
* The given buffer must be pinned and read-locked. This will be dropped
|
|
|
|
* before stepping left. On return, we have pin and read lock on the
|
|
|
|
* returned page, instead.
|
|
|
|
*
|
|
|
|
* Returns InvalidBuffer if there is no page to the left (no lock is held
|
|
|
|
* in that case).
|
|
|
|
*
|
2023-12-09 00:37:53 +01:00
|
|
|
* It is possible for the returned leaf page to be half-dead; caller must
|
|
|
|
* check that condition and step left again when required.
|
2003-02-22 01:45:05 +01:00
|
|
|
*/
|
|
|
|
static Buffer
|
2023-09-08 07:12:12 +02:00
|
|
|
_bt_walk_left(Relation rel, Buffer buf)
|
2003-02-22 01:45:05 +01:00
|
|
|
{
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-22 01:45:05 +01:00
|
|
|
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
BlockNumber obknum;
|
|
|
|
BlockNumber lblkno;
|
|
|
|
BlockNumber blkno;
|
|
|
|
int tries;
|
|
|
|
|
|
|
|
/* if we're at end of tree, release buf and return failure */
|
|
|
|
if (P_LEFTMOST(opaque))
|
|
|
|
{
|
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* remember original page we are stepping left from */
|
|
|
|
obknum = BufferGetBlockNumber(buf);
|
|
|
|
/* step left */
|
|
|
|
blkno = lblkno = opaque->btpo_prev;
|
2009-05-05 21:36:32 +02:00
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
/* check for interrupts while we're not holding any buffer lock */
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
2023-06-10 23:08:25 +02:00
|
|
|
buf = _bt_getbuf(rel, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-08-04 02:43:34 +02:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/*
|
|
|
|
* If this isn't the page we want, walk right till we find what we
|
|
|
|
* want --- but go no more than four hops (an arbitrary limit). If we
|
|
|
|
* don't find the correct page by then, the most likely bet is that
|
|
|
|
* the original page got deleted and isn't in the sibling chain at all
|
|
|
|
* anymore, not that its left sibling got split more than four times.
|
|
|
|
*
|
|
|
|
* Note that it is correct to test P_ISDELETED not P_IGNORE here,
|
2023-12-09 00:37:53 +01:00
|
|
|
* because half-dead pages are still in the sibling chain.
|
2003-02-22 01:45:05 +01:00
|
|
|
*/
|
|
|
|
tries = 0;
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
if (!P_ISDELETED(opaque) && opaque->btpo_next == obknum)
|
|
|
|
{
|
|
|
|
/* Found desired page, return it */
|
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
if (P_RIGHTMOST(opaque) || ++tries > 4)
|
|
|
|
break;
|
|
|
|
blkno = opaque->btpo_next;
|
2004-04-21 20:24:26 +02:00
|
|
|
buf = _bt_relandgetbuf(rel, buf, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-22 01:45:05 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Return to the original page to see what's up */
|
2004-04-21 20:24:26 +02:00
|
|
|
buf = _bt_relandgetbuf(rel, buf, obknum, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-22 01:45:05 +01:00
|
|
|
if (P_ISDELETED(opaque))
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* It was deleted. Move right to first nondeleted page (there
|
|
|
|
* must be one); that is the page that has acquired the deleted
|
|
|
|
* one's keyspace, so stepping left from it will take us where we
|
|
|
|
* want to be.
|
|
|
|
*/
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
if (P_RIGHTMOST(opaque))
|
2007-12-31 05:52:05 +01:00
|
|
|
elog(ERROR, "fell off the end of index \"%s\"",
|
2003-02-22 01:45:05 +01:00
|
|
|
RelationGetRelationName(rel));
|
|
|
|
blkno = opaque->btpo_next;
|
2004-04-21 20:24:26 +02:00
|
|
|
buf = _bt_relandgetbuf(rel, buf, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-22 01:45:05 +01:00
|
|
|
if (!P_ISDELETED(opaque))
|
|
|
|
break;
|
|
|
|
}
|
2003-08-04 02:43:34 +02:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/*
|
|
|
|
* Now return to top of loop, resetting obknum to point to this
|
|
|
|
* nondeleted page, and try again.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* It wasn't deleted; the explanation had better be that the page
|
|
|
|
* to the left got split or deleted. Without this check, we'd go
|
|
|
|
* into an infinite loop if there's anything wrong.
|
|
|
|
*/
|
|
|
|
if (opaque->btpo_prev == lblkno)
|
2007-12-31 05:52:05 +01:00
|
|
|
elog(ERROR, "could not find left sibling of block %u in index \"%s\"",
|
|
|
|
obknum, RelationGetRelationName(rel));
|
2003-02-22 01:45:05 +01:00
|
|
|
/* Okay to try again with new lblkno value */
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return InvalidBuffer;
|
|
|
|
}
|
|
|
|
|
2003-02-21 01:06:22 +01:00
|
|
|
/*
|
|
|
|
* _bt_get_endpoint() -- Find the first or last page on a given tree level
|
|
|
|
*
|
|
|
|
* If the index is empty, we will return InvalidBuffer; any other failure
|
2003-07-21 22:29:40 +02:00
|
|
|
* condition causes ereport(). We will not return a dead page.
|
2003-02-21 01:06:22 +01:00
|
|
|
*
|
|
|
|
* The returned buffer is pinned and read-locked.
|
|
|
|
*/
|
|
|
|
Buffer
|
2023-09-08 07:12:12 +02:00
|
|
|
_bt_get_endpoint(Relation rel, uint32 level, bool rightmost)
|
2003-02-21 01:06:22 +01:00
|
|
|
{
|
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
|
|
|
OffsetNumber offnum;
|
|
|
|
BlockNumber blkno;
|
|
|
|
IndexTuple itup;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we are looking for a leaf page, okay to descend from fast root;
|
|
|
|
* otherwise better descend from true root. (There is no point in being
|
|
|
|
* smarter about intermediate levels.)
|
|
|
|
*/
|
|
|
|
if (level == 0)
|
2023-06-10 23:08:25 +02:00
|
|
|
buf = _bt_getroot(rel, NULL, BT_READ);
|
2003-02-21 01:06:22 +01:00
|
|
|
else
|
2023-06-10 23:08:25 +02:00
|
|
|
buf = _bt_gettrueroot(rel);
|
2003-02-21 01:06:22 +01:00
|
|
|
|
|
|
|
if (!BufferIsValid(buf))
|
|
|
|
return InvalidBuffer;
|
|
|
|
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-21 01:06:22 +01:00
|
|
|
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If we landed on a deleted page, step right to find a live page
|
|
|
|
* (there must be one). Also, if we want the rightmost page, step
|
|
|
|
* right if needed to get to it (this could happen if the page split
|
|
|
|
* since we obtained a pointer to it).
|
|
|
|
*/
|
2003-02-22 01:45:05 +01:00
|
|
|
while (P_IGNORE(opaque) ||
|
2003-02-21 01:06:22 +01:00
|
|
|
(rightmost && !P_RIGHTMOST(opaque)))
|
|
|
|
{
|
|
|
|
blkno = opaque->btpo_next;
|
|
|
|
if (blkno == P_NONE)
|
2007-12-31 05:52:05 +01:00
|
|
|
elog(ERROR, "fell off the end of index \"%s\"",
|
2003-02-22 01:45:05 +01:00
|
|
|
RelationGetRelationName(rel));
|
2004-04-21 20:24:26 +02:00
|
|
|
buf = _bt_relandgetbuf(rel, buf, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-21 01:06:22 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Done? */
|
2021-02-25 03:41:34 +01:00
|
|
|
if (opaque->btpo_level == level)
|
2003-02-21 01:06:22 +01:00
|
|
|
break;
|
2021-02-25 03:41:34 +01:00
|
|
|
if (opaque->btpo_level < level)
|
2019-08-01 11:05:08 +02:00
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INDEX_CORRUPTED),
|
|
|
|
errmsg_internal("btree level %u not found in index \"%s\"",
|
|
|
|
level, RelationGetRelationName(rel))));
|
2003-02-21 01:06:22 +01:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/* Descend to leftmost or rightmost child page */
|
2003-02-21 01:06:22 +01:00
|
|
|
if (rightmost)
|
|
|
|
offnum = PageGetMaxOffsetNumber(page);
|
|
|
|
else
|
|
|
|
offnum = P_FIRSTDATAKEY(opaque);
|
|
|
|
|
2006-01-26 00:04:21 +01:00
|
|
|
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
|
2019-12-17 02:49:45 +01:00
|
|
|
blkno = BTreeTupleGetDownLink(itup);
|
2003-02-21 01:06:22 +01:00
|
|
|
|
2004-04-21 20:24:26 +02:00
|
|
|
buf = _bt_relandgetbuf(rel, buf, blkno, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-21 01:06:22 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2006-05-07 03:21:30 +02:00
|
|
|
* _bt_endpoint() -- Find the first or last page in the index, and scan
|
2003-11-12 22:15:59 +01:00
|
|
|
* from there to the first key satisfying all the quals.
|
2000-07-21 08:42:39 +02:00
|
|
|
*
|
|
|
|
* This is used by _bt_first() to set up a scan when we've determined
|
|
|
|
* that the scan must start at the beginning or end of the index (for
|
2006-05-07 03:21:30 +02:00
|
|
|
* a forward or backward scan respectively). Exit conditions are the
|
|
|
|
* same as for _bt_first().
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2002-05-21 01:51:44 +02:00
|
|
|
static bool
|
1996-07-09 08:22:35 +02:00
|
|
|
_bt_endpoint(IndexScanDesc scan, ScanDirection dir)
|
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
Relation rel = scan->indexRelation;
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
1996-07-09 08:22:35 +02:00
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
|
|
|
BTPageOpaque opaque;
|
2000-07-21 08:42:39 +02:00
|
|
|
OffsetNumber start;
|
2011-10-09 06:21:08 +02:00
|
|
|
BTScanPosItem *currItem;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/*
|
|
|
|
* Scan down to the leftmost or rightmost leaf page. This is a simplified
|
|
|
|
* version of _bt_search(). We don't maintain a stack since we know we
|
|
|
|
* won't need it.
|
|
|
|
*/
|
2023-09-08 07:12:12 +02:00
|
|
|
buf = _bt_get_endpoint(rel, 0, ScanDirectionIsBackward(dir));
|
2000-07-21 08:42:39 +02:00
|
|
|
|
|
|
|
if (!BufferIsValid(buf))
|
|
|
|
{
|
2011-06-15 10:43:05 +02:00
|
|
|
/*
|
|
|
|
* Empty index. Lock the whole relation, as nothing finer to lock
|
|
|
|
* exists.
|
|
|
|
*/
|
|
|
|
PredicateLockRelation(rel, scan->xs_snapshot);
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosInvalidate(so->currPos);
|
2002-05-21 01:51:44 +02:00
|
|
|
return false;
|
2000-07-21 08:42:39 +02:00
|
|
|
}
|
|
|
|
|
2011-06-15 10:43:05 +02:00
|
|
|
PredicateLockPage(rel, BufferGetBlockNumber(buf), scan->xs_snapshot);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2022-04-01 06:24:50 +02:00
|
|
|
opaque = BTPageGetOpaque(page);
|
2003-02-21 01:06:22 +01:00
|
|
|
Assert(P_ISLEAF(opaque));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
if (ScanDirectionIsForward(dir))
|
|
|
|
{
|
2003-02-21 01:06:22 +01:00
|
|
|
/* There could be dead pages to the left, so not this: */
|
|
|
|
/* Assert(P_LEFTMOST(opaque)); */
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
start = P_FIRSTDATAKEY(opaque);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
else if (ScanDirectionIsBackward(dir))
|
1996-12-06 10:41:45 +01:00
|
|
|
{
|
2000-07-21 08:42:39 +02:00
|
|
|
Assert(P_RIGHTMOST(opaque));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
start = PageGetMaxOffsetNumber(page);
|
1996-12-06 10:41:45 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
else
|
2000-07-21 08:42:39 +02:00
|
|
|
{
|
2003-07-21 22:29:40 +02:00
|
|
|
elog(ERROR, "invalid scan direction: %d", (int) dir);
|
2000-07-21 08:42:39 +02:00
|
|
|
start = 0; /* keep compiler quiet */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* remember which buffer we have pinned */
|
2006-05-07 03:21:30 +02:00
|
|
|
so->currPos.buf = buf;
|
2000-07-21 08:42:39 +02:00
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
_bt_initialize_more_data(so, dir);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
/*
|
2006-05-07 03:21:30 +02:00
|
|
|
* Now load data from the first page of the scan.
|
2003-11-12 22:15:59 +01:00
|
|
|
*/
|
2024-03-22 14:25:53 +01:00
|
|
|
if (!_bt_readpage(scan, dir, start, true))
|
1996-12-06 10:41:45 +01:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/*
|
|
|
|
* There's no actually-matching data on this page. Try to advance to
|
|
|
|
* the next page. Return false if there's no matching data at all.
|
|
|
|
*/
|
2020-07-22 00:50:58 +02:00
|
|
|
_bt_unlockbuf(scan->indexRelation, so->currPos.buf);
|
2006-05-07 03:21:30 +02:00
|
|
|
if (!_bt_steppage(scan, dir))
|
|
|
|
return false;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2015-03-25 20:24:43 +01:00
|
|
|
else
|
|
|
|
{
|
Optimize nbtree backward scan boundary cases.
Teach _bt_binsrch (and related helper routines like _bt_search and
_bt_compare) about the initial positioning requirements of backward
scans. Routines like _bt_binsrch already know all about "nextkey"
searches, so it seems natural to teach them about "goback"/backward
searches, too. These concepts are closely related, and are much easier
to understand when discussed together.
Now that certain implementation details are hidden from _bt_first, it's
straightforward to add a new optimization: backward scans using the <
strategy now avoid extra leaf page accesses in certain "boundary cases".
Consider the following example, which uses the tenk1 table (and its
tenk1_hundred index) from the standard regression tests:
SELECT * FROM tenk1 WHERE hundred < 12 ORDER BY hundred DESC LIMIT 1;
Before this commit, nbtree would scan two leaf pages, even though it was
only really necessary to scan one leaf page. We'll now descend straight
to the leaf page containing a (12, -inf) high key instead. The scan
will locate matching non-pivot tuples with "hundred" values starting
from the value 11. The scan won't waste a page access on the right
sibling leaf page, which cannot possibly contain any matching tuples.
You can think of the optimization added by this commit as disabling an
optimization (the _bt_compare "!pivotsearch" behavior that was added to
Postgres 12 in commit dd299df8) for a small subset of cases where it was
always counterproductive.
Equivalently, you can think of the new optimization as extending the
"pivotsearch" behavior that page deletion by VACUUM has long required
(since the aforementioned Postgres 12 commit went in) to other, similar
cases. Obviously, this isn't strictly necessary for these new cases
(unlike VACUUM, _bt_first is prepared to move the scan to the left once
on the leaf level), but the underlying principle is the same.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wz=XPzM8HzaLPq278Vms420mVSHfgs9wi5tjFKHcapZCEw@mail.gmail.com
2023-12-08 20:05:17 +01:00
|
|
|
/* We have at least one item to return as scan's first item */
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_drop_lock_and_maybe_pin(scan, &so->currPos);
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
/* OK, itemIndex says what to return */
|
2011-10-09 06:21:08 +02:00
|
|
|
currItem = &so->currPos.items[so->currPos.itemIndex];
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
scan->xs_heaptid = currItem->heapTid;
|
2011-10-09 06:21:08 +02:00
|
|
|
if (scan->xs_want_itup)
|
|
|
|
scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
|
2006-05-07 03:21:30 +02:00
|
|
|
|
|
|
|
return true;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2017-02-15 13:41:14 +01:00
|
|
|
|
|
|
|
/*
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
* _bt_initialize_more_data() -- initialize moreLeft, moreRight and scan dir
|
|
|
|
* from currPos
|
2017-02-15 13:41:14 +01:00
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
_bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
|
|
|
|
{
|
Enhance nbtree ScalarArrayOp execution.
Commit 9e8da0f7 taught nbtree to handle ScalarArrayOpExpr quals
natively. This works by pushing down the full context (the array keys)
to the nbtree index AM, enabling it to execute multiple primitive index
scans that the planner treats as one continuous index scan/index path.
This earlier enhancement enabled nbtree ScalarArrayOp index-only scans.
It also allowed scans with ScalarArrayOp quals to return ordered results
(with some notable restrictions, described further down).
Take this general approach a lot further: teach nbtree SAOP index scans
to decide how to execute ScalarArrayOp scans (when and where to start
the next primitive index scan) based on physical index characteristics.
This can be far more efficient. All SAOP scans will now reliably avoid
duplicative leaf page accesses (just like any other nbtree index scan).
SAOP scans whose array keys are naturally clustered together now require
far fewer index descents, since we'll reliably avoid starting a new
primitive scan just to get to a later offset from the same leaf page.
The scan's arrays now advance using binary searches for the array
element that best matches the next tuple's attribute value. Required
scan key arrays (i.e. arrays from scan keys that can terminate the scan)
ratchet forward in lockstep with the index scan. Non-required arrays
(i.e. arrays from scan keys that can only exclude non-matching tuples)
"advance" without the process ever rolling over to a higher-order array.
Naturally, only required SAOP scan keys trigger skipping over leaf pages
(non-required arrays cannot safely end or start primitive index scans).
Consequently, even index scans of a composite index with a high-order
inequality scan key (which we'll mark required) and a low-order SAOP
scan key (which we won't mark required) now avoid repeating leaf page
accesses -- that benefit isn't limited to simpler equality-only cases.
In general, all nbtree index scans now output tuples as if they were one
continuous index scan -- even scans that mix a high-order inequality
with lower-order SAOP equalities reliably output tuples in index order.
This allows us to remove a couple of special cases that were applied
when building index paths with SAOP clauses during planning.
Bugfix commit 807a40c5 taught the planner to avoid generating unsafe
path keys: path keys on a multicolumn index path, with a SAOP clause on
any attribute beyond the first/most significant attribute. These cases
are now all safe, so we go back to generating path keys without regard
for the presence of SAOP clauses (just like with any other clause type).
Affected queries can now exploit scan output order in all the usual ways
(e.g., certain "ORDER BY ... LIMIT n" queries can now terminate early).
Also undo changes from follow-up bugfix commit a4523c5a, which taught
the planner to produce alternative index paths, with path keys, but
without low-order SAOP index quals (filter quals were used instead).
We'll no longer generate these alternative paths, since they can no
longer offer any meaningful advantages over standard index qual paths.
Affected queries thereby avoid all of the disadvantages that come from
using filter quals within index scan nodes. They can avoid extra heap
page accesses from using filter quals to exclude non-matching tuples
(index quals will never have that problem). They can also skip over
irrelevant sections of the index in more cases (though only when nbtree
determines that starting another primitive scan actually makes sense).
There is a theoretical risk that removing restrictions on SAOP index
paths from the planner will break compatibility with amcanorder-based
index AMs maintained as extensions. Such an index AM could have the
same limitations around ordered SAOP scans as nbtree had up until now.
Adding a pro forma incompatibility item about the issue to the Postgres
17 release notes seems like a good idea.
Author: Peter Geoghegan <pg@bowt.ie>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
Discussion: https://postgr.es/m/CAH2-Wz=ksvN_sjcnD1+Bt-WtifRA5ok48aDYnq3pkKhxgMQpcw@mail.gmail.com
2024-04-06 17:47:10 +02:00
|
|
|
so->currPos.dir = dir;
|
|
|
|
if (so->needPrimScan)
|
|
|
|
{
|
|
|
|
Assert(so->numArrayKeys);
|
|
|
|
|
|
|
|
so->currPos.moreLeft = true;
|
|
|
|
so->currPos.moreRight = true;
|
|
|
|
so->needPrimScan = false;
|
|
|
|
}
|
|
|
|
else if (ScanDirectionIsForward(dir))
|
2017-02-15 13:41:14 +01:00
|
|
|
{
|
|
|
|
so->currPos.moreLeft = false;
|
|
|
|
so->currPos.moreRight = true;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
so->currPos.moreLeft = true;
|
|
|
|
so->currPos.moreRight = false;
|
|
|
|
}
|
|
|
|
so->numKilled = 0; /* just paranoia */
|
|
|
|
so->markItemIndex = -1; /* ditto */
|
|
|
|
}
|