1996-07-09 08:22:35 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
1999-10-18 00:15:09 +02:00
|
|
|
* nbtree.c
|
1997-09-07 07:04:48 +02:00
|
|
|
* Implementation of Lehman and Yao's btree management algorithm for
|
|
|
|
* Postgres.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
1999-10-18 00:15:09 +02:00
|
|
|
* NOTES
|
|
|
|
* This file contains only the public interface routines.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
*
|
2018-01-03 05:30:12 +01:00
|
|
|
* Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
|
2000-01-26 06:58:53 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
1999-10-18 00:15:09 +02:00
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/backend/access/nbtree/nbtree.c
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
1999-07-16 01:04:24 +02:00
|
|
|
#include "postgres.h"
|
1996-11-05 11:35:38 +01:00
|
|
|
|
1999-07-16 07:00:38 +02:00
|
|
|
#include "access/nbtree.h"
|
2008-06-19 02:46:06 +02:00
|
|
|
#include "access/relscan.h"
|
2014-11-06 12:52:08 +01:00
|
|
|
#include "access/xlog.h"
|
2004-02-10 04:42:45 +01:00
|
|
|
#include "commands/vacuum.h"
|
Support parallel btree index builds.
To make this work, tuplesort.c and logtape.c must also support
parallelism, so this patch adds that infrastructure and then applies
it to the particular case of parallel btree index builds. Testing
to date shows that this can often be 2-3x faster than a serial
index build.
The model for deciding how many workers to use is fairly primitive
at present, but it's better than not having the feature. We can
refine it as we get more experience.
Peter Geoghegan with some help from Rushabh Lathia. While Heikki
Linnakangas is not an author of this patch, he wrote other patches
without which this feature would not have been possible, and
therefore the release notes should possibly credit him as an author
of this feature. Reviewed by Claudio Freire, Heikki Linnakangas,
Thomas Munro, Tels, Amit Kapila, me.
Discussion: http://postgr.es/m/CAM3SWZQKM=Pzc=CAHzRixKjp2eO5Q0Jg1SoFQqeXFQ647JiwqQ@mail.gmail.com
Discussion: http://postgr.es/m/CAH2-Wz=AxWqDoVvGU7dq856S4r6sJAj6DBn7VMtigkB33N5eyg@mail.gmail.com
2018-02-02 19:25:55 +01:00
|
|
|
#include "nodes/execnodes.h"
|
2017-02-15 13:41:14 +01:00
|
|
|
#include "pgstat.h"
|
|
|
|
#include "storage/condition_variable.h"
|
2008-09-30 12:52:14 +02:00
|
|
|
#include "storage/indexfsm.h"
|
2008-04-17 01:59:40 +02:00
|
|
|
#include "storage/ipc.h"
|
2006-05-11 01:18:39 +02:00
|
|
|
#include "storage/lmgr.h"
|
2010-12-29 12:48:53 +01:00
|
|
|
#include "storage/smgr.h"
|
2016-12-28 18:00:00 +01:00
|
|
|
#include "utils/builtins.h"
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
#include "utils/index_selfuncs.h"
|
2005-05-06 19:24:55 +02:00
|
|
|
#include "utils/memutils.h"
|
2000-07-21 08:42:39 +02:00
|
|
|
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/* Working state needed by btvacuumpage */
|
|
|
|
typedef struct
|
|
|
|
{
|
|
|
|
IndexVacuumInfo *info;
|
|
|
|
IndexBulkDeleteResult *stats;
|
|
|
|
IndexBulkDeleteCallback callback;
|
|
|
|
void *callback_state;
|
|
|
|
BTCycleId cycleid;
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
BlockNumber lastBlockVacuumed; /* highest blkno actually vacuumed */
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
BlockNumber lastBlockLocked; /* highest blkno we've cleanup-locked */
|
2006-10-04 02:30:14 +02:00
|
|
|
BlockNumber totFreePages; /* true total # of free pages */
|
2006-05-08 02:00:17 +02:00
|
|
|
MemoryContext pagedelcontext;
|
|
|
|
} BTVacState;
|
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
/*
|
|
|
|
* BTPARALLEL_NOT_INITIALIZED indicates that the scan has not started.
|
|
|
|
*
|
|
|
|
* BTPARALLEL_ADVANCING indicates that some process is advancing the scan to
|
|
|
|
* a new page; others must wait.
|
|
|
|
*
|
|
|
|
* BTPARALLEL_IDLE indicates that no backend is currently advancing the scan
|
|
|
|
* to a new page; some process can start doing that.
|
|
|
|
*
|
|
|
|
* BTPARALLEL_DONE indicates that the scan is complete (including error exit).
|
|
|
|
* We reach this state once for every distinct combination of array keys.
|
|
|
|
*/
|
|
|
|
typedef enum
|
|
|
|
{
|
|
|
|
BTPARALLEL_NOT_INITIALIZED,
|
|
|
|
BTPARALLEL_ADVANCING,
|
|
|
|
BTPARALLEL_IDLE,
|
|
|
|
BTPARALLEL_DONE
|
|
|
|
} BTPS_State;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* BTParallelScanDescData contains btree specific shared information required
|
|
|
|
* for parallel scan.
|
|
|
|
*/
|
|
|
|
typedef struct BTParallelScanDescData
|
|
|
|
{
|
|
|
|
BlockNumber btps_scanPage; /* latest or next page to be scanned */
|
2017-06-21 20:39:04 +02:00
|
|
|
BTPS_State btps_pageStatus; /* indicates whether next page is
|
|
|
|
* available for scan. see above for
|
|
|
|
* possible states of parallel scan. */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
int btps_arrayKeyCount; /* count indicating number of array scan
|
|
|
|
* keys processed by parallel scan */
|
2017-02-15 13:41:14 +01:00
|
|
|
slock_t btps_mutex; /* protects above variables */
|
|
|
|
ConditionVariable btps_cv; /* used to synchronize parallel scan */
|
2017-06-21 20:39:04 +02:00
|
|
|
} BTParallelScanDescData;
|
2017-02-15 13:41:14 +01:00
|
|
|
|
|
|
|
typedef struct BTParallelScanDescData *BTParallelScanDesc;
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
static void btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
|
|
|
|
IndexBulkDeleteCallback callback, void *callback_state,
|
|
|
|
BTCycleId cycleid);
|
|
|
|
static void btvacuumpage(BTVacState *vstate, BlockNumber blkno,
|
2006-10-04 02:30:14 +02:00
|
|
|
BlockNumber orig_blkno);
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
* Btree handler function: return IndexAmRoutine with access method parameters
|
|
|
|
* and callbacks.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2000-06-13 09:35:40 +02:00
|
|
|
Datum
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
bthandler(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
IndexAmRoutine *amroutine = makeNode(IndexAmRoutine);
|
|
|
|
|
2016-04-28 15:39:25 +02:00
|
|
|
amroutine->amstrategies = BTMaxStrategyNumber;
|
|
|
|
amroutine->amsupport = BTNProcs;
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
amroutine->amcanorder = true;
|
|
|
|
amroutine->amcanorderbyop = false;
|
|
|
|
amroutine->amcanbackward = true;
|
|
|
|
amroutine->amcanunique = true;
|
|
|
|
amroutine->amcanmulticol = true;
|
|
|
|
amroutine->amoptionalkey = true;
|
|
|
|
amroutine->amsearcharray = true;
|
|
|
|
amroutine->amsearchnulls = true;
|
|
|
|
amroutine->amstorage = false;
|
|
|
|
amroutine->amclusterable = true;
|
|
|
|
amroutine->ampredlocks = true;
|
2017-02-15 19:53:24 +01:00
|
|
|
amroutine->amcanparallel = true;
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
amroutine->amkeytype = InvalidOid;
|
|
|
|
|
|
|
|
amroutine->ambuild = btbuild;
|
|
|
|
amroutine->ambuildempty = btbuildempty;
|
|
|
|
amroutine->aminsert = btinsert;
|
|
|
|
amroutine->ambulkdelete = btbulkdelete;
|
|
|
|
amroutine->amvacuumcleanup = btvacuumcleanup;
|
|
|
|
amroutine->amcanreturn = btcanreturn;
|
|
|
|
amroutine->amcostestimate = btcostestimate;
|
|
|
|
amroutine->amoptions = btoptions;
|
2016-08-14 00:31:14 +02:00
|
|
|
amroutine->amproperty = btproperty;
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
amroutine->amvalidate = btvalidate;
|
|
|
|
amroutine->ambeginscan = btbeginscan;
|
|
|
|
amroutine->amrescan = btrescan;
|
|
|
|
amroutine->amgettuple = btgettuple;
|
|
|
|
amroutine->amgetbitmap = btgetbitmap;
|
|
|
|
amroutine->amendscan = btendscan;
|
|
|
|
amroutine->ammarkpos = btmarkpos;
|
|
|
|
amroutine->amrestrpos = btrestrpos;
|
2017-02-15 13:41:14 +01:00
|
|
|
amroutine->amestimateparallelscan = btestimateparallelscan;
|
|
|
|
amroutine->aminitparallelscan = btinitparallelscan;
|
|
|
|
amroutine->amparallelrescan = btparallelrescan;
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
|
|
|
|
PG_RETURN_POINTER(amroutine);
|
|
|
|
}
|
|
|
|
|
2010-12-29 12:48:53 +01:00
|
|
|
/*
|
|
|
|
* btbuildempty() -- build an empty btree index in the initialization fork
|
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
void
|
|
|
|
btbuildempty(Relation index)
|
2010-12-29 12:48:53 +01:00
|
|
|
{
|
|
|
|
Page metapage;
|
|
|
|
|
|
|
|
/* Construct metapage. */
|
|
|
|
metapage = (Page) palloc(BLCKSZ);
|
|
|
|
_bt_initmetapage(metapage, P_NONE, 0);
|
|
|
|
|
2016-12-08 20:09:09 +01:00
|
|
|
/*
|
2017-05-17 22:31:56 +02:00
|
|
|
* Write the page and log it. It might seem that an immediate sync would
|
|
|
|
* be sufficient to guarantee that the file exists on disk, but recovery
|
|
|
|
* itself might remove it while replaying, for example, an
|
|
|
|
* XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
|
|
|
|
* this even when wal_level=minimal.
|
2016-12-08 20:09:09 +01:00
|
|
|
*/
|
2013-03-22 14:54:07 +01:00
|
|
|
PageSetChecksumInplace(metapage, BTREE_METAPAGE);
|
2010-12-29 12:48:53 +01:00
|
|
|
smgrwrite(index->rd_smgr, INIT_FORKNUM, BTREE_METAPAGE,
|
|
|
|
(char *) metapage, true);
|
2016-12-08 20:09:09 +01:00
|
|
|
log_newpage(&index->rd_smgr->smgr_rnode.node, INIT_FORKNUM,
|
2017-11-03 21:31:32 +01:00
|
|
|
BTREE_METAPAGE, metapage, true);
|
2010-12-29 12:48:53 +01:00
|
|
|
|
|
|
|
/*
|
2014-04-23 11:56:41 +02:00
|
|
|
* An immediate sync is required even if we xlog'd the page, because the
|
2010-12-29 12:48:53 +01:00
|
|
|
* write did not go through shared_buffers and therefore a concurrent
|
2014-04-23 11:56:41 +02:00
|
|
|
* checkpoint may have moved the redo pointer past our xlog record.
|
2010-12-29 12:48:53 +01:00
|
|
|
*/
|
|
|
|
smgrimmedsync(index->rd_smgr, INIT_FORKNUM);
|
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btinsert() -- insert an index tuple into a btree.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
1997-09-07 07:04:48 +02:00
|
|
|
* Descend the tree recursively, find the appropriate location for our
|
2005-03-21 02:24:04 +01:00
|
|
|
* new tuple, and put it there.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
bool
|
|
|
|
btinsert(Relation rel, Datum *values, bool *isnull,
|
|
|
|
ItemPointer ht_ctid, Relation heapRel,
|
Allow index AMs to cache data across aminsert calls within a SQL command.
It's always been possible for index AMs to cache data across successive
amgettuple calls within a single SQL command: the IndexScanDesc.opaque
field is meant for precisely that. However, no comparable facility
exists for amortizing setup work across successive aminsert calls.
This patch adds such a feature and teaches GIN, GIST, and BRIN to use it
to amortize catalog lookups they'd previously been doing on every call.
(The other standard index AMs keep everything they need in the relcache,
so there's little to improve there.)
For GIN, the overall improvement in a statement that inserts many rows
can be as much as 10%, though it seems a bit less for the other two.
In addition, this makes a really significant difference in runtime
for CLOBBER_CACHE_ALWAYS tests, since in those builds the repeated
catalog lookups are vastly more expensive.
The reason this has been hard up to now is that the aminsert function is
not passed any useful place to cache per-statement data. What I chose to
do is to add suitable fields to struct IndexInfo and pass that to aminsert.
That's not widening the index AM API very much because IndexInfo is already
within the ken of ambuild; in fact, by passing the same info to aminsert
as to ambuild, this is really removing an inconsistency in the AM API.
Discussion: https://postgr.es/m/27568.1486508680@sss.pgh.pa.us
2017-02-09 17:52:12 +01:00
|
|
|
IndexUniqueCheck checkUnique,
|
|
|
|
IndexInfo *indexInfo)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2009-07-29 22:56:21 +02:00
|
|
|
bool result;
|
1997-09-08 04:41:22 +02:00
|
|
|
IndexTuple itup;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
|
|
|
/* generate an index tuple */
|
2005-03-21 02:24:04 +01:00
|
|
|
itup = index_form_tuple(RelationGetDescr(rel), values, isnull);
|
1997-09-07 07:04:48 +02:00
|
|
|
itup->t_tid = *ht_ctid;
|
|
|
|
|
2009-07-29 22:56:21 +02:00
|
|
|
result = _bt_doinsert(rel, itup, checkUnique, heapRel);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
|
|
|
pfree(itup);
|
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return result;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btgettuple() -- Get the next tuple in the scan.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
bool
|
|
|
|
btgettuple(IndexScanDesc scan, ScanDirection dir)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-05-24 20:57:57 +02:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
bool res;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2008-04-13 21:18:14 +02:00
|
|
|
/* btree indexes are never lossy */
|
|
|
|
scan->xs_recheck = false;
|
|
|
|
|
1997-09-07 07:04:48 +02:00
|
|
|
/*
|
2011-10-16 21:39:24 +02:00
|
|
|
* If we have any array keys, initialize them during first call for a
|
|
|
|
* scan. We can't do this in btrescan because we don't know the scan
|
|
|
|
* direction at that time.
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
2011-10-16 21:39:24 +02:00
|
|
|
if (so->numArrayKeys && !BTScanPosIsValid(so->currPos))
|
|
|
|
{
|
|
|
|
/* punt if we have any unsatisfiable array keys */
|
|
|
|
if (so->numArrayKeys < 0)
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return false;
|
2011-10-16 21:39:24 +02:00
|
|
|
|
|
|
|
_bt_start_array_keys(scan, dir);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* This loop handles advancing to the next array elements, if any */
|
|
|
|
do
|
1998-07-30 07:05:05 +02:00
|
|
|
{
|
2002-05-24 20:57:57 +02:00
|
|
|
/*
|
2011-10-16 21:39:24 +02:00
|
|
|
* If we've already initialized this scan, we can just advance it in
|
|
|
|
* the appropriate direction. If we haven't done so yet, we call
|
|
|
|
* _bt_first() to get the first item in the scan.
|
2002-05-24 20:57:57 +02:00
|
|
|
*/
|
2011-10-16 21:39:24 +02:00
|
|
|
if (!BTScanPosIsValid(so->currPos))
|
|
|
|
res = _bt_first(scan, dir);
|
|
|
|
else
|
2002-05-24 20:57:57 +02:00
|
|
|
{
|
|
|
|
/*
|
2011-10-16 21:39:24 +02:00
|
|
|
* Check to see if we should kill the previously-fetched tuple.
|
2002-05-24 20:57:57 +02:00
|
|
|
*/
|
2011-10-16 21:39:24 +02:00
|
|
|
if (scan->kill_prior_tuple)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Yes, remember it for later. (We'll deal with all such
|
|
|
|
* tuples at once right before leaving the index page.) The
|
|
|
|
* test for numKilled overrun is not just paranoia: if the
|
|
|
|
* caller reverses direction in the indexscan then the same
|
|
|
|
* item might get entered multiple times. It's not worth
|
|
|
|
* trying to optimize that, so we don't detect it, but instead
|
|
|
|
* just forget any excess entries.
|
|
|
|
*/
|
|
|
|
if (so->killedItems == NULL)
|
|
|
|
so->killedItems = (int *)
|
|
|
|
palloc(MaxIndexTuplesPerPage * sizeof(int));
|
|
|
|
if (so->numKilled < MaxIndexTuplesPerPage)
|
|
|
|
so->killedItems[so->numKilled++] = so->currPos.itemIndex;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Now continue the scan.
|
|
|
|
*/
|
|
|
|
res = _bt_next(scan, dir);
|
2002-05-24 20:57:57 +02:00
|
|
|
}
|
2002-09-04 22:31:48 +02:00
|
|
|
|
2011-10-16 21:39:24 +02:00
|
|
|
/* If we have a tuple, return it ... */
|
|
|
|
if (res)
|
|
|
|
break;
|
|
|
|
/* ... otherwise see if we have more array keys to deal with */
|
|
|
|
} while (so->numArrayKeys && _bt_advance_array_keys(scan, dir));
|
1998-09-01 06:40:42 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return res;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2005-03-28 01:53:05 +02:00
|
|
|
/*
|
2008-04-11 00:25:26 +02:00
|
|
|
* btgetbitmap() -- gets all matching tuples, and adds them to a bitmap
|
2005-03-28 01:53:05 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
int64
|
|
|
|
btgetbitmap(IndexScanDesc scan, TIDBitmap *tbm)
|
2005-03-28 01:53:05 +02:00
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
2008-04-11 00:25:26 +02:00
|
|
|
int64 ntids = 0;
|
|
|
|
ItemPointer heapTid;
|
2005-03-28 01:53:05 +02:00
|
|
|
|
2011-10-16 21:39:24 +02:00
|
|
|
/*
|
|
|
|
* If we have any array keys, initialize them.
|
|
|
|
*/
|
|
|
|
if (so->numArrayKeys)
|
2005-03-28 01:53:05 +02:00
|
|
|
{
|
2011-10-16 21:39:24 +02:00
|
|
|
/* punt if we have any unsatisfiable array keys */
|
|
|
|
if (so->numArrayKeys < 0)
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return ntids;
|
2011-10-16 21:39:24 +02:00
|
|
|
|
|
|
|
_bt_start_array_keys(scan, ForwardScanDirection);
|
2005-03-28 01:53:05 +02:00
|
|
|
}
|
|
|
|
|
2011-10-16 21:39:24 +02:00
|
|
|
/* This loop handles advancing to the next array elements, if any */
|
|
|
|
do
|
2005-03-28 01:53:05 +02:00
|
|
|
{
|
2011-10-16 21:39:24 +02:00
|
|
|
/* Fetch the first page & tuple */
|
|
|
|
if (_bt_first(scan, ForwardScanDirection))
|
2006-05-07 03:21:30 +02:00
|
|
|
{
|
2011-10-16 21:39:24 +02:00
|
|
|
/* Save tuple ID, and continue scanning */
|
|
|
|
heapTid = &scan->xs_ctup.t_self;
|
|
|
|
tbm_add_tuples(tbm, heapTid, 1, false);
|
|
|
|
ntids++;
|
|
|
|
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Advance to next tuple within page. This is the same as the
|
|
|
|
* easy case in _bt_next().
|
|
|
|
*/
|
|
|
|
if (++so->currPos.itemIndex > so->currPos.lastItem)
|
|
|
|
{
|
|
|
|
/* let _bt_next do the heavy lifting */
|
|
|
|
if (!_bt_next(scan, ForwardScanDirection))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Save tuple ID, and continue scanning */
|
|
|
|
heapTid = &so->currPos.items[so->currPos.itemIndex].heapTid;
|
|
|
|
tbm_add_tuples(tbm, heapTid, 1, false);
|
|
|
|
ntids++;
|
|
|
|
}
|
2006-05-07 03:21:30 +02:00
|
|
|
}
|
2011-10-16 21:39:24 +02:00
|
|
|
/* Now see if we have more array keys to deal with */
|
|
|
|
} while (so->numArrayKeys && _bt_advance_array_keys(scan, ForwardScanDirection));
|
2005-03-28 01:53:05 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return ntids;
|
2005-03-28 01:53:05 +02:00
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btbeginscan() -- start a scan on a btree index
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
IndexScanDesc
|
|
|
|
btbeginscan(Relation rel, int nkeys, int norderbys)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
1997-09-08 04:41:22 +02:00
|
|
|
IndexScanDesc scan;
|
2010-12-03 02:50:48 +01:00
|
|
|
BTScanOpaque so;
|
|
|
|
|
|
|
|
/* no order by operators allowed */
|
|
|
|
Assert(norderbys == 0);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
|
|
|
/* get the scan */
|
2010-12-03 02:50:48 +01:00
|
|
|
scan = RelationGetIndexScan(rel, nkeys, norderbys);
|
|
|
|
|
|
|
|
/* allocate private workspace */
|
|
|
|
so = (BTScanOpaque) palloc(sizeof(BTScanOpaqueData));
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosInvalidate(so->currPos);
|
|
|
|
BTScanPosInvalidate(so->markPos);
|
2010-12-03 02:50:48 +01:00
|
|
|
if (scan->numberOfKeys > 0)
|
|
|
|
so->keyData = (ScanKey) palloc(scan->numberOfKeys * sizeof(ScanKeyData));
|
|
|
|
else
|
|
|
|
so->keyData = NULL;
|
2011-10-16 21:39:24 +02:00
|
|
|
|
|
|
|
so->arrayKeyData = NULL; /* assume no array keys for now */
|
|
|
|
so->numArrayKeys = 0;
|
|
|
|
so->arrayKeys = NULL;
|
|
|
|
so->arrayContext = NULL;
|
|
|
|
|
2010-12-03 02:50:48 +01:00
|
|
|
so->killedItems = NULL; /* until needed */
|
|
|
|
so->numKilled = 0;
|
2011-10-09 06:21:08 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't know yet whether the scan will be index-only, so we do not
|
2014-05-06 18:12:18 +02:00
|
|
|
* allocate the tuple workspace arrays until btrescan. However, we set up
|
2011-10-17 01:15:04 +02:00
|
|
|
* scan->xs_itupdesc whether we'll need it or not, since that's so cheap.
|
2011-10-09 06:21:08 +02:00
|
|
|
*/
|
|
|
|
so->currTuples = so->markTuples = NULL;
|
|
|
|
|
2011-10-17 01:15:04 +02:00
|
|
|
scan->xs_itupdesc = RelationGetDescr(rel);
|
|
|
|
|
2010-12-03 02:50:48 +01:00
|
|
|
scan->opaque = so;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return scan;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btrescan() -- rescan an index relation
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
void
|
|
|
|
btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
|
|
|
|
ScanKey orderbys, int norderbys)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2010-12-03 02:50:48 +01:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
2000-07-21 08:42:39 +02:00
|
|
|
|
|
|
|
/* we aren't holding any read locks, but gotta drop the pins */
|
2006-05-07 03:21:30 +02:00
|
|
|
if (BTScanPosIsValid(so->currPos))
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Before leaving current page, deal with any killed items */
|
|
|
|
if (so->numKilled > 0)
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_killitems(scan);
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
2006-08-24 03:18:34 +02:00
|
|
|
so->markItemIndex = -1;
|
2017-02-15 13:41:14 +01:00
|
|
|
so->arrayKeyCount = 0;
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosUnpinIfPinned(so->markPos);
|
|
|
|
BTScanPosInvalidate(so->markPos);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2011-10-09 06:21:08 +02:00
|
|
|
/*
|
|
|
|
* Allocate tuple workspace arrays, if needed for an index-only scan and
|
2014-05-06 18:12:18 +02:00
|
|
|
* not already done in a previous rescan call. To save on palloc
|
2011-10-09 06:21:08 +02:00
|
|
|
* overhead, both workspaces are allocated as one palloc block; only this
|
|
|
|
* function and btendscan know that.
|
2011-10-12 00:40:53 +02:00
|
|
|
*
|
|
|
|
* NOTE: this data structure also makes it safe to return data from a
|
|
|
|
* "name" column, even though btree name_ops uses an underlying storage
|
|
|
|
* datatype of cstring. The risk there is that "name" is supposed to be
|
|
|
|
* padded to NAMEDATALEN, but the actual index tuple is probably shorter.
|
|
|
|
* However, since we only return data out of tuples sitting in the
|
|
|
|
* currTuples array, a fetch of NAMEDATALEN bytes can at worst pull some
|
|
|
|
* data out of the markTuples array --- running off the end of memory for
|
|
|
|
* a SIGSEGV is not possible. Yeah, this is ugly as sin, but it beats
|
|
|
|
* adding special-case treatment for name_ops elsewhere.
|
2011-10-09 06:21:08 +02:00
|
|
|
*/
|
|
|
|
if (scan->xs_want_itup && so->currTuples == NULL)
|
|
|
|
{
|
|
|
|
so->currTuples = (char *) palloc(BLCKSZ * 2);
|
|
|
|
so->markTuples = so->currTuples + BLCKSZ;
|
|
|
|
}
|
|
|
|
|
1997-09-07 07:04:48 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* Reset the scan keys. Note that keys ordering stuff moved to _bt_first.
|
|
|
|
* - vadim 05/05/97
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
2003-03-24 00:01:03 +01:00
|
|
|
if (scankey && scan->numberOfKeys > 0)
|
1997-09-07 07:04:48 +02:00
|
|
|
memmove(scan->keyData,
|
|
|
|
scankey,
|
|
|
|
scan->numberOfKeys * sizeof(ScanKeyData));
|
2003-11-12 22:15:59 +01:00
|
|
|
so->numberOfKeys = 0; /* until _bt_preprocess_keys sets it */
|
1996-07-30 09:56:04 +02:00
|
|
|
|
2011-10-16 21:39:24 +02:00
|
|
|
/* If any keys are SK_SEARCHARRAY type, set up array-key info */
|
|
|
|
_bt_preprocess_array_keys(scan);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btendscan() -- close down a scan
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
void
|
|
|
|
btendscan(IndexScanDesc scan)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-07-21 08:42:39 +02:00
|
|
|
/* we aren't holding any read locks, but gotta drop the pins */
|
2006-05-07 03:21:30 +02:00
|
|
|
if (BTScanPosIsValid(so->currPos))
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
/* Before leaving current page, deal with any killed items */
|
|
|
|
if (so->numKilled > 0)
|
2015-03-25 20:24:43 +01:00
|
|
|
_bt_killitems(scan);
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
2006-08-24 03:18:34 +02:00
|
|
|
so->markItemIndex = -1;
|
2015-03-25 20:24:43 +01:00
|
|
|
BTScanPosUnpinIfPinned(so->markPos);
|
|
|
|
|
|
|
|
/* No need to invalidate positions, the RAM is about to be freed. */
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2011-10-09 06:21:08 +02:00
|
|
|
/* Release storage */
|
2004-01-07 19:56:30 +01:00
|
|
|
if (so->keyData != NULL)
|
1997-09-07 07:04:48 +02:00
|
|
|
pfree(so->keyData);
|
2011-10-16 21:39:24 +02:00
|
|
|
/* so->arrayKeyData and so->arrayKeys are in arrayContext */
|
|
|
|
if (so->arrayContext != NULL)
|
|
|
|
MemoryContextDelete(so->arrayContext);
|
|
|
|
if (so->killedItems != NULL)
|
|
|
|
pfree(so->killedItems);
|
2011-10-09 06:21:08 +02:00
|
|
|
if (so->currTuples != NULL)
|
|
|
|
pfree(so->currTuples);
|
|
|
|
/* so->markTuples should not be pfree'd, see btrescan */
|
1997-09-07 07:04:48 +02:00
|
|
|
pfree(so);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btmarkpos() -- save current scan position
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
void
|
|
|
|
btmarkpos(IndexScanDesc scan)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2015-03-25 20:24:43 +01:00
|
|
|
/* There may be an old mark with a pin (but no lock). */
|
|
|
|
BTScanPosUnpinIfPinned(so->markPos);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-08-24 03:18:34 +02:00
|
|
|
/*
|
|
|
|
* Just record the current itemIndex. If we later step to next page
|
2006-10-04 02:30:14 +02:00
|
|
|
* before releasing the marked position, _bt_steppage makes a full copy of
|
|
|
|
* the currPos struct in markPos. If (as often happens) the mark is moved
|
|
|
|
* before we leave the page, we don't have to do that work.
|
2006-08-24 03:18:34 +02:00
|
|
|
*/
|
2006-05-07 03:21:30 +02:00
|
|
|
if (BTScanPosIsValid(so->currPos))
|
2006-08-24 03:18:34 +02:00
|
|
|
so->markItemIndex = so->currPos.itemIndex;
|
|
|
|
else
|
2015-03-25 20:24:43 +01:00
|
|
|
{
|
|
|
|
BTScanPosInvalidate(so->markPos);
|
2006-08-24 03:18:34 +02:00
|
|
|
so->markItemIndex = -1;
|
2015-03-25 20:24:43 +01:00
|
|
|
}
|
2000-06-13 09:35:40 +02:00
|
|
|
|
2012-09-27 22:59:59 +02:00
|
|
|
/* Also record the current positions of any array keys */
|
|
|
|
if (so->numArrayKeys)
|
|
|
|
_bt_mark_array_keys(scan);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1997-09-07 07:04:48 +02:00
|
|
|
* btrestrpos() -- restore scan to last saved position
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
void
|
|
|
|
btrestrpos(IndexScanDesc scan)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2006-05-07 03:21:30 +02:00
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2012-09-27 22:59:59 +02:00
|
|
|
/* Restore the marked positions of any array keys */
|
|
|
|
if (so->numArrayKeys)
|
|
|
|
_bt_restore_array_keys(scan);
|
|
|
|
|
2006-08-24 03:18:34 +02:00
|
|
|
if (so->markItemIndex >= 0)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2006-08-24 03:18:34 +02:00
|
|
|
/*
|
2015-03-25 20:24:43 +01:00
|
|
|
* The scan has never moved to a new page since the last mark. Just
|
2006-10-04 02:30:14 +02:00
|
|
|
* restore the itemIndex.
|
2015-03-25 20:24:43 +01:00
|
|
|
*
|
|
|
|
* NB: In this case we can't count on anything in so->markPos to be
|
|
|
|
* accurate.
|
2006-08-24 03:18:34 +02:00
|
|
|
*/
|
|
|
|
so->currPos.itemIndex = so->markItemIndex;
|
2006-10-04 02:30:14 +02:00
|
|
|
}
|
2006-08-24 03:18:34 +02:00
|
|
|
else
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2015-03-25 20:24:43 +01:00
|
|
|
/*
|
|
|
|
* The scan moved to a new page after last mark or restore, and we are
|
|
|
|
* now restoring to the marked page. We aren't holding any read
|
|
|
|
* locks, but if we're still holding the pin for the current position,
|
|
|
|
* we must drop it.
|
|
|
|
*/
|
2006-08-24 03:18:34 +02:00
|
|
|
if (BTScanPosIsValid(so->currPos))
|
|
|
|
{
|
|
|
|
/* Before leaving current page, deal with any killed items */
|
2015-03-25 20:24:43 +01:00
|
|
|
if (so->numKilled > 0)
|
|
|
|
_bt_killitems(scan);
|
|
|
|
BTScanPosUnpinIfPinned(so->currPos);
|
2006-08-24 03:18:34 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
if (BTScanPosIsValid(so->markPos))
|
|
|
|
{
|
|
|
|
/* bump pin on mark buffer for assignment to current buffer */
|
2015-03-25 20:24:43 +01:00
|
|
|
if (BTScanPosIsPinned(so->markPos))
|
|
|
|
IncrBufferRefCount(so->markPos.buf);
|
2006-08-24 03:18:34 +02:00
|
|
|
memcpy(&so->currPos, &so->markPos,
|
|
|
|
offsetof(BTScanPosData, items[1]) +
|
|
|
|
so->markPos.lastItem * sizeof(BTScanPosItem));
|
2011-10-09 06:21:08 +02:00
|
|
|
if (so->currTuples)
|
|
|
|
memcpy(so->currTuples, so->markTuples,
|
|
|
|
so->markPos.nextTupleOffset);
|
2006-08-24 03:18:34 +02:00
|
|
|
}
|
2015-03-25 20:24:43 +01:00
|
|
|
else
|
|
|
|
BTScanPosInvalidate(so->currPos);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2017-02-15 13:41:14 +01:00
|
|
|
/*
|
|
|
|
* btestimateparallelscan -- estimate storage for BTParallelScanDescData
|
|
|
|
*/
|
|
|
|
Size
|
|
|
|
btestimateparallelscan(void)
|
|
|
|
{
|
|
|
|
return sizeof(BTParallelScanDescData);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* btinitparallelscan -- initialize BTParallelScanDesc for parallel btree scan
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
btinitparallelscan(void *target)
|
|
|
|
{
|
|
|
|
BTParallelScanDesc bt_target = (BTParallelScanDesc) target;
|
|
|
|
|
|
|
|
SpinLockInit(&bt_target->btps_mutex);
|
|
|
|
bt_target->btps_scanPage = InvalidBlockNumber;
|
|
|
|
bt_target->btps_pageStatus = BTPARALLEL_NOT_INITIALIZED;
|
|
|
|
bt_target->btps_arrayKeyCount = 0;
|
|
|
|
ConditionVariableInit(&bt_target->btps_cv);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* btparallelrescan() -- reset parallel scan
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
btparallelrescan(IndexScanDesc scan)
|
|
|
|
{
|
|
|
|
BTParallelScanDesc btscan;
|
|
|
|
ParallelIndexScanDesc parallel_scan = scan->parallel_scan;
|
|
|
|
|
|
|
|
Assert(parallel_scan);
|
|
|
|
|
|
|
|
btscan = (BTParallelScanDesc) OffsetToPointer((void *) parallel_scan,
|
|
|
|
parallel_scan->ps_offset);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In theory, we don't need to acquire the spinlock here, because there
|
|
|
|
* shouldn't be any other workers running at this point, but we do so for
|
|
|
|
* consistency.
|
|
|
|
*/
|
|
|
|
SpinLockAcquire(&btscan->btps_mutex);
|
|
|
|
btscan->btps_scanPage = InvalidBlockNumber;
|
|
|
|
btscan->btps_pageStatus = BTPARALLEL_NOT_INITIALIZED;
|
|
|
|
btscan->btps_arrayKeyCount = 0;
|
|
|
|
SpinLockRelease(&btscan->btps_mutex);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_parallel_seize() -- Begin the process of advancing the scan to a new
|
|
|
|
* page. Other scans must wait until we call bt_parallel_release() or
|
|
|
|
* bt_parallel_done().
|
|
|
|
*
|
|
|
|
* The return value is true if we successfully seized the scan and false
|
|
|
|
* if we did not. The latter case occurs if no pages remain for the current
|
|
|
|
* set of scankeys.
|
|
|
|
*
|
|
|
|
* If the return value is true, *pageno returns the next or current page
|
|
|
|
* of the scan (depending on the scan direction). An invalid block number
|
|
|
|
* means the scan hasn't yet started, and P_NONE means we've reached the end.
|
|
|
|
* The first time a participating process reaches the last page, it will return
|
|
|
|
* true and set *pageno to P_NONE; after that, further attempts to seize the
|
|
|
|
* scan will return false.
|
|
|
|
*
|
|
|
|
* Callers should ignore the value of pageno if the return value is false.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
_bt_parallel_seize(IndexScanDesc scan, BlockNumber *pageno)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
BTPS_State pageStatus;
|
|
|
|
bool exit_loop = false;
|
|
|
|
bool status = true;
|
|
|
|
ParallelIndexScanDesc parallel_scan = scan->parallel_scan;
|
|
|
|
BTParallelScanDesc btscan;
|
|
|
|
|
|
|
|
*pageno = P_NONE;
|
|
|
|
|
|
|
|
btscan = (BTParallelScanDesc) OffsetToPointer((void *) parallel_scan,
|
|
|
|
parallel_scan->ps_offset);
|
|
|
|
|
|
|
|
while (1)
|
|
|
|
{
|
|
|
|
SpinLockAcquire(&btscan->btps_mutex);
|
|
|
|
pageStatus = btscan->btps_pageStatus;
|
|
|
|
|
|
|
|
if (so->arrayKeyCount < btscan->btps_arrayKeyCount)
|
|
|
|
{
|
|
|
|
/* Parallel scan has already advanced to a new set of scankeys. */
|
|
|
|
status = false;
|
|
|
|
}
|
|
|
|
else if (pageStatus == BTPARALLEL_DONE)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We're done with this set of scankeys. This may be the end, or
|
|
|
|
* there could be more sets to try.
|
|
|
|
*/
|
|
|
|
status = false;
|
|
|
|
}
|
|
|
|
else if (pageStatus != BTPARALLEL_ADVANCING)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We have successfully seized control of the scan for the purpose
|
|
|
|
* of advancing it to a new page!
|
|
|
|
*/
|
|
|
|
btscan->btps_pageStatus = BTPARALLEL_ADVANCING;
|
|
|
|
*pageno = btscan->btps_scanPage;
|
|
|
|
exit_loop = true;
|
|
|
|
}
|
|
|
|
SpinLockRelease(&btscan->btps_mutex);
|
|
|
|
if (exit_loop || !status)
|
|
|
|
break;
|
|
|
|
ConditionVariableSleep(&btscan->btps_cv, WAIT_EVENT_BTREE_PAGE);
|
|
|
|
}
|
|
|
|
ConditionVariableCancelSleep();
|
|
|
|
|
|
|
|
return status;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_parallel_release() -- Complete the process of advancing the scan to a
|
|
|
|
* new page. We now have the new value btps_scanPage; some other backend
|
|
|
|
* can now begin advancing the scan.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
_bt_parallel_release(IndexScanDesc scan, BlockNumber scan_page)
|
|
|
|
{
|
|
|
|
ParallelIndexScanDesc parallel_scan = scan->parallel_scan;
|
|
|
|
BTParallelScanDesc btscan;
|
|
|
|
|
|
|
|
btscan = (BTParallelScanDesc) OffsetToPointer((void *) parallel_scan,
|
|
|
|
parallel_scan->ps_offset);
|
|
|
|
|
|
|
|
SpinLockAcquire(&btscan->btps_mutex);
|
|
|
|
btscan->btps_scanPage = scan_page;
|
|
|
|
btscan->btps_pageStatus = BTPARALLEL_IDLE;
|
|
|
|
SpinLockRelease(&btscan->btps_mutex);
|
|
|
|
ConditionVariableSignal(&btscan->btps_cv);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_parallel_done() -- Mark the parallel scan as complete.
|
|
|
|
*
|
|
|
|
* When there are no pages left to scan, this function should be called to
|
|
|
|
* notify other workers. Otherwise, they might wait forever for the scan to
|
|
|
|
* advance to the next page.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
_bt_parallel_done(IndexScanDesc scan)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
ParallelIndexScanDesc parallel_scan = scan->parallel_scan;
|
|
|
|
BTParallelScanDesc btscan;
|
|
|
|
bool status_changed = false;
|
|
|
|
|
|
|
|
/* Do nothing, for non-parallel scans */
|
|
|
|
if (parallel_scan == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
btscan = (BTParallelScanDesc) OffsetToPointer((void *) parallel_scan,
|
|
|
|
parallel_scan->ps_offset);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Mark the parallel scan as done for this combination of scan keys,
|
|
|
|
* unless some other process already did so. See also
|
|
|
|
* _bt_advance_array_keys.
|
|
|
|
*/
|
|
|
|
SpinLockAcquire(&btscan->btps_mutex);
|
|
|
|
if (so->arrayKeyCount >= btscan->btps_arrayKeyCount &&
|
|
|
|
btscan->btps_pageStatus != BTPARALLEL_DONE)
|
|
|
|
{
|
|
|
|
btscan->btps_pageStatus = BTPARALLEL_DONE;
|
|
|
|
status_changed = true;
|
|
|
|
}
|
|
|
|
SpinLockRelease(&btscan->btps_mutex);
|
|
|
|
|
|
|
|
/* wake up all the workers associated with this parallel scan */
|
|
|
|
if (status_changed)
|
|
|
|
ConditionVariableBroadcast(&btscan->btps_cv);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* _bt_parallel_advance_array_keys() -- Advances the parallel scan for array
|
|
|
|
* keys.
|
|
|
|
*
|
|
|
|
* Updates the count of array keys processed for both local and parallel
|
|
|
|
* scans.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
_bt_parallel_advance_array_keys(IndexScanDesc scan)
|
|
|
|
{
|
|
|
|
BTScanOpaque so = (BTScanOpaque) scan->opaque;
|
|
|
|
ParallelIndexScanDesc parallel_scan = scan->parallel_scan;
|
|
|
|
BTParallelScanDesc btscan;
|
|
|
|
|
|
|
|
btscan = (BTParallelScanDesc) OffsetToPointer((void *) parallel_scan,
|
|
|
|
parallel_scan->ps_offset);
|
|
|
|
|
|
|
|
so->arrayKeyCount++;
|
|
|
|
SpinLockAcquire(&btscan->btps_mutex);
|
|
|
|
if (btscan->btps_pageStatus == BTPARALLEL_DONE)
|
|
|
|
{
|
|
|
|
btscan->btps_scanPage = InvalidBlockNumber;
|
|
|
|
btscan->btps_pageStatus = BTPARALLEL_NOT_INITIALIZED;
|
|
|
|
btscan->btps_arrayKeyCount++;
|
|
|
|
}
|
|
|
|
SpinLockRelease(&btscan->btps_mutex);
|
|
|
|
}
|
|
|
|
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
/*
|
|
|
|
* Bulk deletion of all index entries pointing to a set of heap tuples.
|
|
|
|
* The set of target tuples is specified via a callback routine that tells
|
|
|
|
* whether any given heap tuple (identified by ItemPointer) is being deleted.
|
|
|
|
*
|
|
|
|
* Result: a palloc'd struct containing statistical info for VACUUM displays.
|
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
IndexBulkDeleteResult *
|
|
|
|
btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
|
|
|
|
IndexBulkDeleteCallback callback, void *callback_state)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2006-05-03 00:25:10 +02:00
|
|
|
Relation rel = info->index;
|
2006-05-08 02:00:17 +02:00
|
|
|
BTCycleId cycleid;
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/* allocate stats if first time through, else re-use existing struct */
|
|
|
|
if (stats == NULL)
|
|
|
|
stats = (IndexBulkDeleteResult *) palloc0(sizeof(IndexBulkDeleteResult));
|
2006-02-12 00:31:34 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/* Establish the vacuum cycle ID to use for this scan */
|
2008-04-17 01:59:40 +02:00
|
|
|
/* The ENSURE stuff ensures we clean up shared memory on failure */
|
|
|
|
PG_ENSURE_ERROR_CLEANUP(_bt_end_vacuum_callback, PointerGetDatum(rel));
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
{
|
2006-05-08 02:00:17 +02:00
|
|
|
cycleid = _bt_start_vacuum(rel);
|
2003-08-04 02:43:34 +02:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
btvacuumscan(info, stats, callback, callback_state, cycleid);
|
|
|
|
}
|
2008-04-17 01:59:40 +02:00
|
|
|
PG_END_ENSURE_ERROR_CLEANUP(_bt_end_vacuum_callback, PointerGetDatum(rel));
|
|
|
|
_bt_end_vacuum(rel);
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return stats;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1998-07-30 07:05:05 +02:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/*
|
|
|
|
* Post-VACUUM cleanup.
|
|
|
|
*
|
|
|
|
* Result: a palloc'd struct containing statistical info for VACUUM displays.
|
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
IndexBulkDeleteResult *
|
|
|
|
btvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
|
2003-02-22 01:45:05 +01:00
|
|
|
{
|
2009-03-24 21:17:18 +01:00
|
|
|
/* No-op in ANALYZE ONLY mode */
|
|
|
|
if (info->analyze_only)
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return stats;
|
2009-03-24 21:17:18 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* If btbulkdelete was called, we need not do anything, just return the
|
|
|
|
* stats from the latest btbulkdelete call. If it wasn't called, we must
|
|
|
|
* still do a pass over the index, to recycle any newly-recyclable pages
|
|
|
|
* and to obtain index statistics.
|
2006-05-08 02:00:17 +02:00
|
|
|
*
|
|
|
|
* Since we aren't going to actually delete any leaf items, there's no
|
|
|
|
* need to go through all the vacuum-cycle-ID pushups.
|
|
|
|
*/
|
2006-05-03 00:25:10 +02:00
|
|
|
if (stats == NULL)
|
2006-05-08 02:00:17 +02:00
|
|
|
{
|
2006-05-03 00:25:10 +02:00
|
|
|
stats = (IndexBulkDeleteResult *) palloc0(sizeof(IndexBulkDeleteResult));
|
2006-05-08 02:00:17 +02:00
|
|
|
btvacuumscan(info, stats, NULL, NULL, 0);
|
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2008-10-06 10:04:11 +02:00
|
|
|
/* Finally, vacuum the FSM */
|
|
|
|
IndexFreeSpaceMapVacuum(info->index);
|
|
|
|
|
2005-05-07 23:32:24 +02:00
|
|
|
/*
|
2010-02-08 05:33:55 +01:00
|
|
|
* It's quite possible for us to be fooled by concurrent page splits into
|
|
|
|
* double-counting some index tuples, so disbelieve any total that exceeds
|
|
|
|
* the underlying heap's count ... if we know that accurately. Otherwise
|
|
|
|
* this might just make matters worse.
|
2005-05-07 23:32:24 +02:00
|
|
|
*/
|
2010-02-08 05:33:55 +01:00
|
|
|
if (!info->estimated_count)
|
2006-05-08 02:00:17 +02:00
|
|
|
{
|
|
|
|
if (stats->num_index_tuples > info->num_heap_tuples)
|
|
|
|
stats->num_index_tuples = info->num_heap_tuples;
|
|
|
|
}
|
2005-05-07 23:32:24 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return stats;
|
2006-05-08 02:00:17 +02:00
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/*
|
|
|
|
* btvacuumscan --- scan the index for VACUUMing purposes
|
|
|
|
*
|
|
|
|
* This combines the functions of looking for leaf tuples that are deletable
|
|
|
|
* according to the vacuum callback, looking for empty pages that can be
|
|
|
|
* deleted, and looking for old deleted pages that can be recycled. Both
|
|
|
|
* btbulkdelete and btvacuumcleanup invoke this (the latter only if no
|
|
|
|
* btbulkdelete call occurred).
|
|
|
|
*
|
|
|
|
* The caller is responsible for initially allocating/zeroing a stats struct
|
|
|
|
* and for obtaining a vacuum cycle ID if necessary.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
|
|
|
|
IndexBulkDeleteCallback callback, void *callback_state,
|
|
|
|
BTCycleId cycleid)
|
|
|
|
{
|
|
|
|
Relation rel = info->index;
|
|
|
|
BTVacState vstate;
|
|
|
|
BlockNumber num_pages;
|
|
|
|
BlockNumber blkno;
|
|
|
|
bool needLock;
|
2005-05-07 23:32:24 +02:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* Reset counts that will be incremented during the scan; needed in case
|
|
|
|
* of multiple scans during a single VACUUM command
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
2009-06-07 00:13:52 +02:00
|
|
|
stats->estimated_count = false;
|
2006-05-08 02:00:17 +02:00
|
|
|
stats->num_index_tuples = 0;
|
|
|
|
stats->pages_deleted = 0;
|
|
|
|
|
|
|
|
/* Set up info to pass down to btvacuumpage */
|
|
|
|
vstate.info = info;
|
|
|
|
vstate.stats = stats;
|
|
|
|
vstate.callback = callback;
|
|
|
|
vstate.callback_state = callback_state;
|
|
|
|
vstate.cycleid = cycleid;
|
2010-02-26 03:01:40 +01:00
|
|
|
vstate.lastBlockVacuumed = BTREE_METAPAGE; /* Initialise at first block */
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
vstate.lastBlockLocked = BTREE_METAPAGE;
|
2006-09-21 22:31:22 +02:00
|
|
|
vstate.totFreePages = 0;
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2003-02-23 07:17:13 +01:00
|
|
|
/* Create a temporary memory context to run _bt_pagedel in */
|
2006-05-08 02:00:17 +02:00
|
|
|
vstate.pagedelcontext = AllocSetContextCreate(CurrentMemoryContext,
|
|
|
|
"_bt_pagedel",
|
Add macros to make AllocSetContextCreate() calls simpler and safer.
I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls
had typos in the context-sizing parameters. While none of these led to
especially significant problems, they did create minor inefficiencies,
and it's now clear that expecting people to copy-and-paste those calls
accurately is not a great idea. Let's reduce the risk of future errors
by introducing single macros that encapsulate the common use-cases.
Three such macros are enough to cover all but two special-purpose contexts;
those two calls can be left as-is, I think.
While this patch doesn't in itself improve matters for third-party
extensions, it doesn't break anything for them either, and they can
gradually adopt the simplified notation over time.
In passing, change TopMemoryContext to use the default allocation
parameters. Formerly it could only be extended 8K at a time. That was
probably reasonable when this code was written; but nowadays we create
many more contexts than we did then, so that it's not unusual to have a
couple hundred K in TopMemoryContext, even without considering various
dubious code that sticks other things there. There seems no good reason
not to let it use growing blocks like most other contexts.
Back-patch to 9.6, mostly because that's still close enough to HEAD that
it's easy to do so, and keeping the branches in sync can be expected to
avoid some future back-patching pain. The bugs fixed by these changes
don't seem to be significant enough to justify fixing them further back.
Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
|
|
|
ALLOCSET_DEFAULT_SIZES);
|
2003-02-23 07:17:13 +01:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* The outer loop iterates over all index pages except the metapage, in
|
|
|
|
* physical order (we hope the kernel will cooperate in providing
|
2006-05-08 02:00:17 +02:00
|
|
|
* read-ahead for speed). It is critical that we visit all leaf pages,
|
|
|
|
* including ones added after we start the scan, else we might fail to
|
|
|
|
* delete some deletable tuples. Hence, we must repeatedly check the
|
|
|
|
* relation length. We must acquire the relation-extension lock while
|
|
|
|
* doing so to avoid a race condition: if someone else is extending the
|
|
|
|
* relation, there is a window where bufmgr/smgr have created a new
|
2006-10-04 02:30:14 +02:00
|
|
|
* all-zero page but it hasn't yet been write-locked by _bt_getbuf(). If
|
|
|
|
* we manage to scan such a page here, we'll improperly assume it can be
|
|
|
|
* recycled. Taking the lock synchronizes things enough to prevent a
|
2006-05-08 02:00:17 +02:00
|
|
|
* problem: either num_pages won't include the new page, or _bt_getbuf
|
|
|
|
* already has write lock on the buffer and it will be fully initialized
|
|
|
|
* before we can examine it. (See also vacuumlazy.c, which has the same
|
2006-10-04 02:30:14 +02:00
|
|
|
* issue.) Also, we need not worry if a page is added immediately after
|
2006-05-08 02:00:17 +02:00
|
|
|
* we look; the page splitting code already has write-lock on the left
|
2006-10-04 02:30:14 +02:00
|
|
|
* page before it adds a right page, so we must already have processed any
|
|
|
|
* tuples due to be moved into such a page.
|
2006-05-08 02:00:17 +02:00
|
|
|
*
|
|
|
|
* We can skip locking for new or temp relations, however, since no one
|
|
|
|
* else could be accessing them.
|
2003-02-22 01:45:05 +01:00
|
|
|
*/
|
2006-05-08 02:00:17 +02:00
|
|
|
needLock = !RELATION_IS_LOCAL(rel);
|
2006-02-14 18:20:01 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
blkno = BTREE_METAPAGE + 1;
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
/* Get the current relation length */
|
|
|
|
if (needLock)
|
|
|
|
LockRelationForExtension(rel, ExclusiveLock);
|
|
|
|
num_pages = RelationGetNumberOfBlocks(rel);
|
|
|
|
if (needLock)
|
|
|
|
UnlockRelationForExtension(rel, ExclusiveLock);
|
|
|
|
|
|
|
|
/* Quit if we've scanned the whole relation */
|
|
|
|
if (blkno >= num_pages)
|
|
|
|
break;
|
|
|
|
/* Iterate over pages, then loop back to recheck length */
|
|
|
|
for (; blkno < num_pages; blkno++)
|
2006-02-12 01:18:17 +01:00
|
|
|
{
|
2006-05-08 02:00:17 +02:00
|
|
|
btvacuumpage(&vstate, blkno, blkno);
|
2006-02-12 01:18:17 +01:00
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
}
|
|
|
|
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
/*
|
2016-01-09 11:10:08 +01:00
|
|
|
* Check to see if we need to issue one final WAL record for this index,
|
2016-06-10 00:02:36 +02:00
|
|
|
* which may be needed for correctness on a hot standby node when non-MVCC
|
|
|
|
* index scans could take place.
|
2016-01-09 11:10:08 +01:00
|
|
|
*
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
* If the WAL is replayed in hot standby, the replay process needs to get
|
|
|
|
* cleanup locks on all index leaf pages, just as we've been doing here.
|
|
|
|
* However, we won't issue any WAL records about pages that have no items
|
|
|
|
* to be deleted. For pages between pages we've vacuumed, the replay code
|
|
|
|
* will take locks under the direction of the lastBlockVacuumed fields in
|
|
|
|
* the XLOG_BTREE_VACUUM WAL records. To cover pages after the last one
|
|
|
|
* we vacuum, we need to issue a dummy XLOG_BTREE_VACUUM WAL record
|
|
|
|
* against the last leaf page in the index, if that one wasn't vacuumed.
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
*/
|
|
|
|
if (XLogStandbyInfoActive() &&
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
vstate.lastBlockVacuumed < vstate.lastBlockLocked)
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
{
|
|
|
|
Buffer buf;
|
|
|
|
|
|
|
|
/*
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
* The page should be valid, but we can't use _bt_getbuf() because we
|
|
|
|
* want to use a nondefault buffer access strategy. Since we aren't
|
|
|
|
* going to delete any items, getting cleanup lock again is probably
|
|
|
|
* overkill, but for consistency do that anyway.
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
*/
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
buf = ReadBufferExtended(rel, MAIN_FORKNUM, vstate.lastBlockLocked,
|
|
|
|
RBM_NORMAL, info->strategy);
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
LockBufferForCleanup(buf);
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
_bt_checkpage(rel, buf);
|
2010-03-28 11:27:02 +02:00
|
|
|
_bt_delitems_vacuum(rel, buf, NULL, 0, vstate.lastBlockVacuumed);
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
}
|
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
MemoryContextDelete(vstate.pagedelcontext);
|
2003-02-23 07:17:13 +01:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
/* update statistics */
|
|
|
|
stats->num_pages = num_pages;
|
2006-09-21 22:31:22 +02:00
|
|
|
stats->pages_free = vstate.totFreePages;
|
2006-05-08 02:00:17 +02:00
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/*
|
|
|
|
* btvacuumpage --- VACUUM one page
|
|
|
|
*
|
|
|
|
* This processes a single page for btvacuumscan(). In some cases we
|
|
|
|
* must go back and re-examine previously-scanned pages; this routine
|
|
|
|
* recurses when necessary to handle that case.
|
|
|
|
*
|
|
|
|
* blkno is the page to process. orig_blkno is the highest block number
|
|
|
|
* reached by the outer btvacuumscan loop (the same as blkno, unless we
|
|
|
|
* are recursing to re-examine a previous page).
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
btvacuumpage(BTVacState *vstate, BlockNumber blkno, BlockNumber orig_blkno)
|
|
|
|
{
|
|
|
|
IndexVacuumInfo *info = vstate->info;
|
|
|
|
IndexBulkDeleteResult *stats = vstate->stats;
|
|
|
|
IndexBulkDeleteCallback callback = vstate->callback;
|
|
|
|
void *callback_state = vstate->callback_state;
|
|
|
|
Relation rel = info->index;
|
|
|
|
bool delete_now;
|
2006-10-04 02:30:14 +02:00
|
|
|
BlockNumber recurse_to;
|
2006-05-08 02:00:17 +02:00
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
2015-07-27 11:24:27 +02:00
|
|
|
BTPageOpaque opaque = NULL;
|
2006-05-08 02:00:17 +02:00
|
|
|
|
|
|
|
restart:
|
|
|
|
delete_now = false;
|
|
|
|
recurse_to = P_NONE;
|
|
|
|
|
|
|
|
/* call vacuum_delay_point while not holding any buffer lock */
|
|
|
|
vacuum_delay_point();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We can't use _bt_getbuf() here because it always applies
|
|
|
|
* _bt_checkpage(), which will barf on an all-zero page. We want to
|
2007-05-30 22:12:03 +02:00
|
|
|
* recycle all-zero pages, not fail. Also, we want to use a nondefault
|
|
|
|
* buffer access strategy.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
Unite ReadBufferWithFork, ReadBufferWithStrategy, and ZeroOrReadBuffer
functions into one ReadBufferExtended function, that takes the strategy
and mode as argument. There's three modes, RBM_NORMAL which is the default
used by plain ReadBuffer(), RBM_ZERO, which replaces ZeroOrReadBuffer, and
a new mode RBM_ZERO_ON_ERROR, which allows callers to read corrupt pages
without throwing an error. The FSM needs the new mode to recover from
corrupt pages, which could happend if we crash after extending an FSM file,
and the new page is "torn".
Add fork number to some error messages in bufmgr.c, that still lacked it.
2008-10-31 16:05:00 +01:00
|
|
|
buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
|
|
|
|
info->strategy);
|
2006-05-08 02:00:17 +02:00
|
|
|
LockBuffer(buf, BT_READ);
|
2016-04-20 15:31:19 +02:00
|
|
|
page = BufferGetPage(buf);
|
2006-05-08 02:00:17 +02:00
|
|
|
if (!PageIsNew(page))
|
2015-07-27 11:24:27 +02:00
|
|
|
{
|
2006-05-08 02:00:17 +02:00
|
|
|
_bt_checkpage(rel, buf);
|
2015-07-27 11:24:27 +02:00
|
|
|
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
|
|
|
|
}
|
2006-05-08 02:00:17 +02:00
|
|
|
|
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* If we are recursing, the only case we want to do anything with is a
|
|
|
|
* live leaf page having the current vacuum cycle ID. Any other state
|
2008-09-30 12:52:14 +02:00
|
|
|
* implies we already saw the page (eg, deleted it as being empty).
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (blkno != orig_blkno)
|
|
|
|
{
|
|
|
|
if (_bt_page_recyclable(page) ||
|
2006-11-01 20:43:17 +01:00
|
|
|
P_IGNORE(opaque) ||
|
2006-05-08 02:00:17 +02:00
|
|
|
!P_ISLEAF(opaque) ||
|
|
|
|
opaque->btpo_cycleid != vstate->cycleid)
|
|
|
|
{
|
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Page is valid, see what to do with it */
|
|
|
|
if (_bt_page_recyclable(page))
|
|
|
|
{
|
|
|
|
/* Okay to recycle this page */
|
2008-09-30 12:52:14 +02:00
|
|
|
RecordFreeIndexPage(rel, blkno);
|
2006-09-21 22:31:22 +02:00
|
|
|
vstate->totFreePages++;
|
2006-05-08 02:00:17 +02:00
|
|
|
stats->pages_deleted++;
|
|
|
|
}
|
|
|
|
else if (P_ISDELETED(opaque))
|
|
|
|
{
|
|
|
|
/* Already deleted, but can't recycle yet */
|
|
|
|
stats->pages_deleted++;
|
|
|
|
}
|
2006-11-01 20:43:17 +01:00
|
|
|
else if (P_ISHALFDEAD(opaque))
|
2006-05-08 02:00:17 +02:00
|
|
|
{
|
|
|
|
/* Half-dead, try to delete */
|
|
|
|
delete_now = true;
|
|
|
|
}
|
|
|
|
else if (P_ISLEAF(opaque))
|
|
|
|
{
|
|
|
|
OffsetNumber deletable[MaxOffsetNumber];
|
|
|
|
int ndeletable;
|
|
|
|
OffsetNumber offnum,
|
2006-10-04 02:30:14 +02:00
|
|
|
minoff,
|
|
|
|
maxoff;
|
2006-05-08 02:00:17 +02:00
|
|
|
|
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* Trade in the initial read lock for a super-exclusive write lock on
|
|
|
|
* this page. We must get such a lock on every leaf page over the
|
|
|
|
* course of the vacuum scan, whether or not it actually contains any
|
|
|
|
* deletable tuples --- see nbtree/README.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
|
|
|
|
LockBufferForCleanup(buf);
|
|
|
|
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
/*
|
|
|
|
* Remember highest leaf page number we've taken cleanup lock on; see
|
|
|
|
* notes in btvacuumscan
|
|
|
|
*/
|
|
|
|
if (blkno > vstate->lastBlockLocked)
|
|
|
|
vstate->lastBlockLocked = blkno;
|
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
/*
|
2014-05-06 18:12:18 +02:00
|
|
|
* Check whether we need to recurse back to earlier pages. What we
|
2006-10-04 02:30:14 +02:00
|
|
|
* are concerned about is a page split that happened since we started
|
|
|
|
* the vacuum scan. If the split moved some tuples to a lower page
|
|
|
|
* then we might have missed 'em. If so, set up for tail recursion.
|
|
|
|
* (Must do this before possibly clearing btpo_cycleid below!)
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (vstate->cycleid != 0 &&
|
|
|
|
opaque->btpo_cycleid == vstate->cycleid &&
|
|
|
|
!(opaque->btpo_flags & BTP_SPLIT_END) &&
|
|
|
|
!P_RIGHTMOST(opaque) &&
|
|
|
|
opaque->btpo_next < orig_blkno)
|
|
|
|
recurse_to = opaque->btpo_next;
|
|
|
|
|
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* Scan over all items to see which ones need deleted according to the
|
|
|
|
* callback function.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
ndeletable = 0;
|
|
|
|
minoff = P_FIRSTDATAKEY(opaque);
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
if (callback)
|
|
|
|
{
|
|
|
|
for (offnum = minoff;
|
|
|
|
offnum <= maxoff;
|
|
|
|
offnum = OffsetNumberNext(offnum))
|
|
|
|
{
|
|
|
|
IndexTuple itup;
|
|
|
|
ItemPointer htup;
|
|
|
|
|
|
|
|
itup = (IndexTuple) PageGetItem(page,
|
|
|
|
PageGetItemId(page, offnum));
|
|
|
|
htup = &(itup->t_tid);
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
|
|
|
|
/*
|
2010-02-26 03:01:40 +01:00
|
|
|
* During Hot Standby we currently assume that
|
|
|
|
* XLOG_BTREE_VACUUM records do not produce conflicts. That is
|
|
|
|
* only true as long as the callback function depends only
|
|
|
|
* upon whether the index tuple refers to heap tuples removed
|
|
|
|
* in the initial heap scan. When vacuum starts it derives a
|
|
|
|
* value of OldestXmin. Backends taking later snapshots could
|
|
|
|
* have a RecentGlobalXmin with a later xid than the vacuum's
|
|
|
|
* OldestXmin, so it is possible that row versions deleted
|
|
|
|
* after OldestXmin could be marked as killed by other
|
|
|
|
* backends. The callback function *could* look at the index
|
|
|
|
* tuple state in isolation and decide to delete the index
|
|
|
|
* tuple, though currently it does not. If it ever did, we
|
|
|
|
* would need to reconsider whether XLOG_BTREE_VACUUM records
|
|
|
|
* should cause conflicts. If they did cause conflicts they
|
|
|
|
* would be fairly harsh conflicts, since we haven't yet
|
|
|
|
* worked out a way to pass a useful value for
|
|
|
|
* latestRemovedXid on the XLOG_BTREE_VACUUM records. This
|
|
|
|
* applies to *any* type of index that marks index tuples as
|
|
|
|
* killed.
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
*/
|
2006-05-08 02:00:17 +02:00
|
|
|
if (callback(htup, callback_state))
|
|
|
|
deletable[ndeletable++] = offnum;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2012-02-21 20:14:16 +01:00
|
|
|
* Apply any needed deletes. We issue just one _bt_delitems_vacuum()
|
|
|
|
* call per page, so as to minimize WAL traffic.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (ndeletable > 0)
|
|
|
|
{
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
/*
|
2016-06-10 00:02:36 +02:00
|
|
|
* Notice that the issued XLOG_BTREE_VACUUM WAL record includes
|
|
|
|
* all information to the replay code to allow it to get a cleanup
|
|
|
|
* lock on all pages between the previous lastBlockVacuumed and
|
|
|
|
* this page. This ensures that WAL replay locks all leaf pages at
|
|
|
|
* some point, which is important should non-MVCC scans be
|
|
|
|
* requested. This is currently unused on standby, but we record
|
|
|
|
* it anyway, so that the WAL contains the required information.
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
*
|
|
|
|
* Since we can visit leaf pages out-of-order when recursing,
|
|
|
|
* replay might end up locking such pages an extra time, but it
|
|
|
|
* doesn't seem worth the amount of bookkeeping it'd take to avoid
|
|
|
|
* that.
|
|
|
|
*/
|
2012-02-21 20:14:16 +01:00
|
|
|
_bt_delitems_vacuum(rel, buf, deletable, ndeletable,
|
2016-04-03 18:46:09 +02:00
|
|
|
vstate->lastBlockVacuumed);
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
|
|
|
|
/*
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
* Remember highest leaf page number we've issued a
|
|
|
|
* XLOG_BTREE_VACUUM WAL record for.
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
*/
|
Fix multiple bugs in index page locking during hot-standby WAL replay.
In ordinary operation, VACUUM must be careful to take a cleanup lock on
each leaf page of a btree index; this ensures that no indexscans could
still be "in flight" to heap tuples due to be deleted. (Because of
possible index-tuple motion due to concurrent page splits, it's not enough
to lock only the pages we're deleting index tuples from.) In Hot Standby,
the WAL replay process must likewise lock every leaf page. There were
several bugs in the code for that:
* The replay scan might come across unused, all-zero pages in the index.
While btree_xlog_vacuum itself did the right thing (ie, nothing) with
such pages, xlogutils.c supposed that such pages must be corrupt and
would throw an error. This accounts for various reports of replication
failures with "PANIC: WAL contains references to invalid pages". To
fix, add a ReadBufferMode value that instructs XLogReadBufferExtended
not to complain when we're doing this.
* btree_xlog_vacuum performed the extra locking if standbyState ==
STANDBY_SNAPSHOT_READY, but that's not the correct test: we won't open up
for hot standby queries until the database has reached consistency, and
we don't want to do the extra locking till then either, for fear of reading
corrupted pages (which bufmgr.c would complain about). Fix by exporting a
new function from xlog.c that will report whether we're actually in hot
standby replay mode.
* To ensure full coverage of the index in the replay scan, btvacuumscan
would emit a dummy WAL record for the last page of the index, if no
vacuuming work had been done on that page. However, if the last page
of the index is all-zero, that would result in corruption of said page,
since the functions called on it weren't prepared to handle that case.
There's no need to lock any such pages, so change the logic to target
the last normal leaf page instead.
The first two of these bugs were diagnosed by Andres Freund, the other one
by me. Fixes based on ideas from Heikki Linnakangas and myself.
This has been wrong since Hot Standby was introduced, so back-patch to 9.0.
2014-01-14 23:34:47 +01:00
|
|
|
if (blkno > vstate->lastBlockVacuumed)
|
|
|
|
vstate->lastBlockVacuumed = blkno;
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
|
2006-05-08 02:00:17 +02:00
|
|
|
stats->tuples_removed += ndeletable;
|
|
|
|
/* must recompute maxoff */
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If the page has been split during this vacuum cycle, it seems
|
|
|
|
* worth expending a write to clear btpo_cycleid even if we don't
|
2012-02-21 20:14:16 +01:00
|
|
|
* have any deletions to do. (If we do, _bt_delitems_vacuum takes
|
|
|
|
* care of this.) This ensures we won't process the page again.
|
2006-05-08 02:00:17 +02:00
|
|
|
*
|
2006-10-04 02:30:14 +02:00
|
|
|
* We treat this like a hint-bit update because there's no need to
|
|
|
|
* WAL-log it.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (vstate->cycleid != 0 &&
|
|
|
|
opaque->btpo_cycleid == vstate->cycleid)
|
|
|
|
{
|
|
|
|
opaque->btpo_cycleid = 0;
|
2013-06-17 17:02:12 +02:00
|
|
|
MarkBufferDirtyHint(buf, true);
|
2006-05-08 02:00:17 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* If it's now empty, try to delete; else count the live tuples. We
|
|
|
|
* don't delete when recursing, though, to avoid putting entries into
|
|
|
|
* freePages out-of-order (doesn't seem worth any extra code to handle
|
|
|
|
* the case).
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (minoff > maxoff)
|
|
|
|
delete_now = (blkno == orig_blkno);
|
|
|
|
else
|
|
|
|
stats->num_index_tuples += maxoff - minoff + 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (delete_now)
|
|
|
|
{
|
|
|
|
MemoryContext oldcontext;
|
|
|
|
int ndel;
|
|
|
|
|
|
|
|
/* Run pagedel in a temp context to avoid memory leakage */
|
|
|
|
MemoryContextReset(vstate->pagedelcontext);
|
|
|
|
oldcontext = MemoryContextSwitchTo(vstate->pagedelcontext);
|
|
|
|
|
Fix race condition in B-tree page deletion.
In short, we don't allow a page to be deleted if it's the rightmost child
of its parent, but that situation can change after we check for it.
Problem
-------
We check that the page to be deleted is not the rightmost child of its
parent, and then lock its left sibling, the page itself, its right sibling,
and the parent, in that order. However, if the parent page is split after
the check but before acquiring the locks, the target page might become the
rightmost child, if the split happens at the right place. That leads to an
error in vacuum (I reproduced this by setting a breakpoint in debugger):
ERROR: failed to delete rightmost child 41 of block 3 in index "foo_pkey"
We currently re-check that the page is still the rightmost child, and throw
the above error if it's not. We could easily just give up rather than throw
an error, but that approach doesn't scale to half-dead pages. To recap,
although we don't normally allow deleting the rightmost child, if the page
is the *only* child of its parent, we delete the child page and mark the
parent page as half-dead in one atomic operation. But before we do that, we
check that the parent can later be deleted, by checking that it in turn is
not the rightmost child of the grandparent (potentially recursing all the
way up to the root). But the same situation can arise there - the
grandparent can be split while we're not holding the locks. We end up with
a half-dead page that we cannot delete.
To make things worse, the keyspace of the deleted page has already been
transferred to its right sibling. As the README points out, the keyspace at
the grandparent level is "out-of-whack" until the half-dead page is deleted,
and if enough tuples with keys in the transferred keyspace are inserted, the
page might get split and a downlink might be inserted into the grandparent
that is out-of-order. That might not cause any serious problem if it's
transient (as the README ponders), but is surely bad if it stays that way.
Solution
--------
This patch changes the page deletion algorithm to avoid that problem. After
checking that the topmost page in the chain of to-be-deleted pages is not
the rightmost child of its parent, and then deleting the pages from bottom
up, unlink the pages from top to bottom. This way, the intermediate stages
are similar to the intermediate stages in page splitting, and there is no
transient stage where the keyspace is "out-of-whack". The topmost page in
the to-be-deleted chain doesn't have a downlink pointing to it, like a page
split before the downlink has been inserted.
This also allows us to get rid of the cleanup step after WAL recovery, if we
crash during page deletion. The deletion will be continued at next VACUUM,
but the tree is consistent for searches and insertions at every step.
This bug is old, all supported versions are affected, but this patch is too
big to back-patch (and changes the WAL record formats of related records).
We have not heard any reports of the bug from users, so clearly it's not
easy to bump into. Maybe backpatch later, after this has had some field
testing.
Reviewed by Kevin Grittner and Peter Geoghegan.
2014-03-14 14:43:58 +01:00
|
|
|
ndel = _bt_pagedel(rel, buf);
|
2006-05-08 02:00:17 +02:00
|
|
|
|
|
|
|
/* count only this page, else may double-count parent */
|
|
|
|
if (ndel)
|
|
|
|
stats->pages_deleted++;
|
|
|
|
|
|
|
|
MemoryContextSwitchTo(oldcontext);
|
|
|
|
/* pagedel released buffer, so we shouldn't */
|
|
|
|
}
|
|
|
|
else
|
|
|
|
_bt_relbuf(rel, buf);
|
|
|
|
|
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* This is really tail recursion, but if the compiler is too stupid to
|
|
|
|
* optimize it as such, we'd eat an uncomfortably large amount of stack
|
|
|
|
* space per recursion level (due to the deletable[] array). A failure is
|
|
|
|
* improbable since the number of levels isn't likely to be large ... but
|
|
|
|
* just in case, let's hand-optimize into a loop.
|
2006-05-08 02:00:17 +02:00
|
|
|
*/
|
|
|
|
if (recurse_to != P_NONE)
|
|
|
|
{
|
|
|
|
blkno = recurse_to;
|
|
|
|
goto restart;
|
|
|
|
}
|
2003-02-22 01:45:05 +01:00
|
|
|
}
|
2011-12-18 21:49:00 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* btcanreturn() -- Check whether btree indexes support index-only scans.
|
|
|
|
*
|
|
|
|
* btrees always do, so this is trivial.
|
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
bool
|
|
|
|
btcanreturn(Relation index, int attno)
|
2011-12-18 21:49:00 +01:00
|
|
|
{
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
return true;
|
2011-12-18 21:49:00 +01:00
|
|
|
}
|