tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
|
*
|
|
|
|
|
* heapam_handler.c
|
|
|
|
|
* heap table access method code
|
|
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* IDENTIFICATION
|
|
|
|
|
* src/backend/access/heap/heapam_handler.c
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* NOTES
|
2019-03-29 16:16:35 +01:00
|
|
|
* This files wires up the lower level heapam.c et al routines with the
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
* tableam abstraction.
|
|
|
|
|
*
|
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
#include "postgres.h"
|
|
|
|
|
|
2019-12-27 00:09:00 +01:00
|
|
|
#include "access/genam.h"
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
#include "access/heapam.h"
|
2019-07-08 17:58:05 +02:00
|
|
|
#include "access/heaptoast.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "access/multixact.h"
|
|
|
|
|
#include "access/rewriteheap.h"
|
2020-07-29 06:46:58 +02:00
|
|
|
#include "access/syncscan.h"
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
#include "access/tableam.h"
|
2019-03-31 05:18:53 +02:00
|
|
|
#include "access/tsmapi.h"
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
#include "access/xact.h"
|
2019-03-28 03:59:06 +01:00
|
|
|
#include "catalog/catalog.h"
|
|
|
|
|
#include "catalog/index.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "catalog/storage.h"
|
|
|
|
|
#include "catalog/storage_xlog.h"
|
|
|
|
|
#include "commands/progress.h"
|
2019-03-28 03:59:06 +01:00
|
|
|
#include "executor/executor.h"
|
2019-11-12 04:00:16 +01:00
|
|
|
#include "miscadmin.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "pgstat.h"
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
#include "storage/bufmgr.h"
|
2019-03-28 03:59:06 +01:00
|
|
|
#include "storage/bufpage.h"
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
#include "storage/lmgr.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "storage/predicate.h"
|
2019-03-28 03:59:06 +01:00
|
|
|
#include "storage/procarray.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "storage/smgr.h"
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
#include "utils/builtins.h"
|
2019-03-29 04:01:14 +01:00
|
|
|
#include "utils/rel.h"
|
|
|
|
|
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
static TM_Result heapam_tuple_lock_internal(Relation relation, ItemPointer tid,
|
|
|
|
|
Snapshot snapshot, TupleTableSlot *slot,
|
|
|
|
|
CommandId cid, LockTupleMode mode,
|
|
|
|
|
LockWaitPolicy wait_policy, uint8 flags,
|
|
|
|
|
TM_FailureData *tmfd, bool updated);
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
static void reform_and_rewrite_tuple(HeapTuple tuple,
|
2019-03-29 04:01:14 +01:00
|
|
|
Relation OldHeap, Relation NewHeap,
|
|
|
|
|
Datum *values, bool *isnull, RewriteState rwstate);
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
|
2019-03-31 05:18:53 +02:00
|
|
|
static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer,
|
|
|
|
|
HeapTuple tuple,
|
|
|
|
|
OffsetNumber tupoffset);
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan);
|
|
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
static const TableAmRoutine heapam_methods;
|
|
|
|
|
|
|
|
|
|
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Slot related callbacks for heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static const TupleTableSlotOps *
|
|
|
|
|
heapam_slot_callbacks(Relation relation)
|
|
|
|
|
{
|
|
|
|
|
return &TTSOpsBufferHeapTuple;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Index Scan Callbacks for heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static IndexFetchTableData *
|
|
|
|
|
heapam_index_fetch_begin(Relation rel)
|
|
|
|
|
{
|
|
|
|
|
IndexFetchHeapData *hscan = palloc0(sizeof(IndexFetchHeapData));
|
|
|
|
|
|
|
|
|
|
hscan->xs_base.rel = rel;
|
|
|
|
|
hscan->xs_cbuf = InvalidBuffer;
|
|
|
|
|
|
|
|
|
|
return &hscan->xs_base;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_index_fetch_reset(IndexFetchTableData *scan)
|
|
|
|
|
{
|
|
|
|
|
IndexFetchHeapData *hscan = (IndexFetchHeapData *) scan;
|
|
|
|
|
|
|
|
|
|
if (BufferIsValid(hscan->xs_cbuf))
|
|
|
|
|
{
|
|
|
|
|
ReleaseBuffer(hscan->xs_cbuf);
|
|
|
|
|
hscan->xs_cbuf = InvalidBuffer;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_index_fetch_end(IndexFetchTableData *scan)
|
|
|
|
|
{
|
|
|
|
|
IndexFetchHeapData *hscan = (IndexFetchHeapData *) scan;
|
|
|
|
|
|
|
|
|
|
heapam_index_fetch_reset(scan);
|
|
|
|
|
|
|
|
|
|
pfree(hscan);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static bool
|
|
|
|
|
heapam_index_fetch_tuple(struct IndexFetchTableData *scan,
|
|
|
|
|
ItemPointer tid,
|
|
|
|
|
Snapshot snapshot,
|
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
|
bool *call_again, bool *all_dead)
|
|
|
|
|
{
|
|
|
|
|
IndexFetchHeapData *hscan = (IndexFetchHeapData *) scan;
|
|
|
|
|
BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot;
|
|
|
|
|
bool got_heap_tuple;
|
|
|
|
|
|
|
|
|
|
Assert(TTS_IS_BUFFERTUPLE(slot));
|
|
|
|
|
|
|
|
|
|
/* We can skip the buffer-switching logic if we're in mid-HOT chain. */
|
|
|
|
|
if (!*call_again)
|
|
|
|
|
{
|
|
|
|
|
/* Switch to correct buffer if we don't have it already */
|
|
|
|
|
Buffer prev_buf = hscan->xs_cbuf;
|
|
|
|
|
|
|
|
|
|
hscan->xs_cbuf = ReleaseAndReadBuffer(hscan->xs_cbuf,
|
|
|
|
|
hscan->xs_base.rel,
|
|
|
|
|
ItemPointerGetBlockNumber(tid));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prune page, but only if we weren't already on this page
|
|
|
|
|
*/
|
|
|
|
|
if (prev_buf != hscan->xs_cbuf)
|
|
|
|
|
heap_page_prune_opt(hscan->xs_base.rel, hscan->xs_cbuf);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Obtain share-lock on the buffer so we can examine visibility */
|
|
|
|
|
LockBuffer(hscan->xs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
got_heap_tuple = heap_hot_search_buffer(tid,
|
|
|
|
|
hscan->xs_base.rel,
|
|
|
|
|
hscan->xs_cbuf,
|
|
|
|
|
snapshot,
|
|
|
|
|
&bslot->base.tupdata,
|
|
|
|
|
all_dead,
|
|
|
|
|
!*call_again);
|
|
|
|
|
bslot->base.tupdata.t_self = *tid;
|
|
|
|
|
LockBuffer(hscan->xs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
if (got_heap_tuple)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Only in a non-MVCC snapshot can more than one member of the HOT
|
|
|
|
|
* chain be visible.
|
|
|
|
|
*/
|
|
|
|
|
*call_again = !IsMVCCSnapshot(snapshot);
|
|
|
|
|
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(scan->rel);
|
|
|
|
|
ExecStoreBufferHeapTuple(&bslot->base.tupdata, slot, hscan->xs_cbuf);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* We've reached the end of the HOT chain. */
|
|
|
|
|
*call_again = false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return got_heap_tuple;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Callbacks for non-modifying operations on individual tuples for heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
2019-03-25 08:13:42 +01:00
|
|
|
static bool
|
|
|
|
|
heapam_fetch_row_version(Relation relation,
|
|
|
|
|
ItemPointer tid,
|
|
|
|
|
Snapshot snapshot,
|
|
|
|
|
TupleTableSlot *slot)
|
|
|
|
|
{
|
|
|
|
|
BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot;
|
|
|
|
|
Buffer buffer;
|
|
|
|
|
|
|
|
|
|
Assert(TTS_IS_BUFFERTUPLE(slot));
|
|
|
|
|
|
|
|
|
|
bslot->base.tupdata.t_self = *tid;
|
2022-04-13 19:35:02 +02:00
|
|
|
if (heap_fetch(relation, snapshot, &bslot->base.tupdata, &buffer, false))
|
2019-03-25 08:13:42 +01:00
|
|
|
{
|
|
|
|
|
/* store in slot, transferring existing pin */
|
|
|
|
|
ExecStorePinnedBufferHeapTuple(&bslot->base.tupdata, slot, buffer);
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(relation);
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
2019-05-18 03:52:01 +02:00
|
|
|
static bool
|
|
|
|
|
heapam_tuple_tid_valid(TableScanDesc scan, ItemPointer tid)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
|
|
|
|
|
|
|
|
|
return ItemPointerIsValid(tid) &&
|
|
|
|
|
ItemPointerGetBlockNumber(tid) < hscan->rs_nblocks;
|
|
|
|
|
}
|
|
|
|
|
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
static bool
|
|
|
|
|
heapam_tuple_satisfies_snapshot(Relation rel, TupleTableSlot *slot,
|
|
|
|
|
Snapshot snapshot)
|
|
|
|
|
{
|
|
|
|
|
BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot;
|
|
|
|
|
bool res;
|
|
|
|
|
|
|
|
|
|
Assert(TTS_IS_BUFFERTUPLE(slot));
|
|
|
|
|
Assert(BufferIsValid(bslot->buffer));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We need buffer pin and lock to call HeapTupleSatisfiesVisibility.
|
|
|
|
|
* Caller should be holding pin, but not lock.
|
|
|
|
|
*/
|
|
|
|
|
LockBuffer(bslot->buffer, BUFFER_LOCK_SHARE);
|
|
|
|
|
res = HeapTupleSatisfiesVisibility(bslot->base.tuple, snapshot,
|
|
|
|
|
bslot->buffer);
|
|
|
|
|
LockBuffer(bslot->buffer, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
return res;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
/* ----------------------------------------------------------------------------
|
|
|
|
|
* Functions for manipulations of physical tuples for heap AM.
|
|
|
|
|
* ----------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_tuple_insert(Relation relation, TupleTableSlot *slot, CommandId cid,
|
|
|
|
|
int options, BulkInsertState bistate)
|
|
|
|
|
{
|
|
|
|
|
bool shouldFree = true;
|
|
|
|
|
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
|
|
|
|
|
|
|
|
|
|
/* Update the tuple with table oid */
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(relation);
|
|
|
|
|
tuple->t_tableOid = slot->tts_tableOid;
|
|
|
|
|
|
|
|
|
|
/* Perform the insertion, and copy the resulting ItemPointer */
|
|
|
|
|
heap_insert(relation, tuple, cid, options, bistate);
|
|
|
|
|
ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
|
|
|
|
|
|
|
|
|
|
if (shouldFree)
|
|
|
|
|
pfree(tuple);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
2019-03-31 05:13:56 +02:00
|
|
|
heapam_tuple_insert_speculative(Relation relation, TupleTableSlot *slot,
|
|
|
|
|
CommandId cid, int options,
|
|
|
|
|
BulkInsertState bistate, uint32 specToken)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
|
|
|
|
bool shouldFree = true;
|
|
|
|
|
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
|
|
|
|
|
|
|
|
|
|
/* Update the tuple with table oid */
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(relation);
|
|
|
|
|
tuple->t_tableOid = slot->tts_tableOid;
|
|
|
|
|
|
|
|
|
|
HeapTupleHeaderSetSpeculativeToken(tuple->t_data, specToken);
|
|
|
|
|
options |= HEAP_INSERT_SPECULATIVE;
|
|
|
|
|
|
|
|
|
|
/* Perform the insertion, and copy the resulting ItemPointer */
|
|
|
|
|
heap_insert(relation, tuple, cid, options, bistate);
|
|
|
|
|
ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
|
|
|
|
|
|
|
|
|
|
if (shouldFree)
|
|
|
|
|
pfree(tuple);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
2019-03-31 05:13:56 +02:00
|
|
|
heapam_tuple_complete_speculative(Relation relation, TupleTableSlot *slot,
|
2019-06-08 04:46:38 +02:00
|
|
|
uint32 specToken, bool succeeded)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
|
|
|
|
bool shouldFree = true;
|
|
|
|
|
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
|
|
|
|
|
|
|
|
|
|
/* adjust the tuple's state accordingly */
|
2019-05-14 21:11:26 +02:00
|
|
|
if (succeeded)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
heap_finish_speculative(relation, &slot->tts_tid);
|
|
|
|
|
else
|
|
|
|
|
heap_abort_speculative(relation, &slot->tts_tid);
|
|
|
|
|
|
|
|
|
|
if (shouldFree)
|
|
|
|
|
pfree(tuple);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static TM_Result
|
|
|
|
|
heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid,
|
|
|
|
|
Snapshot snapshot, Snapshot crosscheck, bool wait,
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
TM_FailureData *tmfd, bool changingPart,
|
|
|
|
|
LazyTupleTableSlot *lockedSlot)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
TM_Result result;
|
|
|
|
|
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
/*
|
|
|
|
|
* Currently Deleting of index tuples are handled at vacuum, in case if
|
|
|
|
|
* the storage itself is cleaning the dead tuples by itself, it is the
|
|
|
|
|
* time to call the index tuple deletion also.
|
|
|
|
|
*/
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
result = heap_delete(relation, tid, cid, crosscheck, wait,
|
|
|
|
|
tmfd, changingPart);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the tuple has been concurrently updated, then get the lock on it.
|
|
|
|
|
* (Do this if caller asked for tat by providing a 'lockedSlot'.) With the
|
|
|
|
|
* lock held retry of delete should succeed even if there are more
|
|
|
|
|
* concurrent update attempts.
|
|
|
|
|
*/
|
|
|
|
|
if (result == TM_Updated && lockedSlot)
|
|
|
|
|
{
|
|
|
|
|
TupleTableSlot *evalSlot;
|
|
|
|
|
|
|
|
|
|
Assert(wait);
|
|
|
|
|
|
|
|
|
|
evalSlot = LAZY_TTS_EVAL(lockedSlot);
|
|
|
|
|
result = heapam_tuple_lock_internal(relation, tid, snapshot,
|
|
|
|
|
evalSlot, cid, LockTupleExclusive,
|
|
|
|
|
LockWaitBlock,
|
|
|
|
|
TUPLE_LOCK_FLAG_FIND_LAST_VERSION,
|
|
|
|
|
tmfd, true);
|
|
|
|
|
|
|
|
|
|
if (result == TM_Ok)
|
|
|
|
|
{
|
|
|
|
|
tmfd->traversed = true;
|
|
|
|
|
return TM_Updated;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return result;
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
static TM_Result
|
|
|
|
|
heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
|
|
|
|
|
CommandId cid, Snapshot snapshot, Snapshot crosscheck,
|
|
|
|
|
bool wait, TM_FailureData *tmfd,
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes,
|
|
|
|
|
LazyTupleTableSlot *lockedSlot)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
|
|
|
|
bool shouldFree = true;
|
|
|
|
|
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
|
|
|
|
|
TM_Result result;
|
|
|
|
|
|
|
|
|
|
/* Update the tuple with table oid */
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(relation);
|
|
|
|
|
tuple->t_tableOid = slot->tts_tableOid;
|
|
|
|
|
|
|
|
|
|
result = heap_update(relation, otid, tuple, cid, crosscheck, wait,
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
tmfd, lockmode, update_indexes);
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Decide whether new index entries are needed for the tuple
|
|
|
|
|
*
|
|
|
|
|
* Note: heap_update returns the tid (location) of the new tuple in the
|
|
|
|
|
* t_self field.
|
|
|
|
|
*
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
* If the update is not HOT, we must update all indexes. If the update
|
|
|
|
|
* is HOT, it could be that we updated summarized columns, so we either
|
|
|
|
|
* update only summarized indexes, or none at all.
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
*/
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
if (result != TM_Ok)
|
|
|
|
|
{
|
|
|
|
|
Assert(*update_indexes == TU_None);
|
|
|
|
|
*update_indexes = TU_None;
|
|
|
|
|
}
|
|
|
|
|
else if (!HeapTupleIsHeapOnly(tuple))
|
|
|
|
|
Assert(*update_indexes == TU_All);
|
|
|
|
|
else
|
|
|
|
|
Assert((*update_indexes == TU_Summarizing) ||
|
|
|
|
|
(*update_indexes == TU_None));
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
|
|
|
|
|
if (shouldFree)
|
|
|
|
|
pfree(tuple);
|
|
|
|
|
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
/*
|
|
|
|
|
* If the tuple has been concurrently updated, then get the lock on it.
|
|
|
|
|
* (Do this if caller asked for tat by providing a 'lockedSlot'.) With the
|
|
|
|
|
* lock held retry of update should succeed even if there are more
|
|
|
|
|
* concurrent update attempts.
|
|
|
|
|
*/
|
|
|
|
|
if (result == TM_Updated && lockedSlot)
|
|
|
|
|
{
|
|
|
|
|
TupleTableSlot *evalSlot;
|
|
|
|
|
|
|
|
|
|
Assert(wait);
|
|
|
|
|
|
|
|
|
|
evalSlot = LAZY_TTS_EVAL(lockedSlot);
|
|
|
|
|
result = heapam_tuple_lock_internal(relation, otid, snapshot,
|
|
|
|
|
evalSlot, cid, *lockmode,
|
|
|
|
|
LockWaitBlock,
|
|
|
|
|
TUPLE_LOCK_FLAG_FIND_LAST_VERSION,
|
|
|
|
|
tmfd, true);
|
|
|
|
|
|
|
|
|
|
if (result == TM_Ok)
|
|
|
|
|
{
|
|
|
|
|
tmfd->traversed = true;
|
|
|
|
|
return TM_Updated;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
return result;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static TM_Result
|
|
|
|
|
heapam_tuple_lock(Relation relation, ItemPointer tid, Snapshot snapshot,
|
|
|
|
|
TupleTableSlot *slot, CommandId cid, LockTupleMode mode,
|
|
|
|
|
LockWaitPolicy wait_policy, uint8 flags,
|
|
|
|
|
TM_FailureData *tmfd)
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
{
|
|
|
|
|
return heapam_tuple_lock_internal(relation, tid, snapshot, slot, cid,
|
|
|
|
|
mode, wait_policy, flags, tmfd, false);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* This routine does the work for heapam_tuple_lock(), but also support
|
|
|
|
|
* `updated` argument to re-use the work done by heapam_tuple_update() or
|
|
|
|
|
* heapam_tuple_delete() on figuring out that tuple was concurrently updated.
|
|
|
|
|
*/
|
|
|
|
|
static TM_Result
|
|
|
|
|
heapam_tuple_lock_internal(Relation relation, ItemPointer tid,
|
|
|
|
|
Snapshot snapshot, TupleTableSlot *slot,
|
|
|
|
|
CommandId cid, LockTupleMode mode,
|
|
|
|
|
LockWaitPolicy wait_policy, uint8 flags,
|
|
|
|
|
TM_FailureData *tmfd, bool updated)
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
|
|
|
|
BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot;
|
|
|
|
|
TM_Result result;
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
Buffer buffer = InvalidBuffer;
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
HeapTuple tuple = &bslot->base.tupdata;
|
|
|
|
|
bool follow_updates;
|
|
|
|
|
|
|
|
|
|
follow_updates = (flags & TUPLE_LOCK_FLAG_LOCK_UPDATE_IN_PROGRESS) != 0;
|
|
|
|
|
tmfd->traversed = false;
|
|
|
|
|
|
|
|
|
|
Assert(TTS_IS_BUFFERTUPLE(slot));
|
|
|
|
|
|
|
|
|
|
tuple_lock_retry:
|
|
|
|
|
tuple->t_self = *tid;
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
if (!updated)
|
|
|
|
|
result = heap_lock_tuple(relation, tuple, cid, mode, wait_policy,
|
|
|
|
|
follow_updates, &buffer, tmfd);
|
|
|
|
|
else
|
|
|
|
|
result = TM_Updated;
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
|
|
|
|
|
if (result == TM_Updated &&
|
|
|
|
|
(flags & TUPLE_LOCK_FLAG_FIND_LAST_VERSION))
|
|
|
|
|
{
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
if (!updated)
|
|
|
|
|
{
|
|
|
|
|
/* Should not encounter speculative tuple on recheck */
|
|
|
|
|
Assert(!HeapTupleHeaderIsSpeculative(tuple->t_data));
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
|
Allow locking updated tuples in tuple_update() and tuple_delete()
Currently, in read committed transaction isolation mode (default), we have the
following sequence of actions when tuple_update()/tuple_delete() finds
the tuple updated by concurrent transaction.
1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which
returns TM_Updated.
2. Lock tuple with tuple_lock().
3. Re-evaluate plan qual (recheck if we still need to update/delete and
calculate the new tuple for update).
4. Second attempt to update/delete tuple with tuple_update()/tuple_delete().
This attempt should be successful, since the tuple was previously locked.
This patch eliminates step 2 by taking the lock during first
tuple_update()/tuple_delete() call. Heap table access method saves some
efforts by checking the updated tuple once instead of twice. Future
undo-based table access methods, which will start from the latest row version,
can immediately place a lock there.
The code in nodeModifyTable.c is simplified by removing the nested switch/case.
Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com
Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp
Reviewed-by: Andres Freund, Chris Travers
2023-03-22 22:13:37 +01:00
|
|
|
ReleaseBuffer(buffer);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
updated = false;
|
|
|
|
|
}
|
2020-07-20 01:12:51 +02:00
|
|
|
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
if (!ItemPointerEquals(&tmfd->ctid, &tuple->t_self))
|
|
|
|
|
{
|
|
|
|
|
SnapshotData SnapshotDirty;
|
|
|
|
|
TransactionId priorXmax;
|
|
|
|
|
|
|
|
|
|
/* it was updated, so look at the updated version */
|
|
|
|
|
*tid = tmfd->ctid;
|
|
|
|
|
/* updated row should have xmin matching this xmax */
|
|
|
|
|
priorXmax = tmfd->xmax;
|
|
|
|
|
|
|
|
|
|
/* signal that a tuple later in the chain is getting locked */
|
|
|
|
|
tmfd->traversed = true;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* fetch target tuple
|
|
|
|
|
*
|
|
|
|
|
* Loop here to deal with updated or busy tuples
|
|
|
|
|
*/
|
|
|
|
|
InitDirtySnapshot(SnapshotDirty);
|
|
|
|
|
for (;;)
|
|
|
|
|
{
|
|
|
|
|
if (ItemPointerIndicatesMovedPartitions(tid))
|
|
|
|
|
ereport(ERROR,
|
|
|
|
|
(errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
|
|
|
|
|
errmsg("tuple to be locked was already moved to another partition due to concurrent update")));
|
|
|
|
|
|
|
|
|
|
tuple->t_self = *tid;
|
2022-04-13 19:35:02 +02:00
|
|
|
if (heap_fetch(relation, &SnapshotDirty, tuple, &buffer, true))
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* If xmin isn't what we're expecting, the slot must have
|
|
|
|
|
* been recycled and reused for an unrelated tuple. This
|
|
|
|
|
* implies that the latest version of the row was deleted,
|
|
|
|
|
* so we need do nothing. (Should be safe to examine xmin
|
|
|
|
|
* without getting buffer's content lock. We assume
|
|
|
|
|
* reading a TransactionId to be atomic, and Xmin never
|
|
|
|
|
* changes in an existing tuple, except to invalid or
|
|
|
|
|
* frozen, and neither of those can match priorXmax.)
|
|
|
|
|
*/
|
|
|
|
|
if (!TransactionIdEquals(HeapTupleHeaderGetXmin(tuple->t_data),
|
|
|
|
|
priorXmax))
|
|
|
|
|
{
|
|
|
|
|
ReleaseBuffer(buffer);
|
|
|
|
|
return TM_Deleted;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* otherwise xmin should not be dirty... */
|
|
|
|
|
if (TransactionIdIsValid(SnapshotDirty.xmin))
|
2019-08-01 11:05:08 +02:00
|
|
|
ereport(ERROR,
|
|
|
|
|
(errcode(ERRCODE_DATA_CORRUPTED),
|
2021-08-30 22:29:12 +02:00
|
|
|
errmsg_internal("t_xmin %u is uncommitted in tuple (%u,%u) to be updated in table \"%s\"",
|
|
|
|
|
SnapshotDirty.xmin,
|
|
|
|
|
ItemPointerGetBlockNumber(&tuple->t_self),
|
|
|
|
|
ItemPointerGetOffsetNumber(&tuple->t_self),
|
|
|
|
|
RelationGetRelationName(relation))));
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If tuple is being updated by other transaction then we
|
|
|
|
|
* have to wait for its commit/abort, or die trying.
|
|
|
|
|
*/
|
|
|
|
|
if (TransactionIdIsValid(SnapshotDirty.xmax))
|
|
|
|
|
{
|
|
|
|
|
ReleaseBuffer(buffer);
|
|
|
|
|
switch (wait_policy)
|
|
|
|
|
{
|
|
|
|
|
case LockWaitBlock:
|
|
|
|
|
XactLockTableWait(SnapshotDirty.xmax,
|
|
|
|
|
relation, &tuple->t_self,
|
|
|
|
|
XLTW_FetchUpdated);
|
|
|
|
|
break;
|
|
|
|
|
case LockWaitSkip:
|
|
|
|
|
if (!ConditionalXactLockTableWait(SnapshotDirty.xmax))
|
|
|
|
|
/* skip instead of waiting */
|
|
|
|
|
return TM_WouldBlock;
|
|
|
|
|
break;
|
|
|
|
|
case LockWaitError:
|
|
|
|
|
if (!ConditionalXactLockTableWait(SnapshotDirty.xmax))
|
|
|
|
|
ereport(ERROR,
|
|
|
|
|
(errcode(ERRCODE_LOCK_NOT_AVAILABLE),
|
|
|
|
|
errmsg("could not obtain lock on row in relation \"%s\"",
|
|
|
|
|
RelationGetRelationName(relation))));
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
continue; /* loop back to repeat heap_fetch */
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If tuple was inserted by our own transaction, we have
|
|
|
|
|
* to check cmin against cid: cmin >= current CID means
|
|
|
|
|
* our command cannot see the tuple, so we should ignore
|
|
|
|
|
* it. Otherwise heap_lock_tuple() will throw an error,
|
|
|
|
|
* and so would any later attempt to update or delete the
|
|
|
|
|
* tuple. (We need not check cmax because
|
|
|
|
|
* HeapTupleSatisfiesDirty will consider a tuple deleted
|
|
|
|
|
* by our transaction dead, regardless of cmax.) We just
|
|
|
|
|
* checked that priorXmax == xmin, so we can test that
|
|
|
|
|
* variable instead of doing HeapTupleHeaderGetXmin again.
|
|
|
|
|
*/
|
|
|
|
|
if (TransactionIdIsCurrentTransactionId(priorXmax) &&
|
|
|
|
|
HeapTupleHeaderGetCmin(tuple->t_data) >= cid)
|
|
|
|
|
{
|
2019-04-08 07:14:47 +02:00
|
|
|
tmfd->xmax = priorXmax;
|
2019-05-22 18:55:34 +02:00
|
|
|
|
2019-04-08 07:14:47 +02:00
|
|
|
/*
|
|
|
|
|
* Cmin is the problematic value, so store that. See
|
|
|
|
|
* above.
|
|
|
|
|
*/
|
|
|
|
|
tmfd->cmax = HeapTupleHeaderGetCmin(tuple->t_data);
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
ReleaseBuffer(buffer);
|
2019-04-08 07:14:47 +02:00
|
|
|
return TM_SelfModified;
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* This is a live tuple, so try to lock it again.
|
|
|
|
|
*/
|
|
|
|
|
ReleaseBuffer(buffer);
|
|
|
|
|
goto tuple_lock_retry;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the referenced slot was actually empty, the latest
|
|
|
|
|
* version of the row must have been deleted, so we need do
|
|
|
|
|
* nothing.
|
|
|
|
|
*/
|
|
|
|
|
if (tuple->t_data == NULL)
|
|
|
|
|
{
|
2022-04-13 19:35:02 +02:00
|
|
|
Assert(!BufferIsValid(buffer));
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
return TM_Deleted;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* As above, if xmin isn't what we're expecting, do nothing.
|
|
|
|
|
*/
|
|
|
|
|
if (!TransactionIdEquals(HeapTupleHeaderGetXmin(tuple->t_data),
|
|
|
|
|
priorXmax))
|
|
|
|
|
{
|
2022-04-13 19:35:02 +02:00
|
|
|
ReleaseBuffer(buffer);
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
return TM_Deleted;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If we get here, the tuple was found but failed
|
|
|
|
|
* SnapshotDirty. Assuming the xmin is either a committed xact
|
|
|
|
|
* or our own xact (as it certainly should be if we're trying
|
|
|
|
|
* to modify the tuple), this must mean that the row was
|
|
|
|
|
* updated or deleted by either a committed xact or our own
|
|
|
|
|
* xact. If it was deleted, we can ignore it; if it was
|
|
|
|
|
* updated then chain up to the next version and repeat the
|
|
|
|
|
* whole process.
|
|
|
|
|
*
|
|
|
|
|
* As above, it should be safe to examine xmax and t_ctid
|
|
|
|
|
* without the buffer content lock, because they can't be
|
2022-04-13 19:35:02 +02:00
|
|
|
* changing. We'd better hold a buffer pin though.
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
*/
|
|
|
|
|
if (ItemPointerEquals(&tuple->t_self, &tuple->t_data->t_ctid))
|
|
|
|
|
{
|
|
|
|
|
/* deleted, so forget about it */
|
2022-04-13 19:35:02 +02:00
|
|
|
ReleaseBuffer(buffer);
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
return TM_Deleted;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* updated, so look at the updated row */
|
|
|
|
|
*tid = tuple->t_data->t_ctid;
|
|
|
|
|
/* updated row should have xmin matching this xmax */
|
|
|
|
|
priorXmax = HeapTupleHeaderGetUpdateXid(tuple->t_data);
|
2022-04-13 19:35:02 +02:00
|
|
|
ReleaseBuffer(buffer);
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
/* loop back to fetch next in chain */
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* tuple was deleted, so give up */
|
|
|
|
|
return TM_Deleted;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
slot->tts_tableOid = RelationGetRelid(relation);
|
|
|
|
|
tuple->t_tableOid = slot->tts_tableOid;
|
|
|
|
|
|
|
|
|
|
/* store in slot, transferring existing pin */
|
|
|
|
|
ExecStorePinnedBufferHeapTuple(tuple, slot, buffer);
|
|
|
|
|
|
|
|
|
|
return result;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* DDL related callbacks for heap AM.
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
2019-03-29 04:01:14 +01:00
|
|
|
static void
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
heapam_relation_set_new_filelocator(Relation rel,
|
|
|
|
|
const RelFileLocator *newrlocator,
|
|
|
|
|
char persistence,
|
|
|
|
|
TransactionId *freezeXid,
|
|
|
|
|
MultiXactId *minmulti)
|
2019-03-29 04:01:14 +01:00
|
|
|
{
|
2019-04-30 04:28:05 +02:00
|
|
|
SMgrRelation srel;
|
|
|
|
|
|
2019-03-29 04:01:14 +01:00
|
|
|
/*
|
|
|
|
|
* Initialize to the minimum XID that could put tuples in the table. We
|
|
|
|
|
* know that no xacts older than RecentXmin are still running, so that
|
|
|
|
|
* will do.
|
|
|
|
|
*/
|
|
|
|
|
*freezeXid = RecentXmin;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Similarly, initialize the minimum Multixact to the first value that
|
|
|
|
|
* could possibly be stored in tuples in the table. Running transactions
|
|
|
|
|
* could reuse values from their local cache, so we are careful to
|
|
|
|
|
* consider all currently running multis.
|
|
|
|
|
*
|
|
|
|
|
* XXX this could be refined further, but is it worth the hassle?
|
|
|
|
|
*/
|
|
|
|
|
*minmulti = GetOldestMultiXactId();
|
|
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
srel = RelationCreateStorage(*newrlocator, persistence, true);
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If required, set up an init fork for an unlogged table so that it can
|
|
|
|
|
* be correctly reinitialized on restart. An immediate sync is required
|
|
|
|
|
* even if the page has been logged, because the write did not go through
|
|
|
|
|
* shared_buffers and therefore a concurrent checkpoint may have moved the
|
|
|
|
|
* redo pointer past our xlog record. Recovery may as well remove it
|
Add new block-by-block strategy for CREATE DATABASE.
Because this strategy logs changes on a block-by-block basis, it
avoids the need to checkpoint before and after the operation.
However, because it logs each changed block individually, it might
generate a lot of extra write-ahead logging if the template database
is large. Therefore, the older strategy remains available via a new
STRATEGY parameter to CREATE DATABASE, and a corresponding --strategy
option to createdb.
Somewhat controversially, this patch assembles the list of relations
to be copied to the new database by reading the pg_class relation of
the template database. Cross-database access like this isn't normally
possible, but it can be made to work here because there can't be any
connections to the database being copied, nor can it contain any
in-doubt transactions. Even so, we have to use lower-level interfaces
than normal, since the table scan and relcache interfaces will not
work for a database to which we're not connected. The advantage of
this approach is that we do not need to rely on the filesystem to
determine what ought to be copied, but instead on PostgreSQL's own
knowledge of the database structure. This avoids, for example,
copying stray files that happen to be located in the source database
directory.
Dilip Kumar, with a fairly large number of cosmetic changes by me.
Reviewed and tested by Ashutosh Sharma, Andres Freund, John Naylor,
Greg Nancarrow, Neha Sharma. Additional feedback from Bruce Momjian,
Heikki Linnakangas, Julien Rouhaud, Adam Brusselback, Kyotaro
Horiguchi, Tomas Vondra, Andrew Dunstan, Álvaro Herrera, and others.
Discussion: http://postgr.es/m/CA+TgmoYtcdxBjLh31DLxUXHxFVMPGzrU5_T=CYCvRyFHywSBUQ@mail.gmail.com
2022-03-29 17:31:43 +02:00
|
|
|
* while replaying, for example, XLOG_DBASE_CREATE* or XLOG_TBLSPC_CREATE
|
2019-03-29 04:01:14 +01:00
|
|
|
* record. Therefore, logging is necessary even if wal_level=minimal.
|
|
|
|
|
*/
|
2019-04-30 04:28:05 +02:00
|
|
|
if (persistence == RELPERSISTENCE_UNLOGGED)
|
2019-03-29 04:01:14 +01:00
|
|
|
{
|
|
|
|
|
Assert(rel->rd_rel->relkind == RELKIND_RELATION ||
|
|
|
|
|
rel->rd_rel->relkind == RELKIND_MATVIEW ||
|
|
|
|
|
rel->rd_rel->relkind == RELKIND_TOASTVALUE);
|
2019-04-30 04:28:05 +02:00
|
|
|
smgrcreate(srel, INIT_FORKNUM, false);
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
log_smgrcreate(newrlocator, INIT_FORKNUM);
|
2019-04-30 04:28:05 +02:00
|
|
|
smgrimmedsync(srel, INIT_FORKNUM);
|
2019-03-29 04:01:14 +01:00
|
|
|
}
|
2019-04-30 04:28:05 +02:00
|
|
|
|
|
|
|
|
smgrclose(srel);
|
2019-03-29 04:01:14 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_relation_nontransactional_truncate(Relation rel)
|
|
|
|
|
{
|
|
|
|
|
RelationTruncate(rel, 0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
heapam_relation_copy_data(Relation rel, const RelFileLocator *newrlocator)
|
2019-03-29 04:01:14 +01:00
|
|
|
{
|
|
|
|
|
SMgrRelation dstrel;
|
|
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
dstrel = smgropen(*newrlocator, rel->rd_backend);
|
2019-03-29 04:01:14 +01:00
|
|
|
|
2019-04-30 04:28:05 +02:00
|
|
|
/*
|
|
|
|
|
* Since we copy the file directly without looking at the shared buffers,
|
|
|
|
|
* we'd better first flush out any pages of the source relation that are
|
|
|
|
|
* in shared buffers. We assume no new changes will be made while we are
|
|
|
|
|
* holding exclusive lock on the rel.
|
|
|
|
|
*/
|
|
|
|
|
FlushRelationBuffers(rel);
|
|
|
|
|
|
2019-03-29 04:01:14 +01:00
|
|
|
/*
|
|
|
|
|
* Create and copy all forks of the relation, and schedule unlinking of
|
|
|
|
|
* old physical files.
|
|
|
|
|
*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* NOTE: any conflict in relfilenumber value will be caught in
|
2019-03-29 04:01:14 +01:00
|
|
|
* RelationCreateStorage().
|
|
|
|
|
*/
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelationCreateStorage(*newrlocator, rel->rd_rel->relpersistence, true);
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
/* copy main fork */
|
2021-07-12 23:01:29 +02:00
|
|
|
RelationCopyStorage(RelationGetSmgr(rel), dstrel, MAIN_FORKNUM,
|
2019-03-29 04:01:14 +01:00
|
|
|
rel->rd_rel->relpersistence);
|
|
|
|
|
|
|
|
|
|
/* copy those extra forks that exist */
|
|
|
|
|
for (ForkNumber forkNum = MAIN_FORKNUM + 1;
|
|
|
|
|
forkNum <= MAX_FORKNUM; forkNum++)
|
|
|
|
|
{
|
2021-07-12 23:01:29 +02:00
|
|
|
if (smgrexists(RelationGetSmgr(rel), forkNum))
|
2019-03-29 04:01:14 +01:00
|
|
|
{
|
|
|
|
|
smgrcreate(dstrel, forkNum, false);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* WAL log creation if the relation is persistent, or this is the
|
|
|
|
|
* init fork of an unlogged relation.
|
|
|
|
|
*/
|
2021-03-23 01:22:48 +01:00
|
|
|
if (RelationIsPermanent(rel) ||
|
2019-03-29 04:01:14 +01:00
|
|
|
(rel->rd_rel->relpersistence == RELPERSISTENCE_UNLOGGED &&
|
|
|
|
|
forkNum == INIT_FORKNUM))
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
log_smgrcreate(newrlocator, forkNum);
|
2021-07-12 23:01:29 +02:00
|
|
|
RelationCopyStorage(RelationGetSmgr(rel), dstrel, forkNum,
|
2019-03-29 04:01:14 +01:00
|
|
|
rel->rd_rel->relpersistence);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/* drop old relation, and close new one */
|
|
|
|
|
RelationDropStorage(rel);
|
|
|
|
|
smgrclose(dstrel);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
|
|
|
|
|
Relation OldIndex, bool use_sort,
|
|
|
|
|
TransactionId OldestXmin,
|
2019-04-24 06:42:12 +02:00
|
|
|
TransactionId *xid_cutoff,
|
|
|
|
|
MultiXactId *multi_cutoff,
|
2019-03-29 04:01:14 +01:00
|
|
|
double *num_tuples,
|
|
|
|
|
double *tups_vacuumed,
|
|
|
|
|
double *tups_recently_dead)
|
|
|
|
|
{
|
|
|
|
|
RewriteState rwstate;
|
|
|
|
|
IndexScanDesc indexScan;
|
|
|
|
|
TableScanDesc tableScan;
|
|
|
|
|
HeapScanDesc heapScan;
|
|
|
|
|
bool is_system_catalog;
|
|
|
|
|
Tuplesortstate *tuplesort;
|
|
|
|
|
TupleDesc oldTupDesc = RelationGetDescr(OldHeap);
|
|
|
|
|
TupleDesc newTupDesc = RelationGetDescr(NewHeap);
|
|
|
|
|
TupleTableSlot *slot;
|
|
|
|
|
int natts;
|
|
|
|
|
Datum *values;
|
|
|
|
|
bool *isnull;
|
|
|
|
|
BufferHeapTupleTableSlot *hslot;
|
2020-11-27 12:16:44 +01:00
|
|
|
BlockNumber prev_cblock = InvalidBlockNumber;
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
/* Remember if it's a system catalog */
|
|
|
|
|
is_system_catalog = IsSystemRelation(OldHeap);
|
|
|
|
|
|
|
|
|
|
/*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* Valid smgr_targblock implies something already wrote to the relation.
|
|
|
|
|
* This may be harmless, but this function hasn't planned for it.
|
2019-03-29 04:01:14 +01:00
|
|
|
*/
|
|
|
|
|
Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber);
|
|
|
|
|
|
|
|
|
|
/* Preallocate values/isnull arrays */
|
|
|
|
|
natts = newTupDesc->natts;
|
|
|
|
|
values = (Datum *) palloc(natts * sizeof(Datum));
|
|
|
|
|
isnull = (bool *) palloc(natts * sizeof(bool));
|
|
|
|
|
|
|
|
|
|
/* Initialize the rewrite operation */
|
2019-04-24 06:42:12 +02:00
|
|
|
rwstate = begin_heap_rewrite(OldHeap, NewHeap, OldestXmin, *xid_cutoff,
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
*multi_cutoff);
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
/* Set up sorting if wanted */
|
|
|
|
|
if (use_sort)
|
|
|
|
|
tuplesort = tuplesort_begin_cluster(oldTupDesc, OldIndex,
|
|
|
|
|
maintenance_work_mem,
|
2022-04-04 12:24:59 +02:00
|
|
|
NULL, TUPLESORT_NONE);
|
2019-03-29 04:01:14 +01:00
|
|
|
else
|
|
|
|
|
tuplesort = NULL;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prepare to scan the OldHeap. To ensure we see recently-dead tuples
|
|
|
|
|
* that still need to be copied, we scan with SnapshotAny and use
|
|
|
|
|
* HeapTupleSatisfiesVacuum for the visibility test.
|
|
|
|
|
*/
|
|
|
|
|
if (OldIndex != NULL && !use_sort)
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
const int ci_index[] = {
|
2019-03-29 04:01:14 +01:00
|
|
|
PROGRESS_CLUSTER_PHASE,
|
|
|
|
|
PROGRESS_CLUSTER_INDEX_RELID
|
|
|
|
|
};
|
2019-03-31 05:13:56 +02:00
|
|
|
int64 ci_val[2];
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
/* Set phase and OIDOldIndex to columns */
|
|
|
|
|
ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP;
|
|
|
|
|
ci_val[1] = RelationGetRelid(OldIndex);
|
|
|
|
|
pgstat_progress_update_multi_param(2, ci_index, ci_val);
|
|
|
|
|
|
|
|
|
|
tableScan = NULL;
|
|
|
|
|
heapScan = NULL;
|
|
|
|
|
indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0);
|
|
|
|
|
index_rescan(indexScan, NULL, 0, NULL, 0);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* In scan-and-sort mode and also VACUUM FULL, set phase */
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
|
|
|
|
|
PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP);
|
|
|
|
|
|
|
|
|
|
tableScan = table_beginscan(OldHeap, SnapshotAny, 0, (ScanKey) NULL);
|
|
|
|
|
heapScan = (HeapScanDesc) tableScan;
|
|
|
|
|
indexScan = NULL;
|
|
|
|
|
|
|
|
|
|
/* Set total heap blocks */
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_TOTAL_HEAP_BLKS,
|
|
|
|
|
heapScan->rs_nblocks);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
slot = table_slot_create(OldHeap, NULL);
|
|
|
|
|
hslot = (BufferHeapTupleTableSlot *) slot;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Scan through the OldHeap, either in OldIndex order or sequentially;
|
|
|
|
|
* copy each tuple into the NewHeap, or transiently to the tuplesort
|
|
|
|
|
* module. Note that we don't bother sorting dead tuples (they won't get
|
|
|
|
|
* to the new table anyway).
|
|
|
|
|
*/
|
|
|
|
|
for (;;)
|
|
|
|
|
{
|
|
|
|
|
HeapTuple tuple;
|
|
|
|
|
Buffer buf;
|
|
|
|
|
bool isdead;
|
|
|
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
|
|
|
|
|
if (indexScan != NULL)
|
|
|
|
|
{
|
|
|
|
|
if (!index_getnext_slot(indexScan, ForwardScanDirection, slot))
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/* Since we used no scan keys, should never need to recheck */
|
|
|
|
|
if (indexScan->xs_recheck)
|
|
|
|
|
elog(ERROR, "CLUSTER does not support lossy index conditions");
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
if (!table_scan_getnextslot(tableScan, ForwardScanDirection, slot))
|
2020-11-27 12:16:44 +01:00
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* If the last pages of the scan were empty, we would go to
|
|
|
|
|
* the next phase while heap_blks_scanned != heap_blks_total.
|
|
|
|
|
* Instead, to ensure that heap_blks_scanned is equivalent to
|
|
|
|
|
* total_heap_blks after the table scan phase, this parameter
|
|
|
|
|
* is manually updated to the correct value when the table
|
|
|
|
|
* scan finishes.
|
|
|
|
|
*/
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED,
|
|
|
|
|
heapScan->rs_nblocks);
|
2019-03-29 04:01:14 +01:00
|
|
|
break;
|
2020-11-27 12:16:44 +01:00
|
|
|
}
|
2019-03-29 04:01:14 +01:00
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
/*
|
|
|
|
|
* In scan-and-sort mode and also VACUUM FULL, set heap blocks
|
|
|
|
|
* scanned
|
2020-11-27 12:16:44 +01:00
|
|
|
*
|
|
|
|
|
* Note that heapScan may start at an offset and wrap around, i.e.
|
|
|
|
|
* rs_startblock may be >0, and rs_cblock may end with a number
|
|
|
|
|
* below rs_startblock. To prevent showing this wraparound to the
|
|
|
|
|
* user, we offset rs_cblock by rs_startblock (modulo rs_nblocks).
|
2019-03-31 05:13:56 +02:00
|
|
|
*/
|
2020-11-27 12:16:44 +01:00
|
|
|
if (prev_cblock != heapScan->rs_cblock)
|
|
|
|
|
{
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED,
|
|
|
|
|
(heapScan->rs_cblock +
|
|
|
|
|
heapScan->rs_nblocks -
|
|
|
|
|
heapScan->rs_startblock
|
|
|
|
|
) % heapScan->rs_nblocks + 1);
|
|
|
|
|
prev_cblock = heapScan->rs_cblock;
|
|
|
|
|
}
|
2019-03-29 04:01:14 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
tuple = ExecFetchSlotHeapTuple(slot, false, NULL);
|
|
|
|
|
buf = hslot->buffer;
|
|
|
|
|
|
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_SHARE);
|
|
|
|
|
|
|
|
|
|
switch (HeapTupleSatisfiesVacuum(tuple, OldestXmin, buf))
|
|
|
|
|
{
|
|
|
|
|
case HEAPTUPLE_DEAD:
|
|
|
|
|
/* Definitely dead */
|
|
|
|
|
isdead = true;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_RECENTLY_DEAD:
|
|
|
|
|
*tups_recently_dead += 1;
|
2020-05-13 21:31:14 +02:00
|
|
|
/* fall through */
|
2019-03-29 04:01:14 +01:00
|
|
|
case HEAPTUPLE_LIVE:
|
|
|
|
|
/* Live or recently dead, must copy it */
|
|
|
|
|
isdead = false;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_INSERT_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Since we hold exclusive lock on the relation, normally the
|
|
|
|
|
* only way to see this is if it was inserted earlier in our
|
|
|
|
|
* own transaction. However, it can happen in system
|
|
|
|
|
* catalogs, since we tend to release write lock before commit
|
|
|
|
|
* there. Give a warning if neither case applies; but in any
|
|
|
|
|
* case we had better copy it.
|
|
|
|
|
*/
|
|
|
|
|
if (!is_system_catalog &&
|
|
|
|
|
!TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple->t_data)))
|
|
|
|
|
elog(WARNING, "concurrent insert in progress within table \"%s\"",
|
|
|
|
|
RelationGetRelationName(OldHeap));
|
|
|
|
|
/* treat as live */
|
|
|
|
|
isdead = false;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_DELETE_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Similar situation to INSERT_IN_PROGRESS case.
|
|
|
|
|
*/
|
|
|
|
|
if (!is_system_catalog &&
|
|
|
|
|
!TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(tuple->t_data)))
|
|
|
|
|
elog(WARNING, "concurrent delete in progress within table \"%s\"",
|
|
|
|
|
RelationGetRelationName(OldHeap));
|
|
|
|
|
/* treat as recently dead */
|
|
|
|
|
*tups_recently_dead += 1;
|
|
|
|
|
isdead = false;
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
|
|
|
|
|
isdead = false; /* keep compiler quiet */
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
if (isdead)
|
|
|
|
|
{
|
|
|
|
|
*tups_vacuumed += 1;
|
|
|
|
|
/* heap rewrite module still needs to see it... */
|
|
|
|
|
if (rewrite_heap_dead_tuple(rwstate, tuple))
|
|
|
|
|
{
|
|
|
|
|
/* A previous recently-dead tuple is now known dead */
|
|
|
|
|
*tups_vacuumed += 1;
|
|
|
|
|
*tups_recently_dead -= 1;
|
|
|
|
|
}
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
*num_tuples += 1;
|
|
|
|
|
if (tuplesort != NULL)
|
|
|
|
|
{
|
|
|
|
|
tuplesort_putheaptuple(tuplesort, tuple);
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
/*
|
|
|
|
|
* In scan-and-sort mode, report increase in number of tuples
|
|
|
|
|
* scanned
|
|
|
|
|
*/
|
2019-03-29 04:01:14 +01:00
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED,
|
|
|
|
|
*num_tuples);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
const int ct_index[] = {
|
2019-03-29 04:01:14 +01:00
|
|
|
PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED,
|
|
|
|
|
PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN
|
|
|
|
|
};
|
2019-03-31 05:13:56 +02:00
|
|
|
int64 ct_val[2];
|
2019-03-29 04:01:14 +01:00
|
|
|
|
|
|
|
|
reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
|
|
|
|
|
values, isnull, rwstate);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In indexscan mode and also VACUUM FULL, report increase in
|
|
|
|
|
* number of tuples scanned and written
|
|
|
|
|
*/
|
|
|
|
|
ct_val[0] = *num_tuples;
|
|
|
|
|
ct_val[1] = *num_tuples;
|
|
|
|
|
pgstat_progress_update_multi_param(2, ct_index, ct_val);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (indexScan != NULL)
|
|
|
|
|
index_endscan(indexScan);
|
|
|
|
|
if (tableScan != NULL)
|
|
|
|
|
table_endscan(tableScan);
|
|
|
|
|
if (slot)
|
|
|
|
|
ExecDropSingleTupleTableSlot(slot);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In scan-and-sort mode, complete the sort, then read out all live tuples
|
|
|
|
|
* from the tuplestore and write them to the new relation.
|
|
|
|
|
*/
|
|
|
|
|
if (tuplesort != NULL)
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
double n_tuples = 0;
|
|
|
|
|
|
2019-03-29 04:01:14 +01:00
|
|
|
/* Report that we are now sorting tuples */
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
|
|
|
|
|
PROGRESS_CLUSTER_PHASE_SORT_TUPLES);
|
|
|
|
|
|
|
|
|
|
tuplesort_performsort(tuplesort);
|
|
|
|
|
|
|
|
|
|
/* Report that we are now writing new heap */
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
|
|
|
|
|
PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP);
|
|
|
|
|
|
|
|
|
|
for (;;)
|
|
|
|
|
{
|
|
|
|
|
HeapTuple tuple;
|
|
|
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
|
|
|
|
|
tuple = tuplesort_getheaptuple(tuplesort, true);
|
|
|
|
|
if (tuple == NULL)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
n_tuples += 1;
|
|
|
|
|
reform_and_rewrite_tuple(tuple,
|
|
|
|
|
OldHeap, NewHeap,
|
|
|
|
|
values, isnull,
|
|
|
|
|
rwstate);
|
|
|
|
|
/* Report n_tuples */
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN,
|
|
|
|
|
n_tuples);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
tuplesort_end(tuplesort);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Write out any remaining tuples, and fsync if needed */
|
|
|
|
|
end_heap_rewrite(rwstate);
|
|
|
|
|
|
|
|
|
|
/* Clean up */
|
|
|
|
|
pfree(values);
|
|
|
|
|
pfree(isnull);
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-31 00:21:09 +01:00
|
|
|
static bool
|
2019-03-31 05:13:56 +02:00
|
|
|
heapam_scan_analyze_next_block(TableScanDesc scan, BlockNumber blockno,
|
2019-03-31 00:21:09 +01:00
|
|
|
BufferAccessStrategy bstrategy)
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
2019-03-31 00:21:09 +01:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We must maintain a pin on the target page's buffer to ensure that
|
|
|
|
|
* concurrent activity - e.g. HOT pruning - doesn't delete tuples out from
|
|
|
|
|
* under us. Hence, pin the page until we are done looking at it. We
|
|
|
|
|
* also choose to hold sharelock on the buffer throughout --- we could
|
|
|
|
|
* release and re-acquire sharelock for each tuple, but since we aren't
|
|
|
|
|
* doing much work per tuple, the extra lock traffic is probably better
|
|
|
|
|
* avoided.
|
|
|
|
|
*/
|
2019-03-31 05:13:56 +02:00
|
|
|
hscan->rs_cblock = blockno;
|
|
|
|
|
hscan->rs_cindex = FirstOffsetNumber;
|
|
|
|
|
hscan->rs_cbuf = ReadBufferExtended(scan->rs_rd, MAIN_FORKNUM,
|
|
|
|
|
blockno, RBM_NORMAL, bstrategy);
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
2019-03-31 00:21:09 +01:00
|
|
|
|
|
|
|
|
/* in heap all blocks can contain tuples, so always return true */
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static bool
|
2019-03-31 05:13:56 +02:00
|
|
|
heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
|
2019-03-31 00:21:09 +01:00
|
|
|
double *liverows, double *deadrows,
|
|
|
|
|
TupleTableSlot *slot)
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
2019-03-31 00:21:09 +01:00
|
|
|
Page targpage;
|
|
|
|
|
OffsetNumber maxoffset;
|
|
|
|
|
BufferHeapTupleTableSlot *hslot;
|
|
|
|
|
|
|
|
|
|
Assert(TTS_IS_BUFFERTUPLE(slot));
|
|
|
|
|
|
|
|
|
|
hslot = (BufferHeapTupleTableSlot *) slot;
|
2019-03-31 05:13:56 +02:00
|
|
|
targpage = BufferGetPage(hscan->rs_cbuf);
|
2019-03-31 00:21:09 +01:00
|
|
|
maxoffset = PageGetMaxOffsetNumber(targpage);
|
|
|
|
|
|
|
|
|
|
/* Inner loop over all tuples on the selected page */
|
2019-03-31 05:13:56 +02:00
|
|
|
for (; hscan->rs_cindex <= maxoffset; hscan->rs_cindex++)
|
2019-03-31 00:21:09 +01:00
|
|
|
{
|
|
|
|
|
ItemId itemid;
|
|
|
|
|
HeapTuple targtuple = &hslot->base.tupdata;
|
|
|
|
|
bool sample_it = false;
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
itemid = PageGetItemId(targpage, hscan->rs_cindex);
|
2019-03-31 00:21:09 +01:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We ignore unused and redirect line pointers. DEAD line pointers
|
|
|
|
|
* should be counted as dead, because we need vacuum to run to get rid
|
|
|
|
|
* of them. Note that this rule agrees with the way that
|
|
|
|
|
* heap_page_prune() counts things.
|
|
|
|
|
*/
|
|
|
|
|
if (!ItemIdIsNormal(itemid))
|
|
|
|
|
{
|
|
|
|
|
if (ItemIdIsDead(itemid))
|
|
|
|
|
*deadrows += 1;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
ItemPointerSet(&targtuple->t_self, hscan->rs_cblock, hscan->rs_cindex);
|
2019-03-31 00:21:09 +01:00
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
targtuple->t_tableOid = RelationGetRelid(scan->rs_rd);
|
2019-03-31 00:21:09 +01:00
|
|
|
targtuple->t_data = (HeapTupleHeader) PageGetItem(targpage, itemid);
|
|
|
|
|
targtuple->t_len = ItemIdGetLength(itemid);
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
switch (HeapTupleSatisfiesVacuum(targtuple, OldestXmin,
|
|
|
|
|
hscan->rs_cbuf))
|
2019-03-31 00:21:09 +01:00
|
|
|
{
|
|
|
|
|
case HEAPTUPLE_LIVE:
|
|
|
|
|
sample_it = true;
|
|
|
|
|
*liverows += 1;
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case HEAPTUPLE_DEAD:
|
|
|
|
|
case HEAPTUPLE_RECENTLY_DEAD:
|
|
|
|
|
/* Count dead and recently-dead rows */
|
|
|
|
|
*deadrows += 1;
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case HEAPTUPLE_INSERT_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert-in-progress rows are not counted. We assume that
|
|
|
|
|
* when the inserting transaction commits or aborts, it will
|
|
|
|
|
* send a stats message to increment the proper count. This
|
|
|
|
|
* works right only if that transaction ends after we finish
|
|
|
|
|
* analyzing the table; if things happen in the other order,
|
|
|
|
|
* its stats update will be overwritten by ours. However, the
|
|
|
|
|
* error will be large only if the other transaction runs long
|
|
|
|
|
* enough to insert many tuples, so assuming it will finish
|
|
|
|
|
* after us is the safer option.
|
|
|
|
|
*
|
|
|
|
|
* A special case is that the inserting transaction might be
|
|
|
|
|
* our own. In this case we should count and sample the row,
|
|
|
|
|
* to accommodate users who load a table and analyze it in one
|
|
|
|
|
* transaction. (pgstat_report_analyze has to adjust the
|
2022-04-06 22:56:06 +02:00
|
|
|
* numbers we report to the cumulative stats system to make
|
|
|
|
|
* this come out right.)
|
2019-03-31 00:21:09 +01:00
|
|
|
*/
|
|
|
|
|
if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(targtuple->t_data)))
|
|
|
|
|
{
|
|
|
|
|
sample_it = true;
|
|
|
|
|
*liverows += 1;
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case HEAPTUPLE_DELETE_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We count and sample delete-in-progress rows the same as
|
|
|
|
|
* live ones, so that the stats counters come out right if the
|
|
|
|
|
* deleting transaction commits after us, per the same
|
|
|
|
|
* reasoning given above.
|
|
|
|
|
*
|
|
|
|
|
* If the delete was done by our own transaction, however, we
|
|
|
|
|
* must count the row as dead to make pgstat_report_analyze's
|
|
|
|
|
* stats adjustments come out right. (Note: this works out
|
|
|
|
|
* properly when the row was both inserted and deleted in our
|
|
|
|
|
* xact.)
|
|
|
|
|
*
|
|
|
|
|
* The net effect of these choices is that we act as though an
|
|
|
|
|
* IN_PROGRESS transaction hasn't happened yet, except if it
|
|
|
|
|
* is our own transaction, which we assume has happened.
|
|
|
|
|
*
|
|
|
|
|
* This approach ensures that we behave sanely if we see both
|
|
|
|
|
* the pre-image and post-image rows for a row being updated
|
|
|
|
|
* by a concurrent transaction: we will sample the pre-image
|
|
|
|
|
* but not the post-image. We also get sane results if the
|
|
|
|
|
* concurrent transaction never commits.
|
|
|
|
|
*/
|
|
|
|
|
if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(targtuple->t_data)))
|
2019-06-19 00:51:04 +02:00
|
|
|
*deadrows += 1;
|
2019-03-31 00:21:09 +01:00
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
sample_it = true;
|
2019-06-19 00:51:04 +02:00
|
|
|
*liverows += 1;
|
2019-03-31 00:21:09 +01:00
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
default:
|
|
|
|
|
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (sample_it)
|
|
|
|
|
{
|
2019-03-31 05:13:56 +02:00
|
|
|
ExecStoreBufferHeapTuple(targtuple, slot, hscan->rs_cbuf);
|
|
|
|
|
hscan->rs_cindex++;
|
2019-03-31 00:21:09 +01:00
|
|
|
|
|
|
|
|
/* note that we leave the buffer locked here! */
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Now release the lock and pin on the page */
|
2019-03-31 05:13:56 +02:00
|
|
|
UnlockReleaseBuffer(hscan->rs_cbuf);
|
|
|
|
|
hscan->rs_cbuf = InvalidBuffer;
|
2019-03-31 00:21:09 +01:00
|
|
|
|
|
|
|
|
/* also prevent old slot contents from having pin on page */
|
|
|
|
|
ExecClearTuple(slot);
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
static double
|
|
|
|
|
heapam_index_build_range_scan(Relation heapRelation,
|
|
|
|
|
Relation indexRelation,
|
|
|
|
|
IndexInfo *indexInfo,
|
|
|
|
|
bool allow_sync,
|
|
|
|
|
bool anyvisible,
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
bool progress,
|
2019-03-28 03:59:06 +01:00
|
|
|
BlockNumber start_blockno,
|
|
|
|
|
BlockNumber numblocks,
|
|
|
|
|
IndexBuildCallback callback,
|
|
|
|
|
void *callback_state,
|
|
|
|
|
TableScanDesc scan)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan;
|
|
|
|
|
bool is_system_catalog;
|
|
|
|
|
bool checking_uniqueness;
|
|
|
|
|
HeapTuple heapTuple;
|
|
|
|
|
Datum values[INDEX_MAX_KEYS];
|
|
|
|
|
bool isnull[INDEX_MAX_KEYS];
|
|
|
|
|
double reltuples;
|
|
|
|
|
ExprState *predicate;
|
|
|
|
|
TupleTableSlot *slot;
|
|
|
|
|
EState *estate;
|
|
|
|
|
ExprContext *econtext;
|
|
|
|
|
Snapshot snapshot;
|
|
|
|
|
bool need_unregister_snapshot = false;
|
|
|
|
|
TransactionId OldestXmin;
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
BlockNumber previous_blkno = InvalidBlockNumber;
|
2019-03-28 03:59:06 +01:00
|
|
|
BlockNumber root_blkno = InvalidBlockNumber;
|
|
|
|
|
OffsetNumber root_offsets[MaxHeapTuplesPerPage];
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* sanity checks
|
|
|
|
|
*/
|
|
|
|
|
Assert(OidIsValid(indexRelation->rd_rel->relam));
|
|
|
|
|
|
|
|
|
|
/* Remember if it's a system catalog */
|
|
|
|
|
is_system_catalog = IsSystemRelation(heapRelation);
|
|
|
|
|
|
|
|
|
|
/* See whether we're verifying uniqueness/exclusion properties */
|
|
|
|
|
checking_uniqueness = (indexInfo->ii_Unique ||
|
|
|
|
|
indexInfo->ii_ExclusionOps != NULL);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* "Any visible" mode is not compatible with uniqueness checks; make sure
|
|
|
|
|
* only one of those is requested.
|
|
|
|
|
*/
|
|
|
|
|
Assert(!(anyvisible && checking_uniqueness));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Need an EState for evaluation of index expressions and partial-index
|
|
|
|
|
* predicates. Also a slot to hold the current tuple.
|
|
|
|
|
*/
|
|
|
|
|
estate = CreateExecutorState();
|
|
|
|
|
econtext = GetPerTupleExprContext(estate);
|
|
|
|
|
slot = table_slot_create(heapRelation, NULL);
|
|
|
|
|
|
|
|
|
|
/* Arrange for econtext's scan tuple to be the tuple under test */
|
|
|
|
|
econtext->ecxt_scantuple = slot;
|
|
|
|
|
|
|
|
|
|
/* Set up execution state for predicate, if any. */
|
|
|
|
|
predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prepare for scan of the base relation. In a normal index build, we use
|
|
|
|
|
* SnapshotAny because we must retrieve all tuples and do our own time
|
|
|
|
|
* qual checks (because we have to index RECENTLY_DEAD tuples). In a
|
|
|
|
|
* concurrent build, or during bootstrap, we take a regular MVCC snapshot
|
|
|
|
|
* and index whatever's live according to that.
|
|
|
|
|
*/
|
|
|
|
|
OldestXmin = InvalidTransactionId;
|
|
|
|
|
|
|
|
|
|
/* okay to ignore lazy VACUUMs here */
|
|
|
|
|
if (!IsBootstrapProcessingMode() && !indexInfo->ii_Concurrent)
|
snapshot scalability: Don't compute global horizons while building snapshots.
To make GetSnapshotData() more scalable, it cannot not look at at each proc's
xmin: While snapshot contents do not need to change whenever a read-only
transaction commits or a snapshot is released, a proc's xmin is modified in
those cases. The frequency of xmin modifications leads to, particularly on
higher core count systems, many cache misses inside GetSnapshotData(), despite
the data underlying a snapshot not changing. That is the most
significant source of GetSnapshotData() scaling poorly on larger systems.
Without accessing xmins, GetSnapshotData() cannot calculate accurate horizons /
thresholds as it has so far. But we don't really have to: The horizons don't
actually change that much between GetSnapshotData() calls. Nor are the horizons
actually used every time a snapshot is built.
The trick this commit introduces is to delay computation of accurate horizons
until there use and using horizon boundaries to determine whether accurate
horizons need to be computed.
The use of RecentGlobal[Data]Xmin to decide whether a row version could be
removed has been replaces with new GlobalVisTest* functions. These use two
thresholds to determine whether a row can be pruned:
1) definitely_needed, indicating that rows deleted by XIDs >= definitely_needed
are definitely still visible.
2) maybe_needed, indicating that rows deleted by XIDs < maybe_needed can
definitely be removed
GetSnapshotData() updates definitely_needed to be the xmin of the computed
snapshot.
When testing whether a row can be removed (with GlobalVisTestIsRemovableXid())
and the tested XID falls in between the two (i.e. XID >= maybe_needed && XID <
definitely_needed) the boundaries can be recomputed to be more accurate. As it
is not cheap to compute accurate boundaries, we limit the number of times that
happens in short succession. As the boundaries used by
GlobalVisTestIsRemovableXid() are never reset (with maybe_needed updated by
GetSnapshotData()), it is likely that further test can benefit from an earlier
computation of accurate horizons.
To avoid regressing performance when old_snapshot_threshold is set (as that
requires an accurate horizon to be computed), heap_page_prune_opt() doesn't
unconditionally call TransactionIdLimitedForOldSnapshots() anymore. Both the
computation of the limited horizon, and the triggering of errors (with
SetOldSnapshotThresholdTimestamp()) is now only done when necessary to remove
tuples.
This commit just removes the accesses to PGXACT->xmin from
GetSnapshotData(), but other members of PGXACT residing in the same
cache line are accessed. Therefore this in itself does not result in a
significant improvement. Subsequent commits will take advantage of the
fact that GetSnapshotData() now does not need to access xmins anymore.
Note: This contains a workaround in heap_page_prune_opt() to keep the
snapshot_too_old tests working. While that workaround is ugly, the tests
currently are not meaningful, and it seems best to address them separately.
Author: Andres Freund <andres@anarazel.de>
Reviewed-By: Robert Haas <robertmhaas@gmail.com>
Reviewed-By: Thomas Munro <thomas.munro@gmail.com>
Reviewed-By: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
2020-08-13 01:03:49 +02:00
|
|
|
OldestXmin = GetOldestNonRemovableTransactionId(heapRelation);
|
2019-03-28 03:59:06 +01:00
|
|
|
|
|
|
|
|
if (!scan)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Serial index build.
|
|
|
|
|
*
|
|
|
|
|
* Must begin our own heap scan in this case. We may also need to
|
|
|
|
|
* register a snapshot whose lifetime is under our direct control.
|
|
|
|
|
*/
|
|
|
|
|
if (!TransactionIdIsValid(OldestXmin))
|
|
|
|
|
{
|
|
|
|
|
snapshot = RegisterSnapshot(GetTransactionSnapshot());
|
|
|
|
|
need_unregister_snapshot = true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
snapshot = SnapshotAny;
|
|
|
|
|
|
|
|
|
|
scan = table_beginscan_strat(heapRelation, /* relation */
|
|
|
|
|
snapshot, /* snapshot */
|
|
|
|
|
0, /* number of keys */
|
|
|
|
|
NULL, /* scan key */
|
|
|
|
|
true, /* buffer access strategy OK */
|
|
|
|
|
allow_sync); /* syncscan OK? */
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Parallel index build.
|
|
|
|
|
*
|
|
|
|
|
* Parallel case never registers/unregisters own snapshot. Snapshot
|
|
|
|
|
* is taken from parallel heap scan, and is SnapshotAny or an MVCC
|
|
|
|
|
* snapshot, based on same criteria as serial case.
|
|
|
|
|
*/
|
|
|
|
|
Assert(!IsBootstrapProcessingMode());
|
|
|
|
|
Assert(allow_sync);
|
|
|
|
|
snapshot = scan->rs_snapshot;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
hscan = (HeapScanDesc) scan;
|
|
|
|
|
|
snapshot scalability: Don't compute global horizons while building snapshots.
To make GetSnapshotData() more scalable, it cannot not look at at each proc's
xmin: While snapshot contents do not need to change whenever a read-only
transaction commits or a snapshot is released, a proc's xmin is modified in
those cases. The frequency of xmin modifications leads to, particularly on
higher core count systems, many cache misses inside GetSnapshotData(), despite
the data underlying a snapshot not changing. That is the most
significant source of GetSnapshotData() scaling poorly on larger systems.
Without accessing xmins, GetSnapshotData() cannot calculate accurate horizons /
thresholds as it has so far. But we don't really have to: The horizons don't
actually change that much between GetSnapshotData() calls. Nor are the horizons
actually used every time a snapshot is built.
The trick this commit introduces is to delay computation of accurate horizons
until there use and using horizon boundaries to determine whether accurate
horizons need to be computed.
The use of RecentGlobal[Data]Xmin to decide whether a row version could be
removed has been replaces with new GlobalVisTest* functions. These use two
thresholds to determine whether a row can be pruned:
1) definitely_needed, indicating that rows deleted by XIDs >= definitely_needed
are definitely still visible.
2) maybe_needed, indicating that rows deleted by XIDs < maybe_needed can
definitely be removed
GetSnapshotData() updates definitely_needed to be the xmin of the computed
snapshot.
When testing whether a row can be removed (with GlobalVisTestIsRemovableXid())
and the tested XID falls in between the two (i.e. XID >= maybe_needed && XID <
definitely_needed) the boundaries can be recomputed to be more accurate. As it
is not cheap to compute accurate boundaries, we limit the number of times that
happens in short succession. As the boundaries used by
GlobalVisTestIsRemovableXid() are never reset (with maybe_needed updated by
GetSnapshotData()), it is likely that further test can benefit from an earlier
computation of accurate horizons.
To avoid regressing performance when old_snapshot_threshold is set (as that
requires an accurate horizon to be computed), heap_page_prune_opt() doesn't
unconditionally call TransactionIdLimitedForOldSnapshots() anymore. Both the
computation of the limited horizon, and the triggering of errors (with
SetOldSnapshotThresholdTimestamp()) is now only done when necessary to remove
tuples.
This commit just removes the accesses to PGXACT->xmin from
GetSnapshotData(), but other members of PGXACT residing in the same
cache line are accessed. Therefore this in itself does not result in a
significant improvement. Subsequent commits will take advantage of the
fact that GetSnapshotData() now does not need to access xmins anymore.
Note: This contains a workaround in heap_page_prune_opt() to keep the
snapshot_too_old tests working. While that workaround is ugly, the tests
currently are not meaningful, and it seems best to address them separately.
Author: Andres Freund <andres@anarazel.de>
Reviewed-By: Robert Haas <robertmhaas@gmail.com>
Reviewed-By: Thomas Munro <thomas.munro@gmail.com>
Reviewed-By: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
2020-08-13 01:03:49 +02:00
|
|
|
/*
|
|
|
|
|
* Must have called GetOldestNonRemovableTransactionId() if using
|
|
|
|
|
* SnapshotAny. Shouldn't have for an MVCC snapshot. (It's especially
|
|
|
|
|
* worth checking this for parallel builds, since ambuild routines that
|
|
|
|
|
* support parallel builds must work these details out for themselves.)
|
|
|
|
|
*/
|
|
|
|
|
Assert(snapshot == SnapshotAny || IsMVCCSnapshot(snapshot));
|
|
|
|
|
Assert(snapshot == SnapshotAny ? TransactionIdIsValid(OldestXmin) :
|
|
|
|
|
!TransactionIdIsValid(OldestXmin));
|
|
|
|
|
Assert(snapshot == SnapshotAny || !anyvisible);
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
/* Publish number of blocks to scan */
|
|
|
|
|
if (progress)
|
|
|
|
|
{
|
|
|
|
|
BlockNumber nblocks;
|
|
|
|
|
|
|
|
|
|
if (hscan->rs_base.rs_parallel != NULL)
|
|
|
|
|
{
|
|
|
|
|
ParallelBlockTableScanDesc pbscan;
|
|
|
|
|
|
|
|
|
|
pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
|
|
|
|
|
nblocks = pbscan->phs_nblocks;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
nblocks = hscan->rs_nblocks;
|
|
|
|
|
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
|
|
|
|
|
nblocks);
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
/* set our scan endpoints */
|
|
|
|
|
if (!allow_sync)
|
|
|
|
|
heap_setscanlimits(scan, start_blockno, numblocks);
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* syncscan can only be requested on whole relation */
|
|
|
|
|
Assert(start_blockno == 0);
|
|
|
|
|
Assert(numblocks == InvalidBlockNumber);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
reltuples = 0;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Scan all tuples in the base relation.
|
|
|
|
|
*/
|
|
|
|
|
while ((heapTuple = heap_getnext(scan, ForwardScanDirection)) != NULL)
|
|
|
|
|
{
|
|
|
|
|
bool tupleIsAlive;
|
|
|
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
/* Report scan progress, if asked to. */
|
|
|
|
|
if (progress)
|
|
|
|
|
{
|
|
|
|
|
BlockNumber blocks_done = heapam_scan_get_blocks_done(hscan);
|
|
|
|
|
|
|
|
|
|
if (blocks_done != previous_blkno)
|
|
|
|
|
{
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
|
|
|
|
|
blocks_done);
|
|
|
|
|
previous_blkno = blocks_done;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
/*
|
|
|
|
|
* When dealing with a HOT-chain of updated tuples, we want to index
|
|
|
|
|
* the values of the live tuple (if any), but index it under the TID
|
|
|
|
|
* of the chain's root tuple. This approach is necessary to preserve
|
|
|
|
|
* the HOT-chain structure in the heap. So we need to be able to find
|
|
|
|
|
* the root item offset for every tuple that's in a HOT-chain. When
|
|
|
|
|
* first reaching a new page of the relation, call
|
|
|
|
|
* heap_get_root_tuples() to build a map of root item offsets on the
|
|
|
|
|
* page.
|
|
|
|
|
*
|
|
|
|
|
* It might look unsafe to use this information across buffer
|
|
|
|
|
* lock/unlock. However, we hold ShareLock on the table so no
|
|
|
|
|
* ordinary insert/update/delete should occur; and we hold pin on the
|
|
|
|
|
* buffer continuously while visiting the page, so no pruning
|
|
|
|
|
* operation can occur either.
|
|
|
|
|
*
|
2020-08-13 23:33:49 +02:00
|
|
|
* In cases with only ShareUpdateExclusiveLock on the table, it's
|
|
|
|
|
* possible for some HOT tuples to appear that we didn't know about
|
|
|
|
|
* when we first read the page. To handle that case, we re-obtain the
|
|
|
|
|
* list of root offsets when a HOT tuple points to a root item that we
|
|
|
|
|
* don't know about.
|
|
|
|
|
*
|
2019-03-28 03:59:06 +01:00
|
|
|
* Also, although our opinions about tuple liveness could change while
|
|
|
|
|
* we scan the page (due to concurrent transaction commits/aborts),
|
|
|
|
|
* the chain root locations won't, so this info doesn't need to be
|
|
|
|
|
* rebuilt after waiting for another transaction.
|
|
|
|
|
*
|
|
|
|
|
* Note the implied assumption that there is no more than one live
|
|
|
|
|
* tuple per HOT-chain --- else we could create more than one index
|
|
|
|
|
* entry pointing to the same root tuple.
|
|
|
|
|
*/
|
|
|
|
|
if (hscan->rs_cblock != root_blkno)
|
|
|
|
|
{
|
|
|
|
|
Page page = BufferGetPage(hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
heap_get_root_tuples(page, root_offsets);
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
root_blkno = hscan->rs_cblock;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (snapshot == SnapshotAny)
|
|
|
|
|
{
|
|
|
|
|
/* do our own time qual check */
|
|
|
|
|
bool indexIt;
|
|
|
|
|
TransactionId xwait;
|
|
|
|
|
|
|
|
|
|
recheck:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We could possibly get away with not locking the buffer here,
|
|
|
|
|
* since caller should hold ShareLock on the relation, but let's
|
|
|
|
|
* be conservative about it. (This remark is still correct even
|
|
|
|
|
* with HOT-pruning: our pin on the buffer prevents pruning.)
|
|
|
|
|
*/
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The criteria for counting a tuple as live in this block need to
|
2019-03-31 05:13:56 +02:00
|
|
|
* match what analyze.c's heapam_scan_analyze_next_tuple() does,
|
|
|
|
|
* otherwise CREATE INDEX and ANALYZE may produce wildly different
|
|
|
|
|
* reltuples values, e.g. when there are many recently-dead
|
|
|
|
|
* tuples.
|
2019-03-28 03:59:06 +01:00
|
|
|
*/
|
|
|
|
|
switch (HeapTupleSatisfiesVacuum(heapTuple, OldestXmin,
|
|
|
|
|
hscan->rs_cbuf))
|
|
|
|
|
{
|
|
|
|
|
case HEAPTUPLE_DEAD:
|
|
|
|
|
/* Definitely dead, we can ignore it */
|
|
|
|
|
indexIt = false;
|
|
|
|
|
tupleIsAlive = false;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_LIVE:
|
|
|
|
|
/* Normal case, index and unique-check it */
|
|
|
|
|
indexIt = true;
|
|
|
|
|
tupleIsAlive = true;
|
|
|
|
|
/* Count it as live, too */
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_RECENTLY_DEAD:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If tuple is recently deleted then we must index it
|
|
|
|
|
* anyway to preserve MVCC semantics. (Pre-existing
|
|
|
|
|
* transactions could try to use the index after we finish
|
|
|
|
|
* building it, and may need to see such tuples.)
|
|
|
|
|
*
|
|
|
|
|
* However, if it was HOT-updated then we must only index
|
|
|
|
|
* the live tuple at the end of the HOT-chain. Since this
|
|
|
|
|
* breaks semantics for pre-existing snapshots, mark the
|
|
|
|
|
* index as unusable for them.
|
|
|
|
|
*
|
|
|
|
|
* We don't count recently-dead tuples in reltuples, even
|
2019-03-31 05:13:56 +02:00
|
|
|
* if we index them; see heapam_scan_analyze_next_tuple().
|
2019-03-28 03:59:06 +01:00
|
|
|
*/
|
|
|
|
|
if (HeapTupleIsHotUpdated(heapTuple))
|
|
|
|
|
{
|
|
|
|
|
indexIt = false;
|
|
|
|
|
/* mark the index as unsafe for old snapshots */
|
|
|
|
|
indexInfo->ii_BrokenHotChain = true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
indexIt = true;
|
|
|
|
|
/* In any case, exclude the tuple from unique-checking */
|
|
|
|
|
tupleIsAlive = false;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_INSERT_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In "anyvisible" mode, this tuple is visible and we
|
|
|
|
|
* don't need any further checks.
|
|
|
|
|
*/
|
|
|
|
|
if (anyvisible)
|
|
|
|
|
{
|
|
|
|
|
indexIt = true;
|
|
|
|
|
tupleIsAlive = true;
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Since caller should hold ShareLock or better, normally
|
|
|
|
|
* the only way to see this is if it was inserted earlier
|
|
|
|
|
* in our own transaction. However, it can happen in
|
|
|
|
|
* system catalogs, since we tend to release write lock
|
|
|
|
|
* before commit there. Give a warning if neither case
|
|
|
|
|
* applies.
|
|
|
|
|
*/
|
|
|
|
|
xwait = HeapTupleHeaderGetXmin(heapTuple->t_data);
|
|
|
|
|
if (!TransactionIdIsCurrentTransactionId(xwait))
|
|
|
|
|
{
|
|
|
|
|
if (!is_system_catalog)
|
|
|
|
|
elog(WARNING, "concurrent insert in progress within table \"%s\"",
|
|
|
|
|
RelationGetRelationName(heapRelation));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If we are performing uniqueness checks, indexing
|
|
|
|
|
* such a tuple could lead to a bogus uniqueness
|
|
|
|
|
* failure. In that case we wait for the inserting
|
|
|
|
|
* transaction to finish and check again.
|
|
|
|
|
*/
|
|
|
|
|
if (checking_uniqueness)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Must drop the lock on the buffer before we wait
|
|
|
|
|
*/
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
XactLockTableWait(xwait, heapRelation,
|
|
|
|
|
&heapTuple->t_self,
|
|
|
|
|
XLTW_InsertIndexUnique);
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
goto recheck;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/*
|
2019-03-31 05:13:56 +02:00
|
|
|
* For consistency with
|
|
|
|
|
* heapam_scan_analyze_next_tuple(), count
|
2019-03-28 03:59:06 +01:00
|
|
|
* HEAPTUPLE_INSERT_IN_PROGRESS tuples as live only
|
|
|
|
|
* when inserted by our own transaction.
|
|
|
|
|
*/
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We must index such tuples, since if the index build
|
|
|
|
|
* commits then they're good.
|
|
|
|
|
*/
|
|
|
|
|
indexIt = true;
|
|
|
|
|
tupleIsAlive = true;
|
|
|
|
|
break;
|
|
|
|
|
case HEAPTUPLE_DELETE_IN_PROGRESS:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* As with INSERT_IN_PROGRESS case, this is unexpected
|
|
|
|
|
* unless it's our own deletion or a system catalog; but
|
|
|
|
|
* in anyvisible mode, this tuple is visible.
|
|
|
|
|
*/
|
|
|
|
|
if (anyvisible)
|
|
|
|
|
{
|
|
|
|
|
indexIt = true;
|
|
|
|
|
tupleIsAlive = false;
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
xwait = HeapTupleHeaderGetUpdateXid(heapTuple->t_data);
|
|
|
|
|
if (!TransactionIdIsCurrentTransactionId(xwait))
|
|
|
|
|
{
|
|
|
|
|
if (!is_system_catalog)
|
|
|
|
|
elog(WARNING, "concurrent delete in progress within table \"%s\"",
|
|
|
|
|
RelationGetRelationName(heapRelation));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If we are performing uniqueness checks, assuming
|
|
|
|
|
* the tuple is dead could lead to missing a
|
|
|
|
|
* uniqueness violation. In that case we wait for the
|
|
|
|
|
* deleting transaction to finish and check again.
|
|
|
|
|
*
|
|
|
|
|
* Also, if it's a HOT-updated tuple, we should not
|
|
|
|
|
* index it but rather the live tuple at the end of
|
|
|
|
|
* the HOT-chain. However, the deleting transaction
|
|
|
|
|
* could abort, possibly leaving this tuple as live
|
|
|
|
|
* after all, in which case it has to be indexed. The
|
|
|
|
|
* only way to know what to do is to wait for the
|
|
|
|
|
* deleting transaction to finish and check again.
|
|
|
|
|
*/
|
|
|
|
|
if (checking_uniqueness ||
|
|
|
|
|
HeapTupleIsHotUpdated(heapTuple))
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Must drop the lock on the buffer before we wait
|
|
|
|
|
*/
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
XactLockTableWait(xwait, heapRelation,
|
|
|
|
|
&heapTuple->t_self,
|
|
|
|
|
XLTW_InsertIndexUnique);
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
goto recheck;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Otherwise index it but don't check for uniqueness,
|
|
|
|
|
* the same as a RECENTLY_DEAD tuple.
|
|
|
|
|
*/
|
|
|
|
|
indexIt = true;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Count HEAPTUPLE_DELETE_IN_PROGRESS tuples as live,
|
|
|
|
|
* if they were not deleted by the current
|
2019-03-31 05:13:56 +02:00
|
|
|
* transaction. That's what
|
|
|
|
|
* heapam_scan_analyze_next_tuple() does, and we want
|
|
|
|
|
* the behavior to be consistent.
|
2019-03-28 03:59:06 +01:00
|
|
|
*/
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
}
|
|
|
|
|
else if (HeapTupleIsHotUpdated(heapTuple))
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* It's a HOT-updated tuple deleted by our own xact.
|
|
|
|
|
* We can assume the deletion will commit (else the
|
|
|
|
|
* index contents don't matter), so treat the same as
|
|
|
|
|
* RECENTLY_DEAD HOT-updated tuples.
|
|
|
|
|
*/
|
|
|
|
|
indexIt = false;
|
|
|
|
|
/* mark the index as unsafe for old snapshots */
|
|
|
|
|
indexInfo->ii_BrokenHotChain = true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* It's a regular tuple deleted by our own xact. Index
|
|
|
|
|
* it, but don't check for uniqueness nor count in
|
|
|
|
|
* reltuples, the same as a RECENTLY_DEAD tuple.
|
|
|
|
|
*/
|
|
|
|
|
indexIt = true;
|
|
|
|
|
}
|
|
|
|
|
/* In any case, exclude the tuple from unique-checking */
|
|
|
|
|
tupleIsAlive = false;
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
|
|
|
|
|
indexIt = tupleIsAlive = false; /* keep compiler quiet */
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
if (!indexIt)
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* heap_getnext did the time qual check */
|
|
|
|
|
tupleIsAlive = true;
|
|
|
|
|
reltuples += 1;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
MemoryContextReset(econtext->ecxt_per_tuple_memory);
|
|
|
|
|
|
|
|
|
|
/* Set up for predicate or expression evaluation */
|
|
|
|
|
ExecStoreBufferHeapTuple(heapTuple, slot, hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In a partial index, discard tuples that don't satisfy the
|
|
|
|
|
* predicate.
|
|
|
|
|
*/
|
|
|
|
|
if (predicate != NULL)
|
|
|
|
|
{
|
|
|
|
|
if (!ExecQual(predicate, econtext))
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* For the current heap tuple, extract all the attributes we use in
|
|
|
|
|
* this index, and note which are null. This also performs evaluation
|
|
|
|
|
* of any expressions needed.
|
|
|
|
|
*/
|
|
|
|
|
FormIndexDatum(indexInfo,
|
|
|
|
|
slot,
|
|
|
|
|
estate,
|
|
|
|
|
values,
|
|
|
|
|
isnull);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* You'd think we should go ahead and build the index tuple here, but
|
|
|
|
|
* some index AMs want to do further processing on the data first. So
|
|
|
|
|
* pass the values[] and isnull[] arrays, instead.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
if (HeapTupleIsHeapOnly(heapTuple))
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* For a heap-only tuple, pretend its TID is that of the root. See
|
|
|
|
|
* src/backend/access/heap/README.HOT for discussion.
|
|
|
|
|
*/
|
2019-11-08 09:44:52 +01:00
|
|
|
ItemPointerData tid;
|
2019-03-28 03:59:06 +01:00
|
|
|
OffsetNumber offnum;
|
|
|
|
|
|
|
|
|
|
offnum = ItemPointerGetOffsetNumber(&heapTuple->t_self);
|
|
|
|
|
|
2020-08-13 23:33:49 +02:00
|
|
|
/*
|
|
|
|
|
* If a HOT tuple points to a root that we don't know about,
|
|
|
|
|
* obtain root items afresh. If that still fails, report it as
|
|
|
|
|
* corruption.
|
|
|
|
|
*/
|
|
|
|
|
if (root_offsets[offnum - 1] == InvalidOffsetNumber)
|
|
|
|
|
{
|
|
|
|
|
Page page = BufferGetPage(hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
heap_get_root_tuples(page, root_offsets);
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
if (!OffsetNumberIsValid(root_offsets[offnum - 1]))
|
|
|
|
|
ereport(ERROR,
|
|
|
|
|
(errcode(ERRCODE_DATA_CORRUPTED),
|
|
|
|
|
errmsg_internal("failed to find parent tuple for heap-only tuple at (%u,%u) in table \"%s\"",
|
|
|
|
|
ItemPointerGetBlockNumber(&heapTuple->t_self),
|
|
|
|
|
offnum,
|
|
|
|
|
RelationGetRelationName(heapRelation))));
|
|
|
|
|
|
2019-11-08 09:44:52 +01:00
|
|
|
ItemPointerSet(&tid, ItemPointerGetBlockNumber(&heapTuple->t_self),
|
|
|
|
|
root_offsets[offnum - 1]);
|
2019-03-28 03:59:06 +01:00
|
|
|
|
|
|
|
|
/* Call the AM's callback routine to process the tuple */
|
2019-11-08 09:44:52 +01:00
|
|
|
callback(indexRelation, &tid, values, isnull, tupleIsAlive,
|
2019-03-28 03:59:06 +01:00
|
|
|
callback_state);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* Call the AM's callback routine to process the tuple */
|
2019-11-08 09:44:52 +01:00
|
|
|
callback(indexRelation, &heapTuple->t_self, values, isnull,
|
|
|
|
|
tupleIsAlive, callback_state);
|
2019-03-28 03:59:06 +01:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
/* Report scan progress one last time. */
|
|
|
|
|
if (progress)
|
|
|
|
|
{
|
|
|
|
|
BlockNumber blks_done;
|
|
|
|
|
|
|
|
|
|
if (hscan->rs_base.rs_parallel != NULL)
|
|
|
|
|
{
|
|
|
|
|
ParallelBlockTableScanDesc pbscan;
|
|
|
|
|
|
|
|
|
|
pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
|
|
|
|
|
blks_done = pbscan->phs_nblocks;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
blks_done = hscan->rs_nblocks;
|
|
|
|
|
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
|
|
|
|
|
blks_done);
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
table_endscan(scan);
|
|
|
|
|
|
|
|
|
|
/* we can now forget our snapshot, if set and registered by us */
|
|
|
|
|
if (need_unregister_snapshot)
|
|
|
|
|
UnregisterSnapshot(snapshot);
|
|
|
|
|
|
|
|
|
|
ExecDropSingleTupleTableSlot(slot);
|
|
|
|
|
|
|
|
|
|
FreeExecutorState(estate);
|
|
|
|
|
|
|
|
|
|
/* These may have been pointing to the now-gone estate */
|
|
|
|
|
indexInfo->ii_ExpressionsState = NIL;
|
|
|
|
|
indexInfo->ii_PredicateState = NULL;
|
|
|
|
|
|
|
|
|
|
return reltuples;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
heapam_index_validate_scan(Relation heapRelation,
|
|
|
|
|
Relation indexRelation,
|
|
|
|
|
IndexInfo *indexInfo,
|
|
|
|
|
Snapshot snapshot,
|
2019-03-31 05:13:56 +02:00
|
|
|
ValidateIndexState *state)
|
2019-03-28 03:59:06 +01:00
|
|
|
{
|
|
|
|
|
TableScanDesc scan;
|
|
|
|
|
HeapScanDesc hscan;
|
|
|
|
|
HeapTuple heapTuple;
|
|
|
|
|
Datum values[INDEX_MAX_KEYS];
|
|
|
|
|
bool isnull[INDEX_MAX_KEYS];
|
|
|
|
|
ExprState *predicate;
|
|
|
|
|
TupleTableSlot *slot;
|
|
|
|
|
EState *estate;
|
|
|
|
|
ExprContext *econtext;
|
|
|
|
|
BlockNumber root_blkno = InvalidBlockNumber;
|
|
|
|
|
OffsetNumber root_offsets[MaxHeapTuplesPerPage];
|
|
|
|
|
bool in_index[MaxHeapTuplesPerPage];
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
BlockNumber previous_blkno = InvalidBlockNumber;
|
2019-03-28 03:59:06 +01:00
|
|
|
|
|
|
|
|
/* state variables for the merge */
|
|
|
|
|
ItemPointer indexcursor = NULL;
|
|
|
|
|
ItemPointerData decoded;
|
|
|
|
|
bool tuplesort_empty = false;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* sanity checks
|
|
|
|
|
*/
|
|
|
|
|
Assert(OidIsValid(indexRelation->rd_rel->relam));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Need an EState for evaluation of index expressions and partial-index
|
|
|
|
|
* predicates. Also a slot to hold the current tuple.
|
|
|
|
|
*/
|
|
|
|
|
estate = CreateExecutorState();
|
|
|
|
|
econtext = GetPerTupleExprContext(estate);
|
|
|
|
|
slot = MakeSingleTupleTableSlot(RelationGetDescr(heapRelation),
|
|
|
|
|
&TTSOpsHeapTuple);
|
|
|
|
|
|
|
|
|
|
/* Arrange for econtext's scan tuple to be the tuple under test */
|
|
|
|
|
econtext->ecxt_scantuple = slot;
|
|
|
|
|
|
|
|
|
|
/* Set up execution state for predicate, if any. */
|
|
|
|
|
predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prepare for scan of the base relation. We need just those tuples
|
|
|
|
|
* satisfying the passed-in reference snapshot. We must disable syncscan
|
|
|
|
|
* here, because it's critical that we read from block zero forward to
|
|
|
|
|
* match the sorted TIDs.
|
|
|
|
|
*/
|
|
|
|
|
scan = table_beginscan_strat(heapRelation, /* relation */
|
|
|
|
|
snapshot, /* snapshot */
|
|
|
|
|
0, /* number of keys */
|
|
|
|
|
NULL, /* scan key */
|
|
|
|
|
true, /* buffer access strategy OK */
|
|
|
|
|
false); /* syncscan not OK */
|
|
|
|
|
hscan = (HeapScanDesc) scan;
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
|
|
|
|
|
hscan->rs_nblocks);
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
/*
|
|
|
|
|
* Scan all tuples matching the snapshot.
|
|
|
|
|
*/
|
|
|
|
|
while ((heapTuple = heap_getnext(scan, ForwardScanDirection)) != NULL)
|
|
|
|
|
{
|
|
|
|
|
ItemPointer heapcursor = &heapTuple->t_self;
|
|
|
|
|
ItemPointerData rootTuple;
|
|
|
|
|
OffsetNumber root_offnum;
|
|
|
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
|
|
|
|
|
state->htups += 1;
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
if ((previous_blkno == InvalidBlockNumber) ||
|
|
|
|
|
(hscan->rs_cblock != previous_blkno))
|
|
|
|
|
{
|
|
|
|
|
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
|
|
|
|
|
hscan->rs_cblock);
|
|
|
|
|
previous_blkno = hscan->rs_cblock;
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
/*
|
|
|
|
|
* As commented in table_index_build_scan, we should index heap-only
|
|
|
|
|
* tuples under the TIDs of their root tuples; so when we advance onto
|
|
|
|
|
* a new heap page, build a map of root item offsets on the page.
|
|
|
|
|
*
|
|
|
|
|
* This complicates merging against the tuplesort output: we will
|
|
|
|
|
* visit the live tuples in order by their offsets, but the root
|
|
|
|
|
* offsets that we need to compare against the index contents might be
|
|
|
|
|
* ordered differently. So we might have to "look back" within the
|
|
|
|
|
* tuplesort output, but only within the current page. We handle that
|
|
|
|
|
* by keeping a bool array in_index[] showing all the
|
|
|
|
|
* already-passed-over tuplesort output TIDs of the current page. We
|
|
|
|
|
* clear that array here, when advancing onto a new heap page.
|
|
|
|
|
*/
|
|
|
|
|
if (hscan->rs_cblock != root_blkno)
|
|
|
|
|
{
|
|
|
|
|
Page page = BufferGetPage(hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
heap_get_root_tuples(page, root_offsets);
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
memset(in_index, 0, sizeof(in_index));
|
|
|
|
|
|
|
|
|
|
root_blkno = hscan->rs_cblock;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Convert actual tuple TID to root TID */
|
|
|
|
|
rootTuple = *heapcursor;
|
|
|
|
|
root_offnum = ItemPointerGetOffsetNumber(heapcursor);
|
|
|
|
|
|
|
|
|
|
if (HeapTupleIsHeapOnly(heapTuple))
|
|
|
|
|
{
|
|
|
|
|
root_offnum = root_offsets[root_offnum - 1];
|
|
|
|
|
if (!OffsetNumberIsValid(root_offnum))
|
|
|
|
|
ereport(ERROR,
|
|
|
|
|
(errcode(ERRCODE_DATA_CORRUPTED),
|
|
|
|
|
errmsg_internal("failed to find parent tuple for heap-only tuple at (%u,%u) in table \"%s\"",
|
|
|
|
|
ItemPointerGetBlockNumber(heapcursor),
|
|
|
|
|
ItemPointerGetOffsetNumber(heapcursor),
|
|
|
|
|
RelationGetRelationName(heapRelation))));
|
|
|
|
|
ItemPointerSetOffsetNumber(&rootTuple, root_offnum);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* "merge" by skipping through the index tuples until we find or pass
|
|
|
|
|
* the current root tuple.
|
|
|
|
|
*/
|
|
|
|
|
while (!tuplesort_empty &&
|
|
|
|
|
(!indexcursor ||
|
|
|
|
|
ItemPointerCompare(indexcursor, &rootTuple) < 0))
|
|
|
|
|
{
|
|
|
|
|
Datum ts_val;
|
|
|
|
|
bool ts_isnull;
|
|
|
|
|
|
|
|
|
|
if (indexcursor)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Remember index items seen earlier on the current heap page
|
|
|
|
|
*/
|
|
|
|
|
if (ItemPointerGetBlockNumber(indexcursor) == root_blkno)
|
|
|
|
|
in_index[ItemPointerGetOffsetNumber(indexcursor) - 1] = true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
tuplesort_empty = !tuplesort_getdatum(state->tuplesort, true,
|
2022-10-27 22:25:12 +02:00
|
|
|
false, &ts_val, &ts_isnull,
|
|
|
|
|
NULL);
|
2019-03-28 03:59:06 +01:00
|
|
|
Assert(tuplesort_empty || !ts_isnull);
|
|
|
|
|
if (!tuplesort_empty)
|
|
|
|
|
{
|
|
|
|
|
itemptr_decode(&decoded, DatumGetInt64(ts_val));
|
|
|
|
|
indexcursor = &decoded;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* Be tidy */
|
|
|
|
|
indexcursor = NULL;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the tuplesort has overshot *and* we didn't see a match earlier,
|
|
|
|
|
* then this tuple is missing from the index, so insert it.
|
|
|
|
|
*/
|
|
|
|
|
if ((tuplesort_empty ||
|
|
|
|
|
ItemPointerCompare(indexcursor, &rootTuple) > 0) &&
|
|
|
|
|
!in_index[root_offnum - 1])
|
|
|
|
|
{
|
|
|
|
|
MemoryContextReset(econtext->ecxt_per_tuple_memory);
|
|
|
|
|
|
|
|
|
|
/* Set up for predicate or expression evaluation */
|
|
|
|
|
ExecStoreHeapTuple(heapTuple, slot, false);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In a partial index, discard tuples that don't satisfy the
|
|
|
|
|
* predicate.
|
|
|
|
|
*/
|
|
|
|
|
if (predicate != NULL)
|
|
|
|
|
{
|
|
|
|
|
if (!ExecQual(predicate, econtext))
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* For the current heap tuple, extract all the attributes we use
|
|
|
|
|
* in this index, and note which are null. This also performs
|
|
|
|
|
* evaluation of any expressions needed.
|
|
|
|
|
*/
|
|
|
|
|
FormIndexDatum(indexInfo,
|
|
|
|
|
slot,
|
|
|
|
|
estate,
|
|
|
|
|
values,
|
|
|
|
|
isnull);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* You'd think we should go ahead and build the index tuple here,
|
|
|
|
|
* but some index AMs want to do further processing on the data
|
|
|
|
|
* first. So pass the values[] and isnull[] arrays, instead.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the tuple is already committed dead, you might think we
|
|
|
|
|
* could suppress uniqueness checking, but this is no longer true
|
|
|
|
|
* in the presence of HOT, because the insert is actually a proxy
|
|
|
|
|
* for a uniqueness check on the whole HOT-chain. That is, the
|
|
|
|
|
* tuple we have here could be dead because it was already
|
|
|
|
|
* HOT-updated, and if so the updating transaction will not have
|
|
|
|
|
* thought it should insert index entries. The index AM will
|
|
|
|
|
* check the whole HOT-chain and correctly detect a conflict if
|
|
|
|
|
* there is one.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
index_insert(indexRelation,
|
|
|
|
|
values,
|
|
|
|
|
isnull,
|
|
|
|
|
&rootTuple,
|
|
|
|
|
heapRelation,
|
|
|
|
|
indexInfo->ii_Unique ?
|
|
|
|
|
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
|
2021-01-13 17:11:00 +01:00
|
|
|
false,
|
2019-03-28 03:59:06 +01:00
|
|
|
indexInfo);
|
|
|
|
|
|
|
|
|
|
state->tups_inserted += 1;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
table_endscan(scan);
|
|
|
|
|
|
|
|
|
|
ExecDropSingleTupleTableSlot(slot);
|
|
|
|
|
|
|
|
|
|
FreeExecutorState(estate);
|
|
|
|
|
|
|
|
|
|
/* These may have been pointing to the now-gone estate */
|
|
|
|
|
indexInfo->ii_ExpressionsState = NIL;
|
|
|
|
|
indexInfo->ii_PredicateState = NULL;
|
|
|
|
|
}
|
|
|
|
|
|
Report progress of CREATE INDEX operations
This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.
There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific. The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.
The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan. (The index validation index scan requires
patching each AM, which has not been included here.)
Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
2019-04-02 20:18:08 +02:00
|
|
|
/*
|
|
|
|
|
* Return the number of blocks that have been read by this scan since
|
|
|
|
|
* starting. This is meant for progress reporting rather than be fully
|
|
|
|
|
* accurate: in a parallel scan, workers can be concurrently reading blocks
|
|
|
|
|
* further ahead than what we report.
|
|
|
|
|
*/
|
|
|
|
|
static BlockNumber
|
|
|
|
|
heapam_scan_get_blocks_done(HeapScanDesc hscan)
|
|
|
|
|
{
|
|
|
|
|
ParallelBlockTableScanDesc bpscan = NULL;
|
|
|
|
|
BlockNumber startblock;
|
|
|
|
|
BlockNumber blocks_done;
|
|
|
|
|
|
|
|
|
|
if (hscan->rs_base.rs_parallel != NULL)
|
|
|
|
|
{
|
|
|
|
|
bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
|
|
|
|
|
startblock = bpscan->phs_startblock;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
startblock = hscan->rs_startblock;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Might have wrapped around the end of the relation, if startblock was
|
|
|
|
|
* not zero.
|
|
|
|
|
*/
|
|
|
|
|
if (hscan->rs_cblock > startblock)
|
|
|
|
|
blocks_done = hscan->rs_cblock - startblock;
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
BlockNumber nblocks;
|
|
|
|
|
|
|
|
|
|
nblocks = bpscan != NULL ? bpscan->phs_nblocks : hscan->rs_nblocks;
|
|
|
|
|
blocks_done = nblocks - startblock +
|
|
|
|
|
hscan->rs_cblock;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return blocks_done;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2019-05-18 03:06:18 +02:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Miscellaneous callbacks for the heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
2019-05-21 17:57:13 +02:00
|
|
|
/*
|
|
|
|
|
* Check to see whether the table needs a TOAST table. It does only if
|
|
|
|
|
* (1) there are any toastable attributes, and (2) the maximum length
|
|
|
|
|
* of a tuple could exceed TOAST_TUPLE_THRESHOLD. (We don't want to
|
|
|
|
|
* create a toast table for something like "f1 varchar(20)".)
|
|
|
|
|
*/
|
|
|
|
|
static bool
|
|
|
|
|
heapam_relation_needs_toast_table(Relation rel)
|
|
|
|
|
{
|
|
|
|
|
int32 data_length = 0;
|
|
|
|
|
bool maxlength_unknown = false;
|
|
|
|
|
bool has_toastable_attrs = false;
|
|
|
|
|
TupleDesc tupdesc = rel->rd_att;
|
|
|
|
|
int32 tuple_length;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < tupdesc->natts; i++)
|
|
|
|
|
{
|
|
|
|
|
Form_pg_attribute att = TupleDescAttr(tupdesc, i);
|
|
|
|
|
|
|
|
|
|
if (att->attisdropped)
|
|
|
|
|
continue;
|
|
|
|
|
data_length = att_align_nominal(data_length, att->attalign);
|
|
|
|
|
if (att->attlen > 0)
|
|
|
|
|
{
|
|
|
|
|
/* Fixed-length types are never toastable */
|
|
|
|
|
data_length += att->attlen;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
int32 maxlen = type_maximum_size(att->atttypid,
|
|
|
|
|
att->atttypmod);
|
|
|
|
|
|
|
|
|
|
if (maxlen < 0)
|
|
|
|
|
maxlength_unknown = true;
|
|
|
|
|
else
|
|
|
|
|
data_length += maxlen;
|
2020-03-04 16:34:25 +01:00
|
|
|
if (att->attstorage != TYPSTORAGE_PLAIN)
|
2019-05-21 17:57:13 +02:00
|
|
|
has_toastable_attrs = true;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if (!has_toastable_attrs)
|
|
|
|
|
return false; /* nothing to toast? */
|
|
|
|
|
if (maxlength_unknown)
|
|
|
|
|
return true; /* any unlimited-length attrs? */
|
|
|
|
|
tuple_length = MAXALIGN(SizeofHeapTupleHeader +
|
|
|
|
|
BITMAPLEN(tupdesc->natts)) +
|
|
|
|
|
MAXALIGN(data_length);
|
|
|
|
|
return (tuple_length > TOAST_TUPLE_THRESHOLD);
|
|
|
|
|
}
|
|
|
|
|
|
2020-01-07 20:23:25 +01:00
|
|
|
/*
|
|
|
|
|
* TOAST tables for heap relations are just heap relations.
|
|
|
|
|
*/
|
|
|
|
|
static Oid
|
|
|
|
|
heapam_relation_toast_am(Relation rel)
|
|
|
|
|
{
|
|
|
|
|
return rel->rd_rel->relam;
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-28 03:59:06 +01:00
|
|
|
|
2019-03-31 00:40:33 +01:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Planner related callbacks for the heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
2019-07-08 20:51:53 +02:00
|
|
|
#define HEAP_OVERHEAD_BYTES_PER_TUPLE \
|
|
|
|
|
(MAXALIGN(SizeofHeapTupleHeader) + sizeof(ItemIdData))
|
|
|
|
|
#define HEAP_USABLE_BYTES_PER_PAGE \
|
|
|
|
|
(BLCKSZ - SizeOfPageHeaderData)
|
|
|
|
|
|
2019-03-31 00:40:33 +01:00
|
|
|
static void
|
|
|
|
|
heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
|
|
|
|
|
BlockNumber *pages, double *tuples,
|
|
|
|
|
double *allvisfrac)
|
|
|
|
|
{
|
2019-07-08 20:51:53 +02:00
|
|
|
table_block_relation_estimate_size(rel, attr_widths, pages,
|
|
|
|
|
tuples, allvisfrac,
|
|
|
|
|
HEAP_OVERHEAD_BYTES_PER_TUPLE,
|
|
|
|
|
HEAP_USABLE_BYTES_PER_PAGE);
|
2019-03-31 00:40:33 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2019-03-31 05:18:53 +02:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Executor related callbacks for the heap AM
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
2019-04-01 02:51:49 +02:00
|
|
|
static bool
|
|
|
|
|
heapam_scan_bitmap_next_block(TableScanDesc scan,
|
|
|
|
|
TBMIterateResult *tbmres)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
2022-11-16 16:17:18 +01:00
|
|
|
BlockNumber block = tbmres->blockno;
|
2019-04-01 02:51:49 +02:00
|
|
|
Buffer buffer;
|
|
|
|
|
Snapshot snapshot;
|
|
|
|
|
int ntup;
|
|
|
|
|
|
|
|
|
|
hscan->rs_cindex = 0;
|
|
|
|
|
hscan->rs_ntuples = 0;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Ignore any claimed entries past what we think is the end of the
|
|
|
|
|
* relation. It may have been extended after the start of our scan (we
|
|
|
|
|
* only hold an AccessShareLock, and it could be inserts from this
|
|
|
|
|
* backend).
|
|
|
|
|
*/
|
2022-11-16 16:17:18 +01:00
|
|
|
if (block >= hscan->rs_nblocks)
|
2019-04-01 02:51:49 +02:00
|
|
|
return false;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Acquire pin on the target heap page, trading in any pin we held before.
|
|
|
|
|
*/
|
|
|
|
|
hscan->rs_cbuf = ReleaseAndReadBuffer(hscan->rs_cbuf,
|
|
|
|
|
scan->rs_rd,
|
2022-11-16 16:17:18 +01:00
|
|
|
block);
|
|
|
|
|
hscan->rs_cblock = block;
|
2019-04-01 02:51:49 +02:00
|
|
|
buffer = hscan->rs_cbuf;
|
|
|
|
|
snapshot = scan->rs_snapshot;
|
|
|
|
|
|
|
|
|
|
ntup = 0;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prune and repair fragmentation for the whole page, if possible.
|
|
|
|
|
*/
|
|
|
|
|
heap_page_prune_opt(scan->rs_rd, buffer);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We must hold share lock on the buffer content while examining tuple
|
|
|
|
|
* visibility. Afterwards, however, the tuples we have found to be
|
|
|
|
|
* visible are guaranteed good as long as we hold the buffer pin.
|
|
|
|
|
*/
|
|
|
|
|
LockBuffer(buffer, BUFFER_LOCK_SHARE);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We need two separate strategies for lossy and non-lossy cases.
|
|
|
|
|
*/
|
|
|
|
|
if (tbmres->ntuples >= 0)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* Bitmap is non-lossy, so we just look through the offsets listed in
|
|
|
|
|
* tbmres; but we have to follow any HOT chain starting at each such
|
|
|
|
|
* offset.
|
|
|
|
|
*/
|
|
|
|
|
int curslot;
|
|
|
|
|
|
|
|
|
|
for (curslot = 0; curslot < tbmres->ntuples; curslot++)
|
|
|
|
|
{
|
|
|
|
|
OffsetNumber offnum = tbmres->offsets[curslot];
|
|
|
|
|
ItemPointerData tid;
|
|
|
|
|
HeapTupleData heapTuple;
|
|
|
|
|
|
2022-11-16 16:17:18 +01:00
|
|
|
ItemPointerSet(&tid, block, offnum);
|
2019-04-01 02:51:49 +02:00
|
|
|
if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot,
|
|
|
|
|
&heapTuple, NULL, true))
|
|
|
|
|
hscan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/*
|
2019-05-14 00:53:39 +02:00
|
|
|
* Bitmap is lossy, so we must examine each line pointer on the page.
|
2019-04-01 02:51:49 +02:00
|
|
|
* But we can ignore HOT chains, since we'll check each tuple anyway.
|
|
|
|
|
*/
|
2022-11-16 16:17:18 +01:00
|
|
|
Page page = BufferGetPage(buffer);
|
|
|
|
|
OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
|
2019-04-01 02:51:49 +02:00
|
|
|
OffsetNumber offnum;
|
|
|
|
|
|
|
|
|
|
for (offnum = FirstOffsetNumber; offnum <= maxoff; offnum = OffsetNumberNext(offnum))
|
|
|
|
|
{
|
|
|
|
|
ItemId lp;
|
|
|
|
|
HeapTupleData loctup;
|
|
|
|
|
bool valid;
|
|
|
|
|
|
2022-11-16 16:17:18 +01:00
|
|
|
lp = PageGetItemId(page, offnum);
|
2019-04-01 02:51:49 +02:00
|
|
|
if (!ItemIdIsNormal(lp))
|
|
|
|
|
continue;
|
2022-11-16 16:17:18 +01:00
|
|
|
loctup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
|
2019-04-01 02:51:49 +02:00
|
|
|
loctup.t_len = ItemIdGetLength(lp);
|
|
|
|
|
loctup.t_tableOid = scan->rs_rd->rd_id;
|
2022-11-16 16:17:18 +01:00
|
|
|
ItemPointerSet(&loctup.t_self, block, offnum);
|
2019-04-01 02:51:49 +02:00
|
|
|
valid = HeapTupleSatisfiesVisibility(&loctup, snapshot, buffer);
|
|
|
|
|
if (valid)
|
|
|
|
|
{
|
|
|
|
|
hscan->rs_vistuples[ntup++] = offnum;
|
2020-01-28 01:13:04 +01:00
|
|
|
PredicateLockTID(scan->rs_rd, &loctup.t_self, snapshot,
|
|
|
|
|
HeapTupleHeaderGetXmin(loctup.t_data));
|
2019-04-01 02:51:49 +02:00
|
|
|
}
|
2020-01-28 01:13:04 +01:00
|
|
|
HeapCheckForSerializableConflictOut(valid, scan->rs_rd, &loctup,
|
|
|
|
|
buffer, snapshot);
|
2019-04-01 02:51:49 +02:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
Assert(ntup <= MaxHeapTuplesPerPage);
|
|
|
|
|
hscan->rs_ntuples = ntup;
|
|
|
|
|
|
|
|
|
|
return ntup > 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static bool
|
|
|
|
|
heapam_scan_bitmap_next_tuple(TableScanDesc scan,
|
|
|
|
|
TBMIterateResult *tbmres,
|
|
|
|
|
TupleTableSlot *slot)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
|
|
|
|
OffsetNumber targoffset;
|
2022-11-16 16:17:18 +01:00
|
|
|
Page page;
|
2019-04-01 02:51:49 +02:00
|
|
|
ItemId lp;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Out of range? If so, nothing more to look at on this page
|
|
|
|
|
*/
|
|
|
|
|
if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples)
|
|
|
|
|
return false;
|
|
|
|
|
|
|
|
|
|
targoffset = hscan->rs_vistuples[hscan->rs_cindex];
|
2022-11-16 16:17:18 +01:00
|
|
|
page = BufferGetPage(hscan->rs_cbuf);
|
|
|
|
|
lp = PageGetItemId(page, targoffset);
|
2019-04-01 02:51:49 +02:00
|
|
|
Assert(ItemIdIsNormal(lp));
|
|
|
|
|
|
2022-11-16 16:17:18 +01:00
|
|
|
hscan->rs_ctup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
|
2019-04-01 02:51:49 +02:00
|
|
|
hscan->rs_ctup.t_len = ItemIdGetLength(lp);
|
|
|
|
|
hscan->rs_ctup.t_tableOid = scan->rs_rd->rd_id;
|
|
|
|
|
ItemPointerSet(&hscan->rs_ctup.t_self, hscan->rs_cblock, targoffset);
|
|
|
|
|
|
|
|
|
|
pgstat_count_heap_fetch(scan->rs_rd);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Set up the result slot to point to this tuple. Note that the slot
|
|
|
|
|
* acquires a pin on the buffer.
|
|
|
|
|
*/
|
|
|
|
|
ExecStoreBufferHeapTuple(&hscan->rs_ctup,
|
|
|
|
|
slot,
|
|
|
|
|
hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
hscan->rs_cindex++;
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-31 05:18:53 +02:00
|
|
|
static bool
|
|
|
|
|
heapam_scan_sample_next_block(TableScanDesc scan, SampleScanState *scanstate)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
|
|
|
|
TsmRoutine *tsm = scanstate->tsmroutine;
|
|
|
|
|
BlockNumber blockno;
|
|
|
|
|
|
|
|
|
|
/* return false immediately if relation is empty */
|
|
|
|
|
if (hscan->rs_nblocks == 0)
|
|
|
|
|
return false;
|
|
|
|
|
|
|
|
|
|
if (tsm->NextSampleBlock)
|
|
|
|
|
{
|
|
|
|
|
blockno = tsm->NextSampleBlock(scanstate, hscan->rs_nblocks);
|
|
|
|
|
hscan->rs_cblock = blockno;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* scanning table sequentially */
|
|
|
|
|
|
|
|
|
|
if (hscan->rs_cblock == InvalidBlockNumber)
|
|
|
|
|
{
|
|
|
|
|
Assert(!hscan->rs_inited);
|
|
|
|
|
blockno = hscan->rs_startblock;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
Assert(hscan->rs_inited);
|
|
|
|
|
|
|
|
|
|
blockno = hscan->rs_cblock + 1;
|
|
|
|
|
|
|
|
|
|
if (blockno >= hscan->rs_nblocks)
|
|
|
|
|
{
|
2019-06-03 06:44:03 +02:00
|
|
|
/* wrap to beginning of rel, might not have started at 0 */
|
2019-03-31 05:18:53 +02:00
|
|
|
blockno = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Report our new scan position for synchronization purposes.
|
|
|
|
|
*
|
|
|
|
|
* Note: we do this before checking for end of scan so that the
|
|
|
|
|
* final state of the position hint is back at the start of the
|
|
|
|
|
* rel. That's not strictly necessary, but otherwise when you run
|
|
|
|
|
* the same query multiple times the starting position would shift
|
|
|
|
|
* a little bit backwards on every invocation, which is confusing.
|
|
|
|
|
* We don't guarantee any specific ordering in general, though.
|
|
|
|
|
*/
|
Don't to predicate lock for analyze scans, refactor scan option passing.
Before this commit, when ANALYZE was run on a table and serializable
was used (either by virtue of an explicit BEGIN TRANSACTION ISOLATION
LEVEL SERIALIZABLE, or default_transaction_isolation being set to
serializable) a null pointer dereference lead to a crash.
The analyze scan doesn't need a snapshot (nor predicate locking), but
before this commit a scan only contained information about being a
bitmap or sample scan.
Refactor the option passing to the scan_begin callback to use a
bitmask instead. Alternatively we could have added a new boolean
parameter, but that seems harder to read. Even before this issue
various people (Heikki, Tom, Robert) suggested doing so.
These changes don't change the scan APIs outside of tableam. The flags
argument could be exposed, it's not necessary to fix this
problem. Also the wrapper table_beginscan* functions encapsulate most
of that complexity.
After these changes fixing the bug is trivial, just don't acquire
predicate lock for analyze style scans. That was already done for
bitmap heap scans. Add an assert that a snapshot is passed when
acquiring the predicate lock, so this kind of bug doesn't require
running with serializable.
Also add a comment about sample scans currently requiring predicate
locking the entire relation, that previously wasn't remarked upon.
Reported-By: Joe Wildish
Author: Andres Freund
Discussion:
https://postgr.es/m/4EA80A20-E9BF-49F1-9F01-5B66CAB21453@elusive.cx
https://postgr.es/m/20190411164947.nkii4gaeilt4bui7@alap3.anarazel.de
https://postgr.es/m/20190518203102.g7peu2fianukjuxm@alap3.anarazel.de
2019-05-20 00:10:28 +02:00
|
|
|
if (scan->rs_flags & SO_ALLOW_SYNC)
|
2019-03-31 05:18:53 +02:00
|
|
|
ss_report_location(scan->rs_rd, blockno);
|
|
|
|
|
|
|
|
|
|
if (blockno == hscan->rs_startblock)
|
|
|
|
|
{
|
|
|
|
|
blockno = InvalidBlockNumber;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!BlockNumberIsValid(blockno))
|
|
|
|
|
{
|
|
|
|
|
if (BufferIsValid(hscan->rs_cbuf))
|
|
|
|
|
ReleaseBuffer(hscan->rs_cbuf);
|
|
|
|
|
hscan->rs_cbuf = InvalidBuffer;
|
|
|
|
|
hscan->rs_cblock = InvalidBlockNumber;
|
|
|
|
|
hscan->rs_inited = false;
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
heapgetpage(scan, blockno);
|
|
|
|
|
hscan->rs_inited = true;
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static bool
|
|
|
|
|
heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
|
|
|
|
|
TupleTableSlot *slot)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
|
|
|
|
TsmRoutine *tsm = scanstate->tsmroutine;
|
|
|
|
|
BlockNumber blockno = hscan->rs_cblock;
|
Don't to predicate lock for analyze scans, refactor scan option passing.
Before this commit, when ANALYZE was run on a table and serializable
was used (either by virtue of an explicit BEGIN TRANSACTION ISOLATION
LEVEL SERIALIZABLE, or default_transaction_isolation being set to
serializable) a null pointer dereference lead to a crash.
The analyze scan doesn't need a snapshot (nor predicate locking), but
before this commit a scan only contained information about being a
bitmap or sample scan.
Refactor the option passing to the scan_begin callback to use a
bitmask instead. Alternatively we could have added a new boolean
parameter, but that seems harder to read. Even before this issue
various people (Heikki, Tom, Robert) suggested doing so.
These changes don't change the scan APIs outside of tableam. The flags
argument could be exposed, it's not necessary to fix this
problem. Also the wrapper table_beginscan* functions encapsulate most
of that complexity.
After these changes fixing the bug is trivial, just don't acquire
predicate lock for analyze style scans. That was already done for
bitmap heap scans. Add an assert that a snapshot is passed when
acquiring the predicate lock, so this kind of bug doesn't require
running with serializable.
Also add a comment about sample scans currently requiring predicate
locking the entire relation, that previously wasn't remarked upon.
Reported-By: Joe Wildish
Author: Andres Freund
Discussion:
https://postgr.es/m/4EA80A20-E9BF-49F1-9F01-5B66CAB21453@elusive.cx
https://postgr.es/m/20190411164947.nkii4gaeilt4bui7@alap3.anarazel.de
https://postgr.es/m/20190518203102.g7peu2fianukjuxm@alap3.anarazel.de
2019-05-20 00:10:28 +02:00
|
|
|
bool pagemode = (scan->rs_flags & SO_ALLOW_PAGEMODE) != 0;
|
2019-03-31 05:18:53 +02:00
|
|
|
|
|
|
|
|
Page page;
|
|
|
|
|
bool all_visible;
|
|
|
|
|
OffsetNumber maxoffset;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* When not using pagemode, we must lock the buffer during tuple
|
|
|
|
|
* visibility checks.
|
|
|
|
|
*/
|
|
|
|
|
if (!pagemode)
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_SHARE);
|
|
|
|
|
|
|
|
|
|
page = (Page) BufferGetPage(hscan->rs_cbuf);
|
|
|
|
|
all_visible = PageIsAllVisible(page) &&
|
|
|
|
|
!scan->rs_snapshot->takenDuringRecovery;
|
|
|
|
|
maxoffset = PageGetMaxOffsetNumber(page);
|
|
|
|
|
|
|
|
|
|
for (;;)
|
|
|
|
|
{
|
|
|
|
|
OffsetNumber tupoffset;
|
|
|
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
|
|
|
|
|
|
|
|
|
/* Ask the tablesample method which tuples to check on this page. */
|
|
|
|
|
tupoffset = tsm->NextSampleTuple(scanstate,
|
|
|
|
|
blockno,
|
|
|
|
|
maxoffset);
|
|
|
|
|
|
|
|
|
|
if (OffsetNumberIsValid(tupoffset))
|
|
|
|
|
{
|
|
|
|
|
ItemId itemid;
|
|
|
|
|
bool visible;
|
|
|
|
|
HeapTuple tuple = &(hscan->rs_ctup);
|
|
|
|
|
|
|
|
|
|
/* Skip invalid tuple pointers. */
|
|
|
|
|
itemid = PageGetItemId(page, tupoffset);
|
|
|
|
|
if (!ItemIdIsNormal(itemid))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
tuple->t_data = (HeapTupleHeader) PageGetItem(page, itemid);
|
|
|
|
|
tuple->t_len = ItemIdGetLength(itemid);
|
|
|
|
|
ItemPointerSet(&(tuple->t_self), blockno, tupoffset);
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if (all_visible)
|
|
|
|
|
visible = true;
|
|
|
|
|
else
|
|
|
|
|
visible = SampleHeapTupleVisible(scan, hscan->rs_cbuf,
|
|
|
|
|
tuple, tupoffset);
|
|
|
|
|
|
|
|
|
|
/* in pagemode, heapgetpage did this for us */
|
|
|
|
|
if (!pagemode)
|
2020-01-28 01:13:04 +01:00
|
|
|
HeapCheckForSerializableConflictOut(visible, scan->rs_rd, tuple,
|
|
|
|
|
hscan->rs_cbuf, scan->rs_snapshot);
|
2019-03-31 05:18:53 +02:00
|
|
|
|
|
|
|
|
/* Try next tuple from same page. */
|
|
|
|
|
if (!visible)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
/* Found visible tuple, return it. */
|
|
|
|
|
if (!pagemode)
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
ExecStoreBufferHeapTuple(tuple, slot, hscan->rs_cbuf);
|
|
|
|
|
|
|
|
|
|
/* Count successfully-fetched tuples as heap fetches */
|
|
|
|
|
pgstat_count_heap_getnext(scan->rs_rd);
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* If we get here, it means we've exhausted the items on this page
|
|
|
|
|
* and it's time to move to the next.
|
|
|
|
|
*/
|
|
|
|
|
if (!pagemode)
|
|
|
|
|
LockBuffer(hscan->rs_cbuf, BUFFER_LOCK_UNLOCK);
|
|
|
|
|
|
|
|
|
|
ExecClearTuple(slot);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Assert(0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
/* ----------------------------------------------------------------------------
|
|
|
|
|
* Helper functions for the above.
|
|
|
|
|
* ----------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Reconstruct and rewrite the given tuple
|
|
|
|
|
*
|
|
|
|
|
* We cannot simply copy the tuple as-is, for several reasons:
|
|
|
|
|
*
|
|
|
|
|
* 1. We'd like to squeeze out the values of any dropped columns, both
|
|
|
|
|
* to save space and to ensure we have no corner-case failures. (It's
|
|
|
|
|
* possible for example that the new table hasn't got a TOAST table
|
|
|
|
|
* and so is unable to store any large values of dropped cols.)
|
|
|
|
|
*
|
|
|
|
|
* 2. The tuple might not even be legal for the new table; this is
|
|
|
|
|
* currently only known to happen as an after-effect of ALTER TABLE
|
|
|
|
|
* SET WITHOUT OIDS.
|
|
|
|
|
*
|
|
|
|
|
* So, we must reconstruct the tuple from component Datums.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
reform_and_rewrite_tuple(HeapTuple tuple,
|
|
|
|
|
Relation OldHeap, Relation NewHeap,
|
|
|
|
|
Datum *values, bool *isnull, RewriteState rwstate)
|
|
|
|
|
{
|
|
|
|
|
TupleDesc oldTupDesc = RelationGetDescr(OldHeap);
|
|
|
|
|
TupleDesc newTupDesc = RelationGetDescr(NewHeap);
|
|
|
|
|
HeapTuple copiedTuple;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
heap_deform_tuple(tuple, oldTupDesc, values, isnull);
|
|
|
|
|
|
2021-06-14 02:25:50 +02:00
|
|
|
/* Be sure to null out any dropped columns */
|
2019-03-31 05:13:56 +02:00
|
|
|
for (i = 0; i < newTupDesc->natts; i++)
|
|
|
|
|
{
|
|
|
|
|
if (TupleDescAttr(newTupDesc, i)->attisdropped)
|
|
|
|
|
isnull[i] = true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
copiedTuple = heap_form_tuple(newTupDesc, values, isnull);
|
|
|
|
|
|
|
|
|
|
/* The heap rewrite module does the rest */
|
|
|
|
|
rewrite_heap_tuple(rwstate, tuple, copiedTuple);
|
|
|
|
|
|
|
|
|
|
heap_freetuple(copiedTuple);
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-31 05:18:53 +02:00
|
|
|
/*
|
|
|
|
|
* Check visibility of the tuple.
|
|
|
|
|
*/
|
|
|
|
|
static bool
|
|
|
|
|
SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer,
|
|
|
|
|
HeapTuple tuple,
|
|
|
|
|
OffsetNumber tupoffset)
|
|
|
|
|
{
|
|
|
|
|
HeapScanDesc hscan = (HeapScanDesc) scan;
|
|
|
|
|
|
Don't to predicate lock for analyze scans, refactor scan option passing.
Before this commit, when ANALYZE was run on a table and serializable
was used (either by virtue of an explicit BEGIN TRANSACTION ISOLATION
LEVEL SERIALIZABLE, or default_transaction_isolation being set to
serializable) a null pointer dereference lead to a crash.
The analyze scan doesn't need a snapshot (nor predicate locking), but
before this commit a scan only contained information about being a
bitmap or sample scan.
Refactor the option passing to the scan_begin callback to use a
bitmask instead. Alternatively we could have added a new boolean
parameter, but that seems harder to read. Even before this issue
various people (Heikki, Tom, Robert) suggested doing so.
These changes don't change the scan APIs outside of tableam. The flags
argument could be exposed, it's not necessary to fix this
problem. Also the wrapper table_beginscan* functions encapsulate most
of that complexity.
After these changes fixing the bug is trivial, just don't acquire
predicate lock for analyze style scans. That was already done for
bitmap heap scans. Add an assert that a snapshot is passed when
acquiring the predicate lock, so this kind of bug doesn't require
running with serializable.
Also add a comment about sample scans currently requiring predicate
locking the entire relation, that previously wasn't remarked upon.
Reported-By: Joe Wildish
Author: Andres Freund
Discussion:
https://postgr.es/m/4EA80A20-E9BF-49F1-9F01-5B66CAB21453@elusive.cx
https://postgr.es/m/20190411164947.nkii4gaeilt4bui7@alap3.anarazel.de
https://postgr.es/m/20190518203102.g7peu2fianukjuxm@alap3.anarazel.de
2019-05-20 00:10:28 +02:00
|
|
|
if (scan->rs_flags & SO_ALLOW_PAGEMODE)
|
2019-03-31 05:18:53 +02:00
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* In pageatatime mode, heapgetpage() already did visibility checks,
|
|
|
|
|
* so just look at the info it left in rs_vistuples[].
|
|
|
|
|
*
|
|
|
|
|
* We use a binary search over the known-sorted array. Note: we could
|
|
|
|
|
* save some effort if we insisted that NextSampleTuple select tuples
|
|
|
|
|
* in increasing order, but it's not clear that there would be enough
|
|
|
|
|
* gain to justify the restriction.
|
|
|
|
|
*/
|
|
|
|
|
int start = 0,
|
|
|
|
|
end = hscan->rs_ntuples - 1;
|
|
|
|
|
|
|
|
|
|
while (start <= end)
|
|
|
|
|
{
|
|
|
|
|
int mid = (start + end) / 2;
|
|
|
|
|
OffsetNumber curoffset = hscan->rs_vistuples[mid];
|
|
|
|
|
|
|
|
|
|
if (tupoffset == curoffset)
|
|
|
|
|
return true;
|
|
|
|
|
else if (tupoffset < curoffset)
|
|
|
|
|
end = mid - 1;
|
|
|
|
|
else
|
|
|
|
|
start = mid + 1;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
/* Otherwise, we have to check the tuple individually. */
|
|
|
|
|
return HeapTupleSatisfiesVisibility(tuple, scan->rs_snapshot,
|
|
|
|
|
buffer);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2019-03-31 05:13:56 +02:00
|
|
|
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
/* ------------------------------------------------------------------------
|
|
|
|
|
* Definition of the heap table access method.
|
|
|
|
|
* ------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
static const TableAmRoutine heapam_methods = {
|
|
|
|
|
.type = T_TableAmRoutine,
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
|
|
|
|
|
.slot_callbacks = heapam_slot_callbacks,
|
|
|
|
|
|
|
|
|
|
.scan_begin = heap_beginscan,
|
|
|
|
|
.scan_end = heap_endscan,
|
|
|
|
|
.scan_rescan = heap_rescan,
|
|
|
|
|
.scan_getnextslot = heap_getnextslot,
|
|
|
|
|
|
2021-02-27 10:59:36 +01:00
|
|
|
.scan_set_tidrange = heap_set_tidrange,
|
|
|
|
|
.scan_getnextslot_tidrange = heap_getnextslot_tidrange,
|
|
|
|
|
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
.parallelscan_estimate = table_block_parallelscan_estimate,
|
|
|
|
|
.parallelscan_initialize = table_block_parallelscan_initialize,
|
|
|
|
|
.parallelscan_reinitialize = table_block_parallelscan_reinitialize,
|
|
|
|
|
|
|
|
|
|
.index_fetch_begin = heapam_index_fetch_begin,
|
|
|
|
|
.index_fetch_reset = heapam_index_fetch_reset,
|
|
|
|
|
.index_fetch_end = heapam_index_fetch_end,
|
|
|
|
|
.index_fetch_tuple = heapam_index_fetch_tuple,
|
|
|
|
|
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
.tuple_insert = heapam_tuple_insert,
|
|
|
|
|
.tuple_insert_speculative = heapam_tuple_insert_speculative,
|
|
|
|
|
.tuple_complete_speculative = heapam_tuple_complete_speculative,
|
tableam: Add table_multi_insert() and revamp/speed-up COPY FROM buffering.
This adds table_multi_insert(), and converts COPY FROM, the only user
of heap_multi_insert, to it.
A simple conversion of COPY FROM use slots would have yielded a
slowdown when inserting into a partitioned table for some
workloads. Different partitions might need different slots (both slot
types and their descriptors), and dropping / creating slots when
there's constant partition changes is measurable.
Thus instead revamp the COPY FROM buffering for partitioned tables to
allow to buffer inserts into multiple tables, flushing only when
limits are reached across all partition buffers. By only dropping
slots when there've been inserts into too many different partitions,
the aforementioned overhead is gone. By allowing larger batches, even
when there are frequent partition changes, we actuall speed such cases
up significantly.
By using slots COPY of very narrow rows into unlogged / temporary
might slow down very slightly (due to the indirect function calls).
Author: David Rowley, Andres Freund, Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190327054923.t3epfuewxfqdt22e@alap3.anarazel.de
2019-04-05 00:47:19 +02:00
|
|
|
.multi_insert = heap_multi_insert,
|
tableam: Add tuple_{insert, delete, update, lock} and use.
This adds new, required, table AM callbacks for insert/delete/update
and lock_tuple. To be able to reasonably use those, the EvalPlanQual
mechanism had to be adapted, moving more logic into the AM.
Previously both delete/update/lock call-sites and the EPQ mechanism had
to have awareness of the specific tuple format to be able to fetch the
latest version of a tuple. Obviously that needs to be abstracted
away. To do so, move the logic that find the latest row version into
the AM. lock_tuple has a new flag argument,
TUPLE_LOCK_FLAG_FIND_LAST_VERSION, that forces it to lock the last
version, rather than the current one. It'd have been possible to do
so via a separate callback as well, but finding the last version
usually also necessitates locking the newest version, making it
sensible to combine the two. This replaces the previous use of
EvalPlanQualFetch(). Additionally HeapTupleUpdated, which previously
signaled either a concurrent update or delete, is now split into two,
to avoid callers needing AM specific knowledge to differentiate.
The move of finding the latest row version into tuple_lock means that
encountering a row concurrently moved into another partition will now
raise an error about "tuple to be locked" rather than "tuple to be
updated/deleted" - which is accurate, as that always happens when
locking rows. While possible slightly less helpful for users, it seems
like an acceptable trade-off.
As part of this commit HTSU_Result has been renamed to TM_Result, and
its members been expanded to differentiated between updating and
deleting. HeapUpdateFailureData has been renamed to TM_FailureData.
The interface to speculative insertion is changed so nodeModifyTable.c
does not have to set the speculative token itself anymore. Instead
there's a version of tuple_insert, tuple_insert_speculative, that
performs the speculative insertion (without requiring a flag to signal
that fact), and the speculative insertion is either made permanent
with table_complete_speculative(succeeded = true) or aborted with
succeeded = false).
Note that multi_insert is not yet routed through tableam, nor is
COPY. Changing multi_insert requires changes to copy.c that are large
enough to better be done separately.
Similarly, although simpler, CREATE TABLE AS and CREATE MATERIALIZED
VIEW are also only going to be adjusted in a later commit.
Author: Andres Freund and Haribabu Kommi
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190313003903.nwvrxi7rw3ywhdel@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-24 03:55:57 +01:00
|
|
|
.tuple_delete = heapam_tuple_delete,
|
|
|
|
|
.tuple_update = heapam_tuple_update,
|
|
|
|
|
.tuple_lock = heapam_tuple_lock,
|
|
|
|
|
|
2019-03-25 08:13:42 +01:00
|
|
|
.tuple_fetch_row_version = heapam_fetch_row_version,
|
2019-03-26 01:14:48 +01:00
|
|
|
.tuple_get_latest_tid = heap_get_latest_tid,
|
2019-05-18 03:52:01 +02:00
|
|
|
.tuple_tid_valid = heapam_tuple_tid_valid,
|
tableam: Add and use scan APIs.
Too allow table accesses to be not directly dependent on heap, several
new abstractions are needed. Specifically:
1) Heap scans need to be generalized into table scans. Do this by
introducing TableScanDesc, which will be the "base class" for
individual AMs. This contains the AM independent fields from
HeapScanDesc.
The previous heap_{beginscan,rescan,endscan} et al. have been
replaced with a table_ version.
There's no direct replacement for heap_getnext(), as that returned
a HeapTuple, which is undesirable for a other AMs. Instead there's
table_scan_getnextslot(). But note that heap_getnext() lives on,
it's still used widely to access catalog tables.
This is achieved by new scan_begin, scan_end, scan_rescan,
scan_getnextslot callbacks.
2) The portion of parallel scans that's shared between backends need
to be able to do so without the user doing per-AM work. To achieve
that new parallelscan_{estimate, initialize, reinitialize}
callbacks are introduced, which operate on a new
ParallelTableScanDesc, which again can be subclassed by AMs.
As it is likely that several AMs are going to be block oriented,
block oriented callbacks that can be shared between such AMs are
provided and used by heap. table_block_parallelscan_{estimate,
intiialize, reinitialize} as callbacks, and
table_block_parallelscan_{nextpage, init} for use in AMs. These
operate on a ParallelBlockTableScanDesc.
3) Index scans need to be able to access tables to return a tuple, and
there needs to be state across individual accesses to the heap to
store state like buffers. That's now handled by introducing a
sort-of-scan IndexFetchTable, which again is intended to be
subclassed by individual AMs (for heap IndexFetchHeap).
The relevant callbacks for an AM are index_fetch_{end, begin,
reset} to create the necessary state, and index_fetch_tuple to
retrieve an indexed tuple. Note that index_fetch_tuple
implementations need to be smarter than just blindly fetching the
tuples for AMs that have optimizations similar to heap's HOT - the
currently alive tuple in the update chain needs to be fetched if
appropriate.
Similar to table_scan_getnextslot(), it's undesirable to continue
to return HeapTuples. Thus index_fetch_heap (might want to rename
that later) now accepts a slot as an argument. Core code doesn't
have a lot of call sites performing index scans without going
through the systable_* API (in contrast to loads of heap_getnext
calls and working directly with HeapTuples).
Index scans now store the result of a search in
IndexScanDesc->xs_heaptid, rather than xs_ctup->t_self. As the
target is not generally a HeapTuple anymore that seems cleaner.
To be able to sensible adapt code to use the above, two further
callbacks have been introduced:
a) slot_callbacks returns a TupleTableSlotOps* suitable for creating
slots capable of holding a tuple of the AMs
type. table_slot_callbacks() and table_slot_create() are based
upon that, but have additional logic to deal with views, foreign
tables, etc.
While this change could have been done separately, nearly all the
call sites that needed to be adapted for the rest of this commit
also would have been needed to be adapted for
table_slot_callbacks(), making separation not worthwhile.
b) tuple_satisfies_snapshot checks whether the tuple in a slot is
currently visible according to a snapshot. That's required as a few
places now don't have a buffer + HeapTuple around, but a
slot (which in heap's case internally has that information).
Additionally a few infrastructure changes were needed:
I) SysScanDesc, as used by systable_{beginscan, getnext} et al. now
internally uses a slot to keep track of tuples. While
systable_getnext() still returns HeapTuples, and will so for the
foreseeable future, the index API (see 1) above) now only deals with
slots.
The remainder, and largest part, of this commit is then adjusting all
scans in postgres to use the new APIs.
Author: Andres Freund, Haribabu Kommi, Alvaro Herrera
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
2019-03-11 20:46:41 +01:00
|
|
|
.tuple_satisfies_snapshot = heapam_tuple_satisfies_snapshot,
|
2021-01-13 18:21:32 +01:00
|
|
|
.index_delete_tuples = heap_index_delete_tuples,
|
2019-03-28 03:59:06 +01:00
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
.relation_set_new_filelocator = heapam_relation_set_new_filelocator,
|
2019-03-29 04:01:14 +01:00
|
|
|
.relation_nontransactional_truncate = heapam_relation_nontransactional_truncate,
|
|
|
|
|
.relation_copy_data = heapam_relation_copy_data,
|
|
|
|
|
.relation_copy_for_cluster = heapam_relation_copy_for_cluster,
|
2019-03-31 00:21:09 +01:00
|
|
|
.relation_vacuum = heap_vacuum_rel,
|
|
|
|
|
.scan_analyze_next_block = heapam_scan_analyze_next_block,
|
|
|
|
|
.scan_analyze_next_tuple = heapam_scan_analyze_next_tuple,
|
2019-03-28 03:59:06 +01:00
|
|
|
.index_build_range_scan = heapam_index_build_range_scan,
|
|
|
|
|
.index_validate_scan = heapam_index_validate_scan,
|
2019-03-31 00:40:33 +01:00
|
|
|
|
2019-07-08 20:51:53 +02:00
|
|
|
.relation_size = table_block_relation_size,
|
2019-05-21 17:57:13 +02:00
|
|
|
.relation_needs_toast_table = heapam_relation_needs_toast_table,
|
2020-01-07 20:23:25 +01:00
|
|
|
.relation_toast_am = heapam_relation_toast_am,
|
2020-01-07 20:35:48 +01:00
|
|
|
.relation_fetch_toast_slice = heap_fetch_toast_slice,
|
2019-05-18 03:06:18 +02:00
|
|
|
|
2019-03-31 00:40:33 +01:00
|
|
|
.relation_estimate_size = heapam_estimate_rel_size,
|
2019-03-31 05:18:53 +02:00
|
|
|
|
2019-04-01 02:51:49 +02:00
|
|
|
.scan_bitmap_next_block = heapam_scan_bitmap_next_block,
|
|
|
|
|
.scan_bitmap_next_tuple = heapam_scan_bitmap_next_tuple,
|
2019-03-31 05:18:53 +02:00
|
|
|
.scan_sample_next_block = heapam_scan_sample_next_block,
|
|
|
|
|
.scan_sample_next_tuple = heapam_scan_sample_next_tuple
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
const TableAmRoutine *
|
|
|
|
|
GetHeapamTableAmRoutine(void)
|
|
|
|
|
{
|
|
|
|
|
return &heapam_methods;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Datum
|
|
|
|
|
heap_tableam_handler(PG_FUNCTION_ARGS)
|
|
|
|
|
{
|
|
|
|
|
PG_RETURN_POINTER(&heapam_methods);
|
|
|
|
|
}
|
