Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
CREATE TABLE bttest_a(id int8);
|
|
|
|
CREATE TABLE bttest_b(id int8);
|
2018-04-07 22:00:39 +02:00
|
|
|
CREATE TABLE bttest_multi(id int8, data int8);
|
2018-04-25 21:05:53 +02:00
|
|
|
CREATE TABLE delete_test_table (a bigint, b bigint, c bigint, d bigint);
|
|
|
|
-- Stabalize tests
|
|
|
|
ALTER TABLE bttest_a SET (autovacuum_enabled = false);
|
|
|
|
ALTER TABLE bttest_b SET (autovacuum_enabled = false);
|
|
|
|
ALTER TABLE bttest_multi SET (autovacuum_enabled = false);
|
|
|
|
ALTER TABLE delete_test_table SET (autovacuum_enabled = false);
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
INSERT INTO bttest_a SELECT * FROM generate_series(1, 100000);
|
|
|
|
INSERT INTO bttest_b SELECT * FROM generate_series(100000, 1, -1);
|
2018-04-07 22:00:39 +02:00
|
|
|
INSERT INTO bttest_multi SELECT i, i%2 FROM generate_series(1, 100000) as i;
|
2020-05-04 20:23:44 +02:00
|
|
|
CREATE INDEX bttest_a_idx ON bttest_a USING btree (id) WITH (deduplicate_items = ON);
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
CREATE INDEX bttest_b_idx ON bttest_b USING btree (id);
|
2018-04-07 22:00:39 +02:00
|
|
|
CREATE UNIQUE INDEX bttest_multi_idx ON bttest_multi
|
|
|
|
USING btree (id) INCLUDE (data);
|
2019-06-26 05:06:17 +02:00
|
|
|
CREATE ROLE regress_bttest_role;
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
-- verify permissions are checked (error due to function not callable)
|
2019-06-26 05:06:17 +02:00
|
|
|
SET ROLE regress_bttest_role;
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
SELECT bt_index_check('bttest_a_idx'::regclass);
|
|
|
|
ERROR: permission denied for function bt_index_check
|
|
|
|
SELECT bt_index_parent_check('bttest_a_idx'::regclass);
|
|
|
|
ERROR: permission denied for function bt_index_parent_check
|
|
|
|
RESET ROLE;
|
|
|
|
-- we, intentionally, don't check relation permissions - it's useful
|
|
|
|
-- to run this cluster-wide with a restricted account, and as tested
|
|
|
|
-- above explicit permission has to be granted for that.
|
2019-06-26 05:06:17 +02:00
|
|
|
GRANT EXECUTE ON FUNCTION bt_index_check(regclass) TO regress_bttest_role;
|
|
|
|
GRANT EXECUTE ON FUNCTION bt_index_parent_check(regclass) TO regress_bttest_role;
|
|
|
|
GRANT EXECUTE ON FUNCTION bt_index_check(regclass, boolean) TO regress_bttest_role;
|
|
|
|
GRANT EXECUTE ON FUNCTION bt_index_parent_check(regclass, boolean) TO regress_bttest_role;
|
|
|
|
SET ROLE regress_bttest_role;
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
SELECT bt_index_check('bttest_a_idx');
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
SELECT bt_index_parent_check('bttest_a_idx');
|
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
RESET ROLE;
|
|
|
|
-- verify plain tables are rejected (error)
|
|
|
|
SELECT bt_index_check('bttest_a');
|
|
|
|
ERROR: "bttest_a" is not an index
|
|
|
|
SELECT bt_index_parent_check('bttest_a');
|
|
|
|
ERROR: "bttest_a" is not an index
|
|
|
|
-- verify non-existing indexes are rejected (error)
|
|
|
|
SELECT bt_index_check(17);
|
|
|
|
ERROR: could not open relation with OID 17
|
|
|
|
SELECT bt_index_parent_check(17);
|
|
|
|
ERROR: could not open relation with OID 17
|
|
|
|
-- verify wrong index types are rejected (error)
|
|
|
|
BEGIN;
|
|
|
|
CREATE INDEX bttest_a_brin_idx ON bttest_a USING brin(id);
|
|
|
|
SELECT bt_index_parent_check('bttest_a_brin_idx');
|
|
|
|
ERROR: only B-Tree indexes are supported as targets for verification
|
|
|
|
DETAIL: Relation "bttest_a_brin_idx" is not a B-Tree index.
|
|
|
|
ROLLBACK;
|
|
|
|
-- normal check outside of xact
|
|
|
|
SELECT bt_index_check('bttest_a_idx');
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2018-04-01 04:52:01 +02:00
|
|
|
-- more expansive tests
|
|
|
|
SELECT bt_index_check('bttest_a_idx', true);
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
SELECT bt_index_parent_check('bttest_b_idx', true);
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
BEGIN;
|
|
|
|
SELECT bt_index_check('bttest_a_idx');
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
SELECT bt_index_parent_check('bttest_b_idx');
|
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
-- make sure we don't have any leftover locks
|
2017-03-14 21:07:38 +01:00
|
|
|
SELECT * FROM pg_locks
|
|
|
|
WHERE relation = ANY(ARRAY['bttest_a', 'bttest_a_idx', 'bttest_b', 'bttest_b_idx']::regclass[])
|
|
|
|
AND pid = pg_backend_pid();
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath
|
|
|
|
----------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-----+------+---------+----------
|
|
|
|
(0 rows)
|
|
|
|
|
|
|
|
COMMIT;
|
2020-05-04 20:23:44 +02:00
|
|
|
-- Deduplication
|
|
|
|
TRUNCATE bttest_a;
|
|
|
|
INSERT INTO bttest_a SELECT 42 FROM generate_series(1, 2000);
|
|
|
|
SELECT bt_index_check('bttest_a_idx', true);
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
-- normal check outside of xact for index with included columns
|
|
|
|
SELECT bt_index_check('bttest_multi_idx');
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2019-04-05 02:25:35 +02:00
|
|
|
-- more expansive tests for index with included columns
|
|
|
|
SELECT bt_index_parent_check('bttest_multi_idx', true, true);
|
2018-04-07 22:00:39 +02:00
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2019-04-05 02:25:35 +02:00
|
|
|
-- repeat expansive tests for index built using insertions
|
2018-04-07 22:00:39 +02:00
|
|
|
TRUNCATE bttest_multi;
|
|
|
|
INSERT INTO bttest_multi SELECT i, i%2 FROM generate_series(1, 100000) as i;
|
2019-04-05 02:25:35 +02:00
|
|
|
SELECT bt_index_parent_check('bttest_multi_idx', true, true);
|
2018-04-07 22:00:39 +02:00
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2018-04-25 21:05:53 +02:00
|
|
|
--
|
2019-03-20 18:41:36 +01:00
|
|
|
-- Test for multilevel page deletion/downlink present checks, and rootdescend
|
|
|
|
-- checks
|
2018-04-25 21:05:53 +02:00
|
|
|
--
|
|
|
|
INSERT INTO delete_test_table SELECT i, 1, 2, 3 FROM generate_series(1,80000) i;
|
|
|
|
ALTER TABLE delete_test_table ADD PRIMARY KEY (a,b,c,d);
|
Make heap TID a tiebreaker nbtree index column.
Make nbtree treat all index tuples as having a heap TID attribute.
Index searches can distinguish duplicates by heap TID, since heap TID is
always guaranteed to be unique. This general approach has numerous
benefits for performance, and is prerequisite to teaching VACUUM to
perform "retail index tuple deletion".
Naively adding a new attribute to every pivot tuple has unacceptable
overhead (it bloats internal pages), so suffix truncation of pivot
tuples is added. This will usually truncate away the "extra" heap TID
attribute from pivot tuples during a leaf page split, and may also
truncate away additional user attributes. This can increase fan-out,
especially in a multi-column index. Truncation can only occur at the
attribute granularity, which isn't particularly effective, but works
well enough for now. A future patch may add support for truncating
"within" text attributes by generating truncated key values using new
opclass infrastructure.
Only new indexes (BTREE_VERSION 4 indexes) will have insertions that
treat heap TID as a tiebreaker attribute, or will have pivot tuples
undergo suffix truncation during a leaf page split (on-disk
compatibility with versions 2 and 3 is preserved). Upgrades to version
4 cannot be performed on-the-fly, unlike upgrades from version 2 to
version 3. contrib/amcheck continues to work with version 2 and 3
indexes, while also enforcing stricter invariants when verifying version
4 indexes. These stricter invariants are the same invariants described
by "3.1.12 Sequencing" from the Lehman and Yao paper.
A later patch will enhance the logic used by nbtree to pick a split
point. This patch is likely to negatively impact performance without
smarter choices around the precise point to split leaf pages at. Making
these two mostly-distinct sets of enhancements into distinct commits
seems like it might clarify their design, even though neither commit is
particularly useful on its own.
The maximum allowed size of new tuples is reduced by an amount equal to
the space required to store an extra MAXALIGN()'d TID in a new high key
during leaf page splits. The user-facing definition of the "1/3 of a
page" restriction is already imprecise, and so does not need to be
revised. However, there should be a compatibility note in the v12
release notes.
Author: Peter Geoghegan
Reviewed-By: Heikki Linnakangas, Alexander Korotkov
Discussion: https://postgr.es/m/CAH2-WzkVb0Kom=R+88fDFb=JSxZMFvbHVC6Mn9LJ2n=X=kS-Uw@mail.gmail.com
2019-03-20 18:04:01 +01:00
|
|
|
-- Delete most entries, and vacuum, deleting internal pages and creating "fast
|
|
|
|
-- root"
|
|
|
|
DELETE FROM delete_test_table WHERE a < 79990;
|
2018-04-25 21:05:53 +02:00
|
|
|
VACUUM delete_test_table;
|
2019-04-05 02:25:35 +02:00
|
|
|
SELECT bt_index_parent_check('delete_test_table_pkey', true);
|
2018-04-25 21:05:53 +02:00
|
|
|
bt_index_parent_check
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
2019-02-07 00:54:19 +01:00
|
|
|
--
|
|
|
|
-- BUG #15597: must not assume consistent input toasting state when forming
|
|
|
|
-- tuple. Bloom filter must fingerprint normalized index tuple representation.
|
|
|
|
--
|
|
|
|
CREATE TABLE toast_bug(buggy text);
|
|
|
|
ALTER TABLE toast_bug ALTER COLUMN buggy SET STORAGE extended;
|
2020-08-01 00:34:26 +02:00
|
|
|
CREATE INDEX toasty ON toast_bug(buggy);
|
|
|
|
-- pg_attribute entry for toasty.buggy (the index) will have plain storage:
|
|
|
|
UPDATE pg_attribute SET attstorage = 'p'
|
|
|
|
WHERE attrelid = 'toasty'::regclass AND attname = 'buggy';
|
|
|
|
-- Whereas pg_attribute entry for toast_bug.buggy (the table) still has extended storage:
|
|
|
|
SELECT attstorage FROM pg_attribute
|
|
|
|
WHERE attrelid = 'toast_bug'::regclass AND attname = 'buggy';
|
|
|
|
attstorage
|
|
|
|
------------
|
|
|
|
x
|
|
|
|
(1 row)
|
|
|
|
|
2019-02-07 00:54:19 +01:00
|
|
|
-- Insert compressible heap tuple (comfortably exceeds TOAST_TUPLE_THRESHOLD):
|
|
|
|
INSERT INTO toast_bug SELECT repeat('a', 2200);
|
|
|
|
-- Should not get false positive report of corruption:
|
|
|
|
SELECT bt_index_check('toasty', true);
|
|
|
|
bt_index_check
|
|
|
|
----------------
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
Add amcheck extension to contrib.
This is the beginning of a collection of SQL-callable functions to
verify the integrity of data files. For now it only contains code to
verify B-Tree indexes.
This adds two SQL-callable functions, validating B-Tree consistency to
a varying degree. Check the, extensive, docs for details.
The goal is to later extend the coverage of the module to further
access methods, possibly including the heap. Once checks for
additional access methods exist, we'll likely add some "dispatch"
functions that cover multiple access methods.
Author: Peter Geoghegan, editorialized by Andres Freund
Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro,
Anastasia Lubennikova, Robert Haas, Amit Langote
Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com
2017-03-10 00:50:40 +01:00
|
|
|
-- cleanup
|
|
|
|
DROP TABLE bttest_a;
|
|
|
|
DROP TABLE bttest_b;
|
2018-04-07 22:00:39 +02:00
|
|
|
DROP TABLE bttest_multi;
|
2018-04-25 21:05:53 +02:00
|
|
|
DROP TABLE delete_test_table;
|
2019-02-07 00:54:19 +01:00
|
|
|
DROP TABLE toast_bug;
|
2019-06-26 05:06:17 +02:00
|
|
|
DROP OWNED BY regress_bttest_role; -- permissions
|
|
|
|
DROP ROLE regress_bttest_role;
|