rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/contrib/test_decoding/expected/toast.out

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

391 lines
26 KiB
Plaintext
Raw Normal View History

Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
-- predictability
SET synchronous_commit = on;
DROP TABLE IF EXISTS xpto;
NOTICE: table "xpto" does not exist, skipping
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
?column?
----------
init
(1 row)
CREATE SEQUENCE xpto_rand_seq START 79 INCREMENT 1499; -- portable "random"
CREATE TABLE xpto (
id serial primary key,
toasted_col1 text,
rand1 float8 DEFAULT nextval('xpto_rand_seq'),
toasted_col2 text,
rand2 float8 DEFAULT nextval('xpto_rand_seq')
);
-- uncompressed external toast data
INSERT INTO xpto (toasted_col1, toasted_col2) SELECT string_agg(g.i::text, ''), string_agg((g.i*2)::text, '') FROM generate_series(1, 2000) g(i);
-- compressed external toast data
INSERT INTO xpto (toasted_col2) SELECT repeat(string_agg(to_char(g.i, 'FM0000'), ''), 50) FROM generate_series(1, 500) g(i);
-- update of existing column
UPDATE xpto SET toasted_col1 = (SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i)) WHERE id = 1;
UPDATE xpto SET rand1 = 123.456 WHERE id = 1;
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE. The newly added ON CONFLICT clause allows to specify an alternative to raising a unique or exclusion constraint violation error when inserting. ON CONFLICT refers to constraints that can either be specified using a inference clause (by specifying the columns of a unique constraint) or by naming a unique or exclusion constraint. DO NOTHING avoids the constraint violation, without touching the pre-existing row. DO UPDATE SET ... [WHERE ...] updates the pre-existing tuple, and has access to both the tuple proposed for insertion and the existing tuple; the optional WHERE clause can be used to prevent an update from being executed. The UPDATE SET and WHERE clauses have access to the tuple proposed for insertion using the "magic" EXCLUDED alias, and to the pre-existing tuple using the table name or its alias. This feature is often referred to as upsert. This is implemented using a new infrastructure called "speculative insertion". It is an optimistic variant of regular insertion that first does a pre-check for existing tuples and then attempts an insert. If a violating tuple was inserted concurrently, the speculatively inserted tuple is deleted and a new attempt is made. If the pre-check finds a matching tuple the alternative DO NOTHING or DO UPDATE action is taken. If the insertion succeeds without detecting a conflict, the tuple is deemed inserted. To handle the possible ambiguity between the excluded alias and a table named excluded, and for convenience with long relation names, INSERT INTO now can alias its target table. Bumps catversion as stored rules change. Author: Peter Geoghegan, with significant contributions from Heikki Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes. Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs, Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
-- updating external via INSERT ... ON CONFLICT DO UPDATE
INSERT INTO xpto(id, toasted_col2) VALUES (2, 'toasted2-upsert')
ON CONFLICT (id)
DO UPDATE SET toasted_col2 = EXCLUDED.toasted_col2 || xpto.toasted_col2;
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
DELETE FROM xpto WHERE id = 1;
DROP TABLE IF EXISTS toasted_key;
NOTICE: table "toasted_key" does not exist, skipping
CREATE TABLE toasted_key (
id serial,
toasted_key text PRIMARY KEY,
toasted_col1 text,
toasted_col2 text
);
ALTER TABLE toasted_key ALTER COLUMN toasted_key SET STORAGE EXTERNAL;
ALTER TABLE toasted_key ALTER COLUMN toasted_col1 SET STORAGE EXTERNAL;
INSERT INTO toasted_key(toasted_key, toasted_col1) VALUES(repeat('1234567890', 200), repeat('9876543210', 200));
-- test update of a toasted key without changing it
UPDATE toasted_key SET toasted_col2 = toasted_col1;
-- test update of a toasted key, changing it
UPDATE toasted_key SET toasted_key = toasted_key || '1';
DELETE FROM toasted_key;
Fix decoding of MULTI_INSERTs when rows other than the last are toasted. When decoding the results of a HEAP2_MULTI_INSERT (currently only generated by COPY FROM) toast columns for all but the last tuple weren't replaced by their actual contents before being handed to the output plugin. The reassembled toast datums where disregarded after every REORDER_BUFFER_CHANGE_(INSERT|UPDATE|DELETE) which is correct for plain inserts, updates, deletes, but not multi inserts - there we generate several REORDER_BUFFER_CHANGE_INSERTs for a single xl_heap_multi_insert record. To solve the problem add a clear_toast_afterwards boolean to ReorderBufferChange's union member that's used by modifications. All row changes but multi_inserts always set that to true, but multi_insert sets it only for the last change generated. Add a regression test covering decoding of multi_inserts - there was none at all before. Backpatch to 9.4 where logical decoding was introduced. Bug found by Petr Jelinek.
2014-07-06 15:58:01 +02:00
-- Test that HEAP2_MULTI_INSERT insertions with and without toasted
-- columns are handled correctly
CREATE TABLE toasted_copy (
id int primary key, -- no default, copy didn't use to handle that with multi inserts
data text
);
ALTER TABLE toasted_copy ALTER COLUMN data SET STORAGE EXTERNAL;
\copy toasted_copy FROM STDIN
Revert "Logical decoding of sequences" This reverts a sequence of commits, implementing features related to logical decoding and replication of sequences: - 0da92dc530c9251735fc70b20cd004d9630a1266 - 80901b32913ffa59bf157a4d88284b2b3a7511d9 - b779d7d8fdae088d70da5ed9fcd8205035676df3 - d5ed9da41d96988d905b49bebb273a9b2d6e2915 - a180c2b34de0989269fdb819bff241a249bf5380 - 75b1521dae1ff1fde17fda2e30e591f2e5d64b6a - 2d2232933b02d9396113662e44dca5f120d6830e - 002c9dd97a0c874fd1693a570383e2dd38cd40d5 - 05843b1aa49df2ecc9b97c693b755bd1b6f856a9 The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch. Discussion: https://postgr.es/m/95345a19-d508-63d1-860a-f5c2f41e8d40@enterprisedb.com
2022-04-07 18:13:13 +02:00
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
substr
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
table public.xpto: INSERT: id[integer]:1 toasted_col1[text]:'1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374
COMMIT
BEGIN
table public.xpto: INSERT: id[integer]:2 toasted_col1[text]:null rand1[double precision]:3077 toasted_col2[text]:'00010002000300040005000600070008000900100011001200130014001500160017001800190020002100
COMMIT
BEGIN
table public.xpto: UPDATE: id[integer]:1 toasted_col1[text]:'1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374
COMMIT
BEGIN
table public.xpto: UPDATE: id[integer]:1 toasted_col1[text]:unchanged-toast-datum rand1[double precision]:123.456 toasted_col2[text]:unchanged-toast-datum rand2[double precision]:1578
COMMIT
BEGIN
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE. The newly added ON CONFLICT clause allows to specify an alternative to raising a unique or exclusion constraint violation error when inserting. ON CONFLICT refers to constraints that can either be specified using a inference clause (by specifying the columns of a unique constraint) or by naming a unique or exclusion constraint. DO NOTHING avoids the constraint violation, without touching the pre-existing row. DO UPDATE SET ... [WHERE ...] updates the pre-existing tuple, and has access to both the tuple proposed for insertion and the existing tuple; the optional WHERE clause can be used to prevent an update from being executed. The UPDATE SET and WHERE clauses have access to the tuple proposed for insertion using the "magic" EXCLUDED alias, and to the pre-existing tuple using the table name or its alias. This feature is often referred to as upsert. This is implemented using a new infrastructure called "speculative insertion". It is an optimistic variant of regular insertion that first does a pre-check for existing tuples and then attempts an insert. If a violating tuple was inserted concurrently, the speculatively inserted tuple is deleted and a new attempt is made. If the pre-check finds a matching tuple the alternative DO NOTHING or DO UPDATE action is taken. If the insertion succeeds without detecting a conflict, the tuple is deemed inserted. To handle the possible ambiguity between the excluded alias and a table named excluded, and for convenience with long relation names, INSERT INTO now can alias its target table. Bumps catversion as stored rules change. Author: Peter Geoghegan, with significant contributions from Heikki Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes. Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs, Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
table public.xpto: UPDATE: id[integer]:2 toasted_col1[text]:null rand1[double precision]:3077 toasted_col2[text]:'toasted2-upsert00010002000300040005000600070008000900100011001200130014001500160017001
COMMIT
BEGIN
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
table public.xpto: DELETE: id[integer]:1
COMMIT
BEGIN
table public.toasted_key: INSERT: id[integer]:1 toasted_key[text]:'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123
COMMIT
BEGIN
WAL log unchanged toasted replica identity key attributes. Currently, during UPDATE, the unchanged replica identity key attributes are not logged separately because they are getting logged as part of the new tuple. But if they are stored externally then the untoasted values are not getting logged as part of the new tuple and logical replication won't be able to replicate such UPDATEs. So we need to log such attributes as part of the old_key_tuple during UPDATE. Reported-by: Haiying Tang Author: Dilip Kumar and Amit Kapila Reviewed-by: Alvaro Herrera, Haiying Tang, Andres Freund Backpatch-through: 10 Discussion: https://postgr.es/m/OS0PR01MB611342D0A92D4F4BF26C0F47FB229@OS0PR01MB6113.jpnprd01.prod.outlook.com
2022-02-14 04:25:58 +01:00
table public.toasted_key: UPDATE: old-key: toasted_key[text]:'123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
COMMIT
BEGIN
table public.toasted_key: UPDATE: old-key: toasted_key[text]:'123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
COMMIT
BEGIN
table public.toasted_key: DELETE: toasted_key[text]:'123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567
COMMIT
Fix decoding of MULTI_INSERTs when rows other than the last are toasted. When decoding the results of a HEAP2_MULTI_INSERT (currently only generated by COPY FROM) toast columns for all but the last tuple weren't replaced by their actual contents before being handed to the output plugin. The reassembled toast datums where disregarded after every REORDER_BUFFER_CHANGE_(INSERT|UPDATE|DELETE) which is correct for plain inserts, updates, deletes, but not multi inserts - there we generate several REORDER_BUFFER_CHANGE_INSERTs for a single xl_heap_multi_insert record. To solve the problem add a clear_toast_afterwards boolean to ReorderBufferChange's union member that's used by modifications. All row changes but multi_inserts always set that to true, but multi_insert sets it only for the last change generated. Add a regression test covering decoding of multi_inserts - there was none at all before. Backpatch to 9.4 where logical decoding was introduced. Bug found by Petr Jelinek.
2014-07-06 15:58:01 +02:00
BEGIN
table public.toasted_copy: INSERT: id[integer]:1 data[text]:'untoasted1'
table public.toasted_copy: INSERT: id[integer]:2 data[text]:'toasted1-1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
table public.toasted_copy: INSERT: id[integer]:3 data[text]:'untoasted2'
table public.toasted_copy: INSERT: id[integer]:4 data[text]:'toasted2-1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
Fix decoding of consecutive MULTI_INSERTs emitted by one heap_multi_insert(). Commit 1b86c81d2d fixed the decoding of toasted columns for the rows contained in one xl_heap_multi_insert record. But that's not actually enough, because heap_multi_insert() will actually first toast all passed in rows and then emit several *_multi_insert records; one for each page it fills with tuples. Add a XLOG_HEAP_LAST_MULTI_INSERT flag which is set in xl_heap_multi_insert->flag denoting that this multi_insert record is the last emitted by one heap_multi_insert() call. Then use that flag in decode.c to only set clear_toast_afterwards in the right situation. Expand the number of rows inserted via COPY in the corresponding regression test to make sure that more than one heap page is filled with tuples by one heap_multi_insert() call. Backpatch to 9.4 like the previous commit.
2014-07-12 14:28:19 +02:00
table public.toasted_copy: INSERT: id[integer]:5 data[text]:'untoasted3'
table public.toasted_copy: INSERT: id[integer]:6 data[text]:'untoasted4'
table public.toasted_copy: INSERT: id[integer]:7 data[text]:'untoasted5'
table public.toasted_copy: INSERT: id[integer]:8 data[text]:'untoasted6'
table public.toasted_copy: INSERT: id[integer]:9 data[text]:'untoasted7'
table public.toasted_copy: INSERT: id[integer]:10 data[text]:'untoasted8'
table public.toasted_copy: INSERT: id[integer]:11 data[text]:'untoasted9'
table public.toasted_copy: INSERT: id[integer]:12 data[text]:'untoasted10'
table public.toasted_copy: INSERT: id[integer]:13 data[text]:'untoasted11'
table public.toasted_copy: INSERT: id[integer]:14 data[text]:'untoasted12'
table public.toasted_copy: INSERT: id[integer]:15 data[text]:'untoasted13'
table public.toasted_copy: INSERT: id[integer]:16 data[text]:'untoasted14'
table public.toasted_copy: INSERT: id[integer]:17 data[text]:'untoasted15'
table public.toasted_copy: INSERT: id[integer]:18 data[text]:'untoasted16'
table public.toasted_copy: INSERT: id[integer]:19 data[text]:'untoasted17'
table public.toasted_copy: INSERT: id[integer]:20 data[text]:'untoasted18'
table public.toasted_copy: INSERT: id[integer]:21 data[text]:'untoasted19'
table public.toasted_copy: INSERT: id[integer]:22 data[text]:'untoasted20'
table public.toasted_copy: INSERT: id[integer]:23 data[text]:'untoasted21'
table public.toasted_copy: INSERT: id[integer]:24 data[text]:'untoasted22'
table public.toasted_copy: INSERT: id[integer]:25 data[text]:'untoasted23'
table public.toasted_copy: INSERT: id[integer]:26 data[text]:'untoasted24'
table public.toasted_copy: INSERT: id[integer]:27 data[text]:'untoasted25'
table public.toasted_copy: INSERT: id[integer]:28 data[text]:'untoasted26'
table public.toasted_copy: INSERT: id[integer]:29 data[text]:'untoasted27'
table public.toasted_copy: INSERT: id[integer]:30 data[text]:'untoasted28'
table public.toasted_copy: INSERT: id[integer]:31 data[text]:'untoasted29'
table public.toasted_copy: INSERT: id[integer]:32 data[text]:'untoasted30'
table public.toasted_copy: INSERT: id[integer]:33 data[text]:'untoasted31'
table public.toasted_copy: INSERT: id[integer]:34 data[text]:'untoasted32'
table public.toasted_copy: INSERT: id[integer]:35 data[text]:'untoasted33'
table public.toasted_copy: INSERT: id[integer]:36 data[text]:'untoasted34'
table public.toasted_copy: INSERT: id[integer]:37 data[text]:'untoasted35'
table public.toasted_copy: INSERT: id[integer]:38 data[text]:'untoasted36'
table public.toasted_copy: INSERT: id[integer]:39 data[text]:'untoasted37'
table public.toasted_copy: INSERT: id[integer]:40 data[text]:'untoasted38'
table public.toasted_copy: INSERT: id[integer]:41 data[text]:'untoasted39'
table public.toasted_copy: INSERT: id[integer]:42 data[text]:'untoasted40'
table public.toasted_copy: INSERT: id[integer]:43 data[text]:'untoasted41'
table public.toasted_copy: INSERT: id[integer]:44 data[text]:'untoasted42'
table public.toasted_copy: INSERT: id[integer]:45 data[text]:'untoasted43'
table public.toasted_copy: INSERT: id[integer]:46 data[text]:'untoasted44'
table public.toasted_copy: INSERT: id[integer]:47 data[text]:'untoasted45'
table public.toasted_copy: INSERT: id[integer]:48 data[text]:'untoasted46'
table public.toasted_copy: INSERT: id[integer]:49 data[text]:'untoasted47'
table public.toasted_copy: INSERT: id[integer]:50 data[text]:'untoasted48'
table public.toasted_copy: INSERT: id[integer]:51 data[text]:'untoasted49'
table public.toasted_copy: INSERT: id[integer]:52 data[text]:'untoasted50'
table public.toasted_copy: INSERT: id[integer]:53 data[text]:'untoasted51'
table public.toasted_copy: INSERT: id[integer]:54 data[text]:'untoasted52'
table public.toasted_copy: INSERT: id[integer]:55 data[text]:'untoasted53'
table public.toasted_copy: INSERT: id[integer]:56 data[text]:'untoasted54'
table public.toasted_copy: INSERT: id[integer]:57 data[text]:'untoasted55'
table public.toasted_copy: INSERT: id[integer]:58 data[text]:'untoasted56'
table public.toasted_copy: INSERT: id[integer]:59 data[text]:'untoasted57'
table public.toasted_copy: INSERT: id[integer]:60 data[text]:'untoasted58'
table public.toasted_copy: INSERT: id[integer]:61 data[text]:'untoasted59'
table public.toasted_copy: INSERT: id[integer]:62 data[text]:'untoasted60'
table public.toasted_copy: INSERT: id[integer]:63 data[text]:'untoasted61'
table public.toasted_copy: INSERT: id[integer]:64 data[text]:'untoasted62'
table public.toasted_copy: INSERT: id[integer]:65 data[text]:'untoasted63'
table public.toasted_copy: INSERT: id[integer]:66 data[text]:'untoasted64'
table public.toasted_copy: INSERT: id[integer]:67 data[text]:'untoasted65'
table public.toasted_copy: INSERT: id[integer]:68 data[text]:'untoasted66'
table public.toasted_copy: INSERT: id[integer]:69 data[text]:'untoasted67'
table public.toasted_copy: INSERT: id[integer]:70 data[text]:'untoasted68'
table public.toasted_copy: INSERT: id[integer]:71 data[text]:'untoasted69'
table public.toasted_copy: INSERT: id[integer]:72 data[text]:'untoasted70'
table public.toasted_copy: INSERT: id[integer]:73 data[text]:'untoasted71'
table public.toasted_copy: INSERT: id[integer]:74 data[text]:'untoasted72'
table public.toasted_copy: INSERT: id[integer]:75 data[text]:'untoasted73'
table public.toasted_copy: INSERT: id[integer]:76 data[text]:'untoasted74'
table public.toasted_copy: INSERT: id[integer]:77 data[text]:'untoasted75'
table public.toasted_copy: INSERT: id[integer]:78 data[text]:'untoasted76'
table public.toasted_copy: INSERT: id[integer]:79 data[text]:'untoasted77'
table public.toasted_copy: INSERT: id[integer]:80 data[text]:'untoasted78'
table public.toasted_copy: INSERT: id[integer]:81 data[text]:'untoasted79'
table public.toasted_copy: INSERT: id[integer]:82 data[text]:'untoasted80'
table public.toasted_copy: INSERT: id[integer]:83 data[text]:'untoasted81'
table public.toasted_copy: INSERT: id[integer]:84 data[text]:'untoasted82'
table public.toasted_copy: INSERT: id[integer]:85 data[text]:'untoasted83'
table public.toasted_copy: INSERT: id[integer]:86 data[text]:'untoasted84'
table public.toasted_copy: INSERT: id[integer]:87 data[text]:'untoasted85'
table public.toasted_copy: INSERT: id[integer]:88 data[text]:'untoasted86'
table public.toasted_copy: INSERT: id[integer]:89 data[text]:'untoasted87'
table public.toasted_copy: INSERT: id[integer]:90 data[text]:'untoasted88'
table public.toasted_copy: INSERT: id[integer]:91 data[text]:'untoasted89'
table public.toasted_copy: INSERT: id[integer]:92 data[text]:'untoasted90'
table public.toasted_copy: INSERT: id[integer]:93 data[text]:'untoasted91'
table public.toasted_copy: INSERT: id[integer]:94 data[text]:'untoasted92'
table public.toasted_copy: INSERT: id[integer]:95 data[text]:'untoasted93'
table public.toasted_copy: INSERT: id[integer]:96 data[text]:'untoasted94'
table public.toasted_copy: INSERT: id[integer]:97 data[text]:'untoasted95'
table public.toasted_copy: INSERT: id[integer]:98 data[text]:'untoasted96'
table public.toasted_copy: INSERT: id[integer]:99 data[text]:'untoasted97'
table public.toasted_copy: INSERT: id[integer]:100 data[text]:'untoasted98'
table public.toasted_copy: INSERT: id[integer]:101 data[text]:'untoasted99'
table public.toasted_copy: INSERT: id[integer]:102 data[text]:'untoasted100'
table public.toasted_copy: INSERT: id[integer]:103 data[text]:'untoasted101'
table public.toasted_copy: INSERT: id[integer]:104 data[text]:'untoasted102'
table public.toasted_copy: INSERT: id[integer]:105 data[text]:'untoasted103'
table public.toasted_copy: INSERT: id[integer]:106 data[text]:'untoasted104'
table public.toasted_copy: INSERT: id[integer]:107 data[text]:'untoasted105'
table public.toasted_copy: INSERT: id[integer]:108 data[text]:'untoasted106'
table public.toasted_copy: INSERT: id[integer]:109 data[text]:'untoasted107'
table public.toasted_copy: INSERT: id[integer]:110 data[text]:'untoasted108'
table public.toasted_copy: INSERT: id[integer]:111 data[text]:'untoasted109'
table public.toasted_copy: INSERT: id[integer]:112 data[text]:'untoasted110'
table public.toasted_copy: INSERT: id[integer]:113 data[text]:'untoasted111'
table public.toasted_copy: INSERT: id[integer]:114 data[text]:'untoasted112'
table public.toasted_copy: INSERT: id[integer]:115 data[text]:'untoasted113'
table public.toasted_copy: INSERT: id[integer]:116 data[text]:'untoasted114'
table public.toasted_copy: INSERT: id[integer]:117 data[text]:'untoasted115'
table public.toasted_copy: INSERT: id[integer]:118 data[text]:'untoasted116'
table public.toasted_copy: INSERT: id[integer]:119 data[text]:'untoasted117'
table public.toasted_copy: INSERT: id[integer]:120 data[text]:'untoasted118'
table public.toasted_copy: INSERT: id[integer]:121 data[text]:'untoasted119'
table public.toasted_copy: INSERT: id[integer]:122 data[text]:'untoasted120'
table public.toasted_copy: INSERT: id[integer]:123 data[text]:'untoasted121'
table public.toasted_copy: INSERT: id[integer]:124 data[text]:'untoasted122'
table public.toasted_copy: INSERT: id[integer]:125 data[text]:'untoasted123'
table public.toasted_copy: INSERT: id[integer]:126 data[text]:'untoasted124'
table public.toasted_copy: INSERT: id[integer]:127 data[text]:'untoasted125'
table public.toasted_copy: INSERT: id[integer]:128 data[text]:'untoasted126'
table public.toasted_copy: INSERT: id[integer]:129 data[text]:'untoasted127'
table public.toasted_copy: INSERT: id[integer]:130 data[text]:'untoasted128'
table public.toasted_copy: INSERT: id[integer]:131 data[text]:'untoasted129'
table public.toasted_copy: INSERT: id[integer]:132 data[text]:'untoasted130'
table public.toasted_copy: INSERT: id[integer]:133 data[text]:'untoasted131'
table public.toasted_copy: INSERT: id[integer]:134 data[text]:'untoasted132'
table public.toasted_copy: INSERT: id[integer]:135 data[text]:'untoasted133'
table public.toasted_copy: INSERT: id[integer]:136 data[text]:'untoasted134'
table public.toasted_copy: INSERT: id[integer]:137 data[text]:'untoasted135'
table public.toasted_copy: INSERT: id[integer]:138 data[text]:'untoasted136'
table public.toasted_copy: INSERT: id[integer]:139 data[text]:'untoasted137'
table public.toasted_copy: INSERT: id[integer]:140 data[text]:'untoasted138'
table public.toasted_copy: INSERT: id[integer]:141 data[text]:'untoasted139'
table public.toasted_copy: INSERT: id[integer]:142 data[text]:'untoasted140'
table public.toasted_copy: INSERT: id[integer]:143 data[text]:'untoasted141'
table public.toasted_copy: INSERT: id[integer]:144 data[text]:'untoasted142'
table public.toasted_copy: INSERT: id[integer]:145 data[text]:'untoasted143'
table public.toasted_copy: INSERT: id[integer]:146 data[text]:'untoasted144'
table public.toasted_copy: INSERT: id[integer]:147 data[text]:'untoasted145'
table public.toasted_copy: INSERT: id[integer]:148 data[text]:'untoasted146'
table public.toasted_copy: INSERT: id[integer]:149 data[text]:'untoasted147'
table public.toasted_copy: INSERT: id[integer]:150 data[text]:'untoasted148'
table public.toasted_copy: INSERT: id[integer]:151 data[text]:'untoasted149'
table public.toasted_copy: INSERT: id[integer]:152 data[text]:'untoasted150'
table public.toasted_copy: INSERT: id[integer]:153 data[text]:'untoasted151'
table public.toasted_copy: INSERT: id[integer]:154 data[text]:'untoasted152'
table public.toasted_copy: INSERT: id[integer]:155 data[text]:'untoasted153'
table public.toasted_copy: INSERT: id[integer]:156 data[text]:'untoasted154'
table public.toasted_copy: INSERT: id[integer]:157 data[text]:'untoasted155'
table public.toasted_copy: INSERT: id[integer]:158 data[text]:'untoasted156'
table public.toasted_copy: INSERT: id[integer]:159 data[text]:'untoasted157'
table public.toasted_copy: INSERT: id[integer]:160 data[text]:'untoasted158'
table public.toasted_copy: INSERT: id[integer]:161 data[text]:'untoasted159'
table public.toasted_copy: INSERT: id[integer]:162 data[text]:'untoasted160'
table public.toasted_copy: INSERT: id[integer]:163 data[text]:'untoasted161'
table public.toasted_copy: INSERT: id[integer]:164 data[text]:'untoasted162'
table public.toasted_copy: INSERT: id[integer]:165 data[text]:'untoasted163'
table public.toasted_copy: INSERT: id[integer]:166 data[text]:'untoasted164'
table public.toasted_copy: INSERT: id[integer]:167 data[text]:'untoasted165'
table public.toasted_copy: INSERT: id[integer]:168 data[text]:'untoasted166'
table public.toasted_copy: INSERT: id[integer]:169 data[text]:'untoasted167'
table public.toasted_copy: INSERT: id[integer]:170 data[text]:'untoasted168'
table public.toasted_copy: INSERT: id[integer]:171 data[text]:'untoasted169'
table public.toasted_copy: INSERT: id[integer]:172 data[text]:'untoasted170'
table public.toasted_copy: INSERT: id[integer]:173 data[text]:'untoasted171'
table public.toasted_copy: INSERT: id[integer]:174 data[text]:'untoasted172'
table public.toasted_copy: INSERT: id[integer]:175 data[text]:'untoasted173'
table public.toasted_copy: INSERT: id[integer]:176 data[text]:'untoasted174'
table public.toasted_copy: INSERT: id[integer]:177 data[text]:'untoasted175'
table public.toasted_copy: INSERT: id[integer]:178 data[text]:'untoasted176'
table public.toasted_copy: INSERT: id[integer]:179 data[text]:'untoasted177'
table public.toasted_copy: INSERT: id[integer]:180 data[text]:'untoasted178'
table public.toasted_copy: INSERT: id[integer]:181 data[text]:'untoasted179'
table public.toasted_copy: INSERT: id[integer]:182 data[text]:'untoasted180'
table public.toasted_copy: INSERT: id[integer]:183 data[text]:'untoasted181'
table public.toasted_copy: INSERT: id[integer]:184 data[text]:'untoasted182'
table public.toasted_copy: INSERT: id[integer]:185 data[text]:'untoasted183'
table public.toasted_copy: INSERT: id[integer]:186 data[text]:'untoasted184'
table public.toasted_copy: INSERT: id[integer]:187 data[text]:'untoasted185'
table public.toasted_copy: INSERT: id[integer]:188 data[text]:'untoasted186'
table public.toasted_copy: INSERT: id[integer]:189 data[text]:'untoasted187'
table public.toasted_copy: INSERT: id[integer]:190 data[text]:'untoasted188'
table public.toasted_copy: INSERT: id[integer]:191 data[text]:'untoasted189'
table public.toasted_copy: INSERT: id[integer]:192 data[text]:'untoasted190'
table public.toasted_copy: INSERT: id[integer]:193 data[text]:'untoasted191'
table public.toasted_copy: INSERT: id[integer]:194 data[text]:'untoasted192'
table public.toasted_copy: INSERT: id[integer]:195 data[text]:'untoasted193'
table public.toasted_copy: INSERT: id[integer]:196 data[text]:'untoasted194'
table public.toasted_copy: INSERT: id[integer]:197 data[text]:'untoasted195'
table public.toasted_copy: INSERT: id[integer]:198 data[text]:'untoasted196'
table public.toasted_copy: INSERT: id[integer]:199 data[text]:'untoasted197'
table public.toasted_copy: INSERT: id[integer]:200 data[text]:'untoasted198'
table public.toasted_copy: INSERT: id[integer]:201 data[text]:'toasted3-12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
table public.toasted_copy: INSERT: id[integer]:202 data[text]:'untoasted199'
table public.toasted_copy: INSERT: id[integer]:203 data[text]:'untoasted200'
Fix decoding of MULTI_INSERTs when rows other than the last are toasted. When decoding the results of a HEAP2_MULTI_INSERT (currently only generated by COPY FROM) toast columns for all but the last tuple weren't replaced by their actual contents before being handed to the output plugin. The reassembled toast datums where disregarded after every REORDER_BUFFER_CHANGE_(INSERT|UPDATE|DELETE) which is correct for plain inserts, updates, deletes, but not multi inserts - there we generate several REORDER_BUFFER_CHANGE_INSERTs for a single xl_heap_multi_insert record. To solve the problem add a clear_toast_afterwards boolean to ReorderBufferChange's union member that's used by modifications. All row changes but multi_inserts always set that to true, but multi_insert sets it only for the last change generated. Add a regression test covering decoding of multi_inserts - there was none at all before. Backpatch to 9.4 where logical decoding was introduced. Bug found by Petr Jelinek.
2014-07-06 15:58:01 +02:00
COMMIT
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE. The newly added ON CONFLICT clause allows to specify an alternative to raising a unique or exclusion constraint violation error when inserting. ON CONFLICT refers to constraints that can either be specified using a inference clause (by specifying the columns of a unique constraint) or by naming a unique or exclusion constraint. DO NOTHING avoids the constraint violation, without touching the pre-existing row. DO UPDATE SET ... [WHERE ...] updates the pre-existing tuple, and has access to both the tuple proposed for insertion and the existing tuple; the optional WHERE clause can be used to prevent an update from being executed. The UPDATE SET and WHERE clauses have access to the tuple proposed for insertion using the "magic" EXCLUDED alias, and to the pre-existing tuple using the table name or its alias. This feature is often referred to as upsert. This is implemented using a new infrastructure called "speculative insertion". It is an optimistic variant of regular insertion that first does a pre-check for existing tuples and then attempts an insert. If a violating tuple was inserted concurrently, the speculatively inserted tuple is deleted and a new attempt is made. If the pre-check finds a matching tuple the alternative DO NOTHING or DO UPDATE action is taken. If the insertion succeeds without detecting a conflict, the tuple is deemed inserted. To handle the possible ambiguity between the excluded alias and a table named excluded, and for convenience with long relation names, INSERT INTO now can alias its target table. Bumps catversion as stored rules change. Author: Peter Geoghegan, with significant contributions from Heikki Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes. Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs, Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
(235 rows)
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
logical decoding: Fix handling of large old tuples with replica identity full. When decoding the old version of an UPDATE or DELETE change, and if that tuple was bigger than MaxHeapTupleSize, we either Assert'ed out, or failed in more subtle ways in non-assert builds. Normally individual tuples aren't bigger than MaxHeapTupleSize, with big datums toasted. But that's not the case for the old version of a tuple for logical decoding; the replica identity is logged as one piece. With the default replica identity btree limits that to small tuples, but that's not the case for FULL. Change the tuple buffer infrastructure to separate allocate over-large tuples, instead of always going through the slab cache. This unfortunately requires changing the ReorderBufferTupleBuf definition, we need to store the allocated size someplace. To avoid requiring output plugins to recompile, don't store HeapTupleHeaderData directly after HeapTupleData, but point to it via t_data; that leaves rooms for the allocated size. As there's no reason for an output plugin to look at ReorderBufferTupleBuf->t_data.header, remove the field. It was just a minor convenience having it directly accessible. Reported-By: Adam Dratwiński Discussion: CAKg6ypLd7773AOX4DiOGRwQk1TVOQKhNwjYiVjJnpq8Wo+i62Q@mail.gmail.com
2016-03-06 03:02:20 +01:00
-- test we can decode "old" tuples bigger than the max heap tuple size correctly
DROP TABLE IF EXISTS toasted_several;
NOTICE: table "toasted_several" does not exist, skipping
CREATE TABLE toasted_several (
id serial unique not null,
toasted_key text primary key,
toasted_col1 text,
toasted_col2 text
);
ALTER TABLE toasted_several REPLICA IDENTITY FULL;
ALTER TABLE toasted_several ALTER COLUMN toasted_key SET STORAGE EXTERNAL;
ALTER TABLE toasted_several ALTER COLUMN toasted_col1 SET STORAGE EXTERNAL;
ALTER TABLE toasted_several ALTER COLUMN toasted_col2 SET STORAGE EXTERNAL;
Propagate ALTER TABLE ... SET STORAGE to indexes When creating a new index, the attstorage setting of the table column is copied to regular (non-expression) index columns. But a later ALTER TABLE ... SET STORAGE is not propagated to indexes, thus creating an inconsistent and undumpable state. Discussion: https://www.postgresql.org/message-id/flat/9765d72b-37c0-06f5-e349-2a580aafd989%402ndquadrant.com
2020-04-09 14:10:01 +02:00
-- Change the storage of the index back to EXTENDED, separately from
-- the table. This is currently not doable via DDL, but it is
-- supported internally.
UPDATE pg_attribute SET attstorage = 'x' WHERE attrelid = 'toasted_several_pkey'::regclass AND attname = 'toasted_key';
Add regression test for wide REPLICA IDENTITY FULL updates. This just contains the regression tests added by a fix for a 9.4 specific bug regarding $subject. Author: Andres Freund Backpatch: 9.5-
2017-08-05 23:41:40 +02:00
INSERT INTO toasted_several(toasted_key) VALUES(repeat('9876543210', 10000));
SELECT pg_column_size(toasted_key) > 2^16 FROM toasted_several;
?column?
----------
t
(1 row)
Revert "Logical decoding of sequences" This reverts a sequence of commits, implementing features related to logical decoding and replication of sequences: - 0da92dc530c9251735fc70b20cd004d9630a1266 - 80901b32913ffa59bf157a4d88284b2b3a7511d9 - b779d7d8fdae088d70da5ed9fcd8205035676df3 - d5ed9da41d96988d905b49bebb273a9b2d6e2915 - a180c2b34de0989269fdb819bff241a249bf5380 - 75b1521dae1ff1fde17fda2e30e591f2e5d64b6a - 2d2232933b02d9396113662e44dca5f120d6830e - 002c9dd97a0c874fd1693a570383e2dd38cd40d5 - 05843b1aa49df2ecc9b97c693b755bd1b6f856a9 The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch. Discussion: https://postgr.es/m/95345a19-d508-63d1-860a-f5c2f41e8d40@enterprisedb.com
2022-04-07 18:13:13 +02:00
SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
logical decoding: Fix handling of large old tuples with replica identity full. When decoding the old version of an UPDATE or DELETE change, and if that tuple was bigger than MaxHeapTupleSize, we either Assert'ed out, or failed in more subtle ways in non-assert builds. Normally individual tuples aren't bigger than MaxHeapTupleSize, with big datums toasted. But that's not the case for the old version of a tuple for logical decoding; the replica identity is logged as one piece. With the default replica identity btree limits that to small tuples, but that's not the case for FULL. Change the tuple buffer infrastructure to separate allocate over-large tuples, instead of always going through the slab cache. This unfortunately requires changing the ReorderBufferTupleBuf definition, we need to store the allocated size someplace. To avoid requiring output plugins to recompile, don't store HeapTupleHeaderData directly after HeapTupleData, but point to it via t_data; that leaves rooms for the allocated size. As there's no reason for an output plugin to look at ReorderBufferTupleBuf->t_data.header, remove the field. It was just a minor convenience having it directly accessible. Reported-By: Adam Dratwiński Discussion: CAKg6ypLd7773AOX4DiOGRwQk1TVOQKhNwjYiVjJnpq8Wo+i62Q@mail.gmail.com
2016-03-06 03:02:20 +01:00
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
table public.toasted_several: INSERT: id[integer]:1 toasted_key[text]:'98765432109876543210987654321..098765432109876543210987654321098765432109876543210' toasted_col1[text]:null toasted_col2[text]:null
COMMIT
(3 rows)
-- test update of a toasted key without changing it
UPDATE toasted_several SET toasted_col1 = toasted_key;
UPDATE toasted_several SET toasted_col2 = toasted_col1;
Revert "Logical decoding of sequences" This reverts a sequence of commits, implementing features related to logical decoding and replication of sequences: - 0da92dc530c9251735fc70b20cd004d9630a1266 - 80901b32913ffa59bf157a4d88284b2b3a7511d9 - b779d7d8fdae088d70da5ed9fcd8205035676df3 - d5ed9da41d96988d905b49bebb273a9b2d6e2915 - a180c2b34de0989269fdb819bff241a249bf5380 - 75b1521dae1ff1fde17fda2e30e591f2e5d64b6a - 2d2232933b02d9396113662e44dca5f120d6830e - 002c9dd97a0c874fd1693a570383e2dd38cd40d5 - 05843b1aa49df2ecc9b97c693b755bd1b6f856a9 The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch. Discussion: https://postgr.es/m/95345a19-d508-63d1-860a-f5c2f41e8d40@enterprisedb.com
2022-04-07 18:13:13 +02:00
SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
logical decoding: Fix handling of large old tuples with replica identity full. When decoding the old version of an UPDATE or DELETE change, and if that tuple was bigger than MaxHeapTupleSize, we either Assert'ed out, or failed in more subtle ways in non-assert builds. Normally individual tuples aren't bigger than MaxHeapTupleSize, with big datums toasted. But that's not the case for the old version of a tuple for logical decoding; the replica identity is logged as one piece. With the default replica identity btree limits that to small tuples, but that's not the case for FULL. Change the tuple buffer infrastructure to separate allocate over-large tuples, instead of always going through the slab cache. This unfortunately requires changing the ReorderBufferTupleBuf definition, we need to store the allocated size someplace. To avoid requiring output plugins to recompile, don't store HeapTupleHeaderData directly after HeapTupleData, but point to it via t_data; that leaves rooms for the allocated size. As there's no reason for an output plugin to look at ReorderBufferTupleBuf->t_data.header, remove the field. It was just a minor convenience having it directly accessible. Reported-By: Adam Dratwiński Discussion: CAKg6ypLd7773AOX4DiOGRwQk1TVOQKhNwjYiVjJnpq8Wo+i62Q@mail.gmail.com
2016-03-06 03:02:20 +01:00
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
table public.toasted_several: INSERT: id[integer]:1 toasted_key[text]:'98765432109876543210987654321..098765432109876543210987654321098765432109876543210' toasted_col1[text]:null toasted_col2[text]:null
COMMIT
BEGIN
table public.toasted_several: UPDATE: old-key: id[integer]:1 toasted_key[text]:'98765432109876543210..432109876543210987654321098765432109876543210987654321098765432109876543210' toasted_col2[text]:null
COMMIT
BEGIN
table public.toasted_several: UPDATE: old-key: id[integer]:1 toasted_key[text]:'98765432109876543210..876543210987654321098765432109876543210987654321098765432109876543210987654321098765432109876543210'
COMMIT
(9 rows)
/*
* update with large tuplebuf, in a transaction large enough to force to spool to disk
*/
BEGIN;
INSERT INTO toasted_several(toasted_key) SELECT * FROM generate_series(1, 10234);
UPDATE toasted_several SET toasted_col1 = toasted_col2 WHERE id = 1;
DELETE FROM toasted_several WHERE id = 1;
COMMIT;
DROP TABLE toasted_several;
Revert "Logical decoding of sequences" This reverts a sequence of commits, implementing features related to logical decoding and replication of sequences: - 0da92dc530c9251735fc70b20cd004d9630a1266 - 80901b32913ffa59bf157a4d88284b2b3a7511d9 - b779d7d8fdae088d70da5ed9fcd8205035676df3 - d5ed9da41d96988d905b49bebb273a9b2d6e2915 - a180c2b34de0989269fdb819bff241a249bf5380 - 75b1521dae1ff1fde17fda2e30e591f2e5d64b6a - 2d2232933b02d9396113662e44dca5f120d6830e - 002c9dd97a0c874fd1693a570383e2dd38cd40d5 - 05843b1aa49df2ecc9b97c693b755bd1b6f856a9 The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch. Discussion: https://postgr.es/m/95345a19-d508-63d1-860a-f5c2f41e8d40@enterprisedb.com
2022-04-07 18:13:13 +02:00
SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
logical decoding: Fix handling of large old tuples with replica identity full. When decoding the old version of an UPDATE or DELETE change, and if that tuple was bigger than MaxHeapTupleSize, we either Assert'ed out, or failed in more subtle ways in non-assert builds. Normally individual tuples aren't bigger than MaxHeapTupleSize, with big datums toasted. But that's not the case for the old version of a tuple for logical decoding; the replica identity is logged as one piece. With the default replica identity btree limits that to small tuples, but that's not the case for FULL. Change the tuple buffer infrastructure to separate allocate over-large tuples, instead of always going through the slab cache. This unfortunately requires changing the ReorderBufferTupleBuf definition, we need to store the allocated size someplace. To avoid requiring output plugins to recompile, don't store HeapTupleHeaderData directly after HeapTupleData, but point to it via t_data; that leaves rooms for the allocated size. As there's no reason for an output plugin to look at ReorderBufferTupleBuf->t_data.header, remove the field. It was just a minor convenience having it directly accessible. Reported-By: Adam Dratwiński Discussion: CAKg6ypLd7773AOX4DiOGRwQk1TVOQKhNwjYiVjJnpq8Wo+i62Q@mail.gmail.com
2016-03-06 03:02:20 +01:00
WHERE data NOT LIKE '%INSERT: %';
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
table public.toasted_several: UPDATE: old-key: id[integer]:1 toasted_key[text]:'98765432109876543210..7654321098765432109876543210987654321098765432109876543210' toasted_col2[text]:unchanged-toast-datum
table public.toasted_several: DELETE: id[integer]:1 toasted_key[text]:'98765432109876543210987654321..876543210987654321098765432109876543210987654321098765432109876543210987654321098765432109876543210'
COMMIT
(4 rows)
Fix toast rewrites in logical decoding. Commit 325f2ec555 introduced pg_class.relwrite to skip operations on tables created as part of a heap rewrite during DDL. It links such transient heaps to the original relation OID via this new field in pg_class but forgot to do anything about toast tables. So, logical decoding was not able to skip operations on internally created toast tables. This leads to an error when we tried to decode the WAL for the next operation for which it appeared that there is a toast data where actually it didn't have any toast data. To fix this, we set pg_class.relwrite for internally created toast tables as well which allowed skipping operations on them during logical decoding. Author: Bertrand Drouvot Reviewed-by: David Zhang, Amit Kapila Backpatch-through: 11, where it was introduced Discussion: https://postgr.es/m/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com
2021-08-25 06:23:07 +02:00
/*
* Test decoding relation rewrite with toast. The insert into tbl2 within the
* same transaction is there to check that there is no remaining toast_hash not
* being reset.
*/
CREATE TABLE tbl1 (a INT, b TEXT);
CREATE TABLE tbl2 (a INT);
ALTER TABLE tbl1 ALTER COLUMN b SET STORAGE EXTERNAL;
BEGIN;
INSERT INTO tbl1 VALUES(1, repeat('a', 4000)) ;
ALTER TABLE tbl1 ADD COLUMN id serial primary key;
INSERT INTO tbl2 VALUES(1);
commit;
Revert "Logical decoding of sequences" This reverts a sequence of commits, implementing features related to logical decoding and replication of sequences: - 0da92dc530c9251735fc70b20cd004d9630a1266 - 80901b32913ffa59bf157a4d88284b2b3a7511d9 - b779d7d8fdae088d70da5ed9fcd8205035676df3 - d5ed9da41d96988d905b49bebb273a9b2d6e2915 - a180c2b34de0989269fdb819bff241a249bf5380 - 75b1521dae1ff1fde17fda2e30e591f2e5d64b6a - 2d2232933b02d9396113662e44dca5f120d6830e - 002c9dd97a0c874fd1693a570383e2dd38cd40d5 - 05843b1aa49df2ecc9b97c693b755bd1b6f856a9 The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch. Discussion: https://postgr.es/m/95345a19-d508-63d1-860a-f5c2f41e8d40@enterprisedb.com
2022-04-07 18:13:13 +02:00
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
Fix toast rewrites in logical decoding. Commit 325f2ec555 introduced pg_class.relwrite to skip operations on tables created as part of a heap rewrite during DDL. It links such transient heaps to the original relation OID via this new field in pg_class but forgot to do anything about toast tables. So, logical decoding was not able to skip operations on internally created toast tables. This leads to an error when we tried to decode the WAL for the next operation for which it appeared that there is a toast data where actually it didn't have any toast data. To fix this, we set pg_class.relwrite for internally created toast tables as well which allowed skipping operations on them during logical decoding. Author: Bertrand Drouvot Reviewed-by: David Zhang, Amit Kapila Backpatch-through: 11, where it was introduced Discussion: https://postgr.es/m/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com
2021-08-25 06:23:07 +02:00
substr
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
table public.tbl1: INSERT: a[integer]:1 b[text]:'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
table public.tbl2: INSERT: a[integer]:1
COMMIT
(4 rows)
Introduce logical decoding. This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering. Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender. Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.
2014-03-03 22:32:18 +01:00
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
(1 row)