Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
* output_plugin.h
|
|
|
|
* PostgreSQL Logical Decode Plugin Interface
|
|
|
|
*
|
2021-01-02 19:06:25 +01:00
|
|
|
* Copyright (c) 2012-2021, PostgreSQL Global Development Group
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
#ifndef OUTPUT_PLUGIN_H
|
|
|
|
#define OUTPUT_PLUGIN_H
|
|
|
|
|
|
|
|
#include "replication/reorderbuffer.h"
|
|
|
|
|
|
|
|
struct LogicalDecodingContext;
|
|
|
|
struct OutputPluginCallbacks;
|
|
|
|
|
|
|
|
typedef enum OutputPluginOutputType
|
|
|
|
{
|
|
|
|
OUTPUT_PLUGIN_BINARY_OUTPUT,
|
|
|
|
OUTPUT_PLUGIN_TEXTUAL_OUTPUT
|
|
|
|
} OutputPluginOutputType;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Options set by the output plugin, in the startup callback.
|
|
|
|
*/
|
|
|
|
typedef struct OutputPluginOptions
|
|
|
|
{
|
|
|
|
OutputPluginOutputType output_type;
|
2018-03-21 14:13:24 +01:00
|
|
|
bool receive_rewrites;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
} OutputPluginOptions;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Type of the shared library symbol _PG_output_plugin_init that is looked up
|
|
|
|
* when loading an output plugin shared library.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback that gets called in a user-defined plugin. ctx->private_data can
|
|
|
|
* be set to some private data.
|
|
|
|
*
|
|
|
|
* "is_init" will be set to "true" if the decoding slot just got defined. When
|
|
|
|
* the same slot is used from there one, it will be "false".
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeStartupCB) (struct LogicalDecodingContext *ctx,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
OutputPluginOptions *options,
|
2017-02-16 00:15:47 +01:00
|
|
|
bool is_init);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback called for every (explicit or implicit) BEGIN of a successful
|
|
|
|
* transaction.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeBeginCB) (struct LogicalDecodingContext *ctx,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ReorderBufferTXN *txn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback for every individual change in a successful transaction.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeChangeCB) (struct LogicalDecodingContext *ctx,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
Relation relation,
|
2017-02-16 00:15:47 +01:00
|
|
|
ReorderBufferChange *change);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2018-04-07 17:17:56 +02:00
|
|
|
/*
|
|
|
|
* Callback for every TRUNCATE in a successful transaction.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeTruncateCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
int nrelations,
|
|
|
|
Relation relations[],
|
|
|
|
ReorderBufferChange *change);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Called for every (explicit or implicit) COMMIT of a successful transaction.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
|
|
|
|
2016-04-06 11:05:41 +02:00
|
|
|
/*
|
|
|
|
* Called for the generic logical decoding messages.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
|
2016-04-06 11:05:41 +02:00
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn,
|
|
|
|
bool transactional,
|
|
|
|
const char *prefix,
|
|
|
|
Size message_size,
|
|
|
|
const char *message);
|
|
|
|
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
/*
|
|
|
|
* Filter changes by origin.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ctx,
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
RepOriginId origin_id);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Called to shutdown an output plugin.
|
|
|
|
*/
|
2017-02-16 00:15:47 +01:00
|
|
|
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
/*
|
|
|
|
* Called before decoding of PREPARE record to decide whether this
|
|
|
|
* transaction should be decoded with separate calls to prepare and
|
|
|
|
* commit_prepared/rollback_prepared callbacks or wait till COMMIT PREPARED
|
|
|
|
* and sent as usual transaction.
|
|
|
|
*/
|
|
|
|
typedef bool (*LogicalDecodeFilterPrepareCB) (struct LogicalDecodingContext *ctx,
|
2021-03-30 07:04:43 +02:00
|
|
|
TransactionId xid,
|
2020-12-30 11:47:26 +01:00
|
|
|
const char *gid);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback called for every BEGIN of a prepared trnsaction.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeBeginPrepareCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called for PREPARE record unless it was filtered by filter_prepare()
|
|
|
|
* callback.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called for COMMIT PREPARED.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called for ROLLBACK PREPARED.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeRollbackPreparedCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_end_lsn,
|
|
|
|
TimestampTz prepare_time);
|
|
|
|
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/*
|
|
|
|
* Called when starting to stream a block of changes from in-progress
|
|
|
|
* transaction (may be called repeatedly, if it's streamed in multiple
|
|
|
|
* chunks).
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called when stopping to stream a block of changes from in-progress
|
|
|
|
* transaction to a remote node (may be called repeatedly, if it's streamed
|
|
|
|
* in multiple chunks).
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called to discard changes streamed to remote node from in-progress
|
|
|
|
* transaction.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr abort_lsn);
|
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
/*
|
|
|
|
* Called to prepare changes streamed to remote node from in-progress
|
|
|
|
* transaction. This is called as part of a two-phase commit.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamPrepareCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn);
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/*
|
|
|
|
* Called to apply changes streamed to remote node from in-progress
|
|
|
|
* transaction.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback for streaming individual changes from in-progress transactions.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
Relation relation,
|
|
|
|
ReorderBufferChange *change);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback for streaming generic logical decoding messages from in-progress
|
|
|
|
* transactions.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn,
|
|
|
|
bool transactional,
|
|
|
|
const char *prefix,
|
|
|
|
Size message_size,
|
|
|
|
const char *message);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback for streaming truncates from in-progress transactions.
|
|
|
|
*/
|
|
|
|
typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
int nrelations,
|
|
|
|
Relation relations[],
|
|
|
|
ReorderBufferChange *change);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Output plugin callbacks
|
|
|
|
*/
|
|
|
|
typedef struct OutputPluginCallbacks
|
|
|
|
{
|
|
|
|
LogicalDecodeStartupCB startup_cb;
|
|
|
|
LogicalDecodeBeginCB begin_cb;
|
|
|
|
LogicalDecodeChangeCB change_cb;
|
2018-04-07 17:17:56 +02:00
|
|
|
LogicalDecodeTruncateCB truncate_cb;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalDecodeCommitCB commit_cb;
|
2016-04-06 11:05:41 +02:00
|
|
|
LogicalDecodeMessageCB message_cb;
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalDecodeShutdownCB shutdown_cb;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* streaming of changes at prepare time */
|
|
|
|
LogicalDecodeFilterPrepareCB filter_prepare_cb;
|
|
|
|
LogicalDecodeBeginPrepareCB begin_prepare_cb;
|
|
|
|
LogicalDecodePrepareCB prepare_cb;
|
|
|
|
LogicalDecodeCommitPreparedCB commit_prepared_cb;
|
|
|
|
LogicalDecodeRollbackPreparedCB rollback_prepared_cb;
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/* streaming of changes */
|
|
|
|
LogicalDecodeStreamStartCB stream_start_cb;
|
|
|
|
LogicalDecodeStreamStopCB stream_stop_cb;
|
|
|
|
LogicalDecodeStreamAbortCB stream_abort_cb;
|
2020-12-30 11:47:26 +01:00
|
|
|
LogicalDecodeStreamPrepareCB stream_prepare_cb;
|
2020-07-28 04:36:44 +02:00
|
|
|
LogicalDecodeStreamCommitCB stream_commit_cb;
|
|
|
|
LogicalDecodeStreamChangeCB stream_change_cb;
|
|
|
|
LogicalDecodeStreamMessageCB stream_message_cb;
|
|
|
|
LogicalDecodeStreamTruncateCB stream_truncate_cb;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
} OutputPluginCallbacks;
|
|
|
|
|
2017-02-16 00:15:47 +01:00
|
|
|
/* Functions in replication/logical/logical.c */
|
|
|
|
extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
|
|
|
|
extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
|
2017-05-12 11:50:56 +02:00
|
|
|
extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
#endif /* OUTPUT_PLUGIN_H */
|