Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
* logical.c
|
|
|
|
* PostgreSQL logical decoding coordination
|
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Copyright (c) 2012-2023, PostgreSQL Global Development Group
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
|
|
|
* src/backend/replication/logical/logical.c
|
|
|
|
*
|
|
|
|
* NOTES
|
|
|
|
* This file coordinates interaction between the various modules that
|
2014-03-04 17:07:54 +01:00
|
|
|
* together provide logical decoding, primarily by providing so
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* called LogicalDecodingContexts. The goal is to encapsulate most of the
|
|
|
|
* internal complexity for consumers of logical decoding, so they can
|
2014-03-12 19:03:09 +01:00
|
|
|
* create and consume a changestream with a low amount of code. Builtin
|
|
|
|
* consumers are the walsender and SQL SRF interface, but it's possible to
|
|
|
|
* add further ones without changing core code, e.g. to consume changes in
|
|
|
|
* a bgworker.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*
|
|
|
|
* The idea is that a consumer provides three callbacks, one to read WAL,
|
|
|
|
* one to prepare a data write, and a final one for actually writing since
|
|
|
|
* their implementation depends on the type of consumer. Check
|
2014-03-12 19:03:09 +01:00
|
|
|
* logicalfuncs.c for an example implementation of a fairly simple consumer
|
2014-05-23 14:16:09 +02:00
|
|
|
* and an implementation of a WAL reading callback that's suitable for
|
2014-03-12 19:03:09 +01:00
|
|
|
* simple consumers.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "postgres.h"
|
|
|
|
|
|
|
|
#include "access/xact.h"
|
Revamp the WAL record format.
Each WAL record now carries information about the modified relation and
block(s) in a standardized format. That makes it easier to write tools that
need that information, like pg_rewind, prefetching the blocks to speed up
recovery, etc.
There's a whole new API for building WAL records, replacing the XLogRecData
chains used previously. The new API consists of XLogRegister* functions,
which are called for each buffer and chunk of data that is added to the
record. The new API also gives more control over when a full-page image is
written, by passing flags to the XLogRegisterBuffer function.
This also simplifies the XLogReadBufferForRedo() calls. The function can dig
the relation and block number from the WAL record, so they no longer need to
be passed as arguments.
For the convenience of redo routines, XLogReader now disects each WAL record
after reading it, copying the main data part and the per-block data into
MAXALIGNed buffers. The data chunks are not aligned within the WAL record,
but the redo routines can assume that the pointers returned by XLogRecGet*
functions are. Redo routines are now passed the XLogReaderState, which
contains the record in the already-disected format, instead of the plain
XLogRecord.
The new record format also makes the fixed size XLogRecord header smaller,
by removing the xl_len field. The length of the "main data" portion is now
stored at the end of the WAL record, and there's a separate header after
XLogRecord for it. The alignment padding at the end of XLogRecord is also
removed. This compansates for the fact that the new format would otherwise
be more bulky than the old format.
Reviewed by Andres Freund, Amit Kapila, Michael Paquier, Alvaro Herrera,
Fujii Masao.
2014-11-20 16:56:26 +01:00
|
|
|
#include "access/xlog_internal.h"
|
2019-11-12 04:00:16 +01:00
|
|
|
#include "fmgr.h"
|
|
|
|
#include "miscadmin.h"
|
2020-10-08 05:39:08 +02:00
|
|
|
#include "pgstat.h"
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
#include "replication/decode.h"
|
|
|
|
#include "replication/logical.h"
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
#include "replication/origin.h"
|
2019-11-12 04:00:16 +01:00
|
|
|
#include "replication/reorderbuffer.h"
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
#include "replication/snapbuild.h"
|
|
|
|
#include "storage/proc.h"
|
|
|
|
#include "storage/procarray.h"
|
2020-08-10 18:51:31 +02:00
|
|
|
#include "utils/builtins.h"
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
#include "utils/memutils.h"
|
|
|
|
|
|
|
|
/* data for errcontext callback */
|
|
|
|
typedef struct LogicalErrorCallbackState
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx;
|
|
|
|
const char *callback_name;
|
|
|
|
XLogRecPtr report_location;
|
|
|
|
} LogicalErrorCallbackState;
|
|
|
|
|
|
|
|
/* wrappers around output plugin callbacks */
|
|
|
|
static void output_plugin_error_callback(void *arg);
|
|
|
|
static void startup_cb_wrapper(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
|
|
|
|
bool is_init);
|
|
|
|
static void shutdown_cb_wrapper(LogicalDecodingContext *ctx);
|
|
|
|
static void begin_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
|
|
|
|
static void commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
2020-12-30 11:47:26 +01:00
|
|
|
static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
|
|
|
|
static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn);
|
|
|
|
static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
|
|
|
static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
Relation relation, ReorderBufferChange *change);
|
2018-04-07 17:17:56 +02:00
|
|
|
static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
int nrelations, Relation relations[], ReorderBufferChange *change);
|
2016-04-06 11:05:41 +02:00
|
|
|
static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn, bool transactional,
|
|
|
|
const char *prefix, Size message_size, const char *message);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/* streaming callbacks */
|
|
|
|
static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr first_lsn);
|
|
|
|
static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr last_lsn);
|
|
|
|
static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr abort_lsn);
|
2020-12-30 11:47:26 +01:00
|
|
|
static void stream_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn);
|
2020-07-28 04:36:44 +02:00
|
|
|
static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn);
|
|
|
|
static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
Relation relation, ReorderBufferChange *change);
|
|
|
|
static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn, bool transactional,
|
|
|
|
const char *prefix, Size message_size, const char *message);
|
|
|
|
static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
int nrelations, Relation relations[], ReorderBufferChange *change);
|
|
|
|
|
2023-02-08 03:28:25 +01:00
|
|
|
/* callback to update txn's progress */
|
|
|
|
static void update_progress_txn_cb_wrapper(ReorderBuffer *cache,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr lsn);
|
|
|
|
|
2020-08-08 07:31:52 +02:00
|
|
|
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, const char *plugin);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure the current settings & environment are capable of doing logical
|
|
|
|
* decoding.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
CheckLogicalDecodingRequirements(void)
|
|
|
|
{
|
|
|
|
CheckSlotRequirements();
|
|
|
|
|
2018-10-31 22:47:41 +01:00
|
|
|
/*
|
|
|
|
* NB: Adding a new requirement likely means that RestoreSlotFromDisk()
|
|
|
|
* needs the same check.
|
|
|
|
*/
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
if (wal_level < WAL_LEVEL_LOGICAL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("logical decoding requires wal_level >= logical")));
|
|
|
|
|
|
|
|
if (MyDatabaseId == InvalidOid)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("logical decoding requires a database connection")));
|
|
|
|
|
|
|
|
if (RecoveryInProgress())
|
2023-04-08 11:20:01 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* This check may have race conditions, but whenever
|
|
|
|
* XLOG_PARAMETER_CHANGE indicates that wal_level has changed, we
|
|
|
|
* verify that there are no existing logical replication slots. And to
|
|
|
|
* avoid races around creating a new slot,
|
|
|
|
* CheckLogicalDecodingRequirements() is called once before creating
|
|
|
|
* the slot, and once when logical decoding is initially starting up.
|
|
|
|
*/
|
|
|
|
if (GetActiveWalLevelOnStandby() < WAL_LEVEL_LOGICAL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2023-04-12 20:00:37 +02:00
|
|
|
errmsg("logical decoding on standby requires wal_level >= logical on the primary")));
|
2023-04-08 11:20:01 +02:00
|
|
|
}
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-07-01 03:00:23 +02:00
|
|
|
* Helper function for CreateInitDecodingContext() and
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* CreateDecodingContext() performing common tasks.
|
|
|
|
*/
|
|
|
|
static LogicalDecodingContext *
|
|
|
|
StartupDecodingContext(List *output_plugin_options,
|
|
|
|
XLogRecPtr start_lsn,
|
|
|
|
TransactionId xmin_horizon,
|
2017-04-28 00:28:24 +02:00
|
|
|
bool need_full_snapshot,
|
2018-01-17 12:38:34 +01:00
|
|
|
bool fast_forward,
|
2021-05-10 06:00:53 +02:00
|
|
|
XLogReaderRoutine *xl_routine,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalOutputPluginWriterPrepareWrite prepare_write,
|
2017-05-12 11:50:56 +02:00
|
|
|
LogicalOutputPluginWriterWrite do_write,
|
|
|
|
LogicalOutputPluginWriterUpdateProgress update_progress)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
ReplicationSlot *slot;
|
|
|
|
MemoryContext context,
|
|
|
|
old_context;
|
|
|
|
LogicalDecodingContext *ctx;
|
|
|
|
|
|
|
|
/* shorter lines... */
|
|
|
|
slot = MyReplicationSlot;
|
|
|
|
|
|
|
|
context = AllocSetContextCreate(CurrentMemoryContext,
|
Add macros to make AllocSetContextCreate() calls simpler and safer.
I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls
had typos in the context-sizing parameters. While none of these led to
especially significant problems, they did create minor inefficiencies,
and it's now clear that expecting people to copy-and-paste those calls
accurately is not a great idea. Let's reduce the risk of future errors
by introducing single macros that encapsulate the common use-cases.
Three such macros are enough to cover all but two special-purpose contexts;
those two calls can be left as-is, I think.
While this patch doesn't in itself improve matters for third-party
extensions, it doesn't break anything for them either, and they can
gradually adopt the simplified notation over time.
In passing, change TopMemoryContext to use the default allocation
parameters. Formerly it could only be extended 8K at a time. That was
probably reasonable when this code was written; but nowadays we create
many more contexts than we did then, so that it's not unusual to have a
couple hundred K in TopMemoryContext, even without considering various
dubious code that sticks other things there. There seems no good reason
not to let it use growing blocks like most other contexts.
Back-patch to 9.6, mostly because that's still close enough to HEAD that
it's easy to do so, and keeping the branches in sync can be expected to
avoid some future back-patching pain. The bugs fixed by these changes
don't seem to be significant enough to justify fixing them further back.
Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
|
|
|
"Logical decoding context",
|
|
|
|
ALLOCSET_DEFAULT_SIZES);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
old_context = MemoryContextSwitchTo(context);
|
|
|
|
ctx = palloc0(sizeof(LogicalDecodingContext));
|
|
|
|
|
|
|
|
ctx->context = context;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* (re-)load output plugins, so we detect a bad (removed) output plugin
|
|
|
|
* now.
|
|
|
|
*/
|
2018-01-17 12:38:34 +01:00
|
|
|
if (!fast_forward)
|
|
|
|
LoadOutputPlugin(&ctx->callbacks, NameStr(slot->data.plugin));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Now that the slot's xmin has been set, we can announce ourselves as a
|
|
|
|
* logical decoding backend which doesn't need to be checked individually
|
|
|
|
* when computing the xmin horizon because the xmin is enforced via
|
|
|
|
* replication slots.
|
2014-12-02 23:42:26 +01:00
|
|
|
*
|
|
|
|
* We can only do so if we're outside of a transaction (i.e. the case when
|
2015-05-20 15:18:11 +02:00
|
|
|
* streaming changes via walsender), otherwise an already setup
|
2014-12-02 23:42:26 +01:00
|
|
|
* snapshot/xid would end up being ignored. That's not a particularly
|
|
|
|
* bothersome restriction since the SQL interface can't be used for
|
|
|
|
* streaming anyway.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*/
|
2014-12-02 23:42:26 +01:00
|
|
|
if (!IsTransactionOrTransactionBlock())
|
|
|
|
{
|
2020-11-26 16:30:48 +01:00
|
|
|
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
|
2020-11-16 23:42:55 +01:00
|
|
|
MyProc->statusFlags |= PROC_IN_LOGICAL_DECODING;
|
|
|
|
ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags;
|
2014-12-02 23:42:26 +01:00
|
|
|
LWLockRelease(ProcArrayLock);
|
|
|
|
}
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
ctx->slot = slot;
|
|
|
|
|
2021-05-10 06:00:53 +02:00
|
|
|
ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, ctx);
|
2015-04-03 14:55:37 +02:00
|
|
|
if (!ctx->reader)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OUT_OF_MEMORY),
|
2021-11-22 17:43:43 +01:00
|
|
|
errmsg("out of memory"),
|
|
|
|
errdetail("Failed while allocating a WAL reading processor.")));
|
2015-04-03 14:55:37 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ctx->reorder = ReorderBufferAllocate();
|
|
|
|
ctx->snapshot_builder =
|
2017-04-28 00:28:24 +02:00
|
|
|
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
need_full_snapshot, slot->data.two_phase_at);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
ctx->reorder->private_data = ctx;
|
|
|
|
|
|
|
|
/* wrap output plugin callbacks, so we can add error context information */
|
|
|
|
ctx->reorder->begin = begin_cb_wrapper;
|
|
|
|
ctx->reorder->apply_change = change_cb_wrapper;
|
2018-04-07 17:17:56 +02:00
|
|
|
ctx->reorder->apply_truncate = truncate_cb_wrapper;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ctx->reorder->commit = commit_cb_wrapper;
|
2016-04-06 11:05:41 +02:00
|
|
|
ctx->reorder->message = message_cb_wrapper;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/*
|
|
|
|
* To support streaming, we require start/stop/abort/commit/change
|
|
|
|
* callbacks. The message and truncate callbacks are optional, similar to
|
|
|
|
* regular output plugins. We however enable streaming when at least one
|
|
|
|
* of the methods is enabled so that we can easily identify missing
|
|
|
|
* methods.
|
|
|
|
*
|
|
|
|
* We decide it here, but only check it later in the wrappers.
|
|
|
|
*/
|
|
|
|
ctx->streaming = (ctx->callbacks.stream_start_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_stop_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_abort_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_commit_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_change_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_message_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_truncate_cb != NULL);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* streaming callbacks
|
|
|
|
*
|
|
|
|
* stream_message and stream_truncate callbacks are optional, so we do not
|
|
|
|
* fail with ERROR when missing, but the wrappers simply do nothing. We
|
|
|
|
* must set the ReorderBuffer callbacks to something, otherwise the calls
|
|
|
|
* from there will crash (we don't want to move the checks there).
|
|
|
|
*/
|
|
|
|
ctx->reorder->stream_start = stream_start_cb_wrapper;
|
|
|
|
ctx->reorder->stream_stop = stream_stop_cb_wrapper;
|
|
|
|
ctx->reorder->stream_abort = stream_abort_cb_wrapper;
|
2020-12-30 11:47:26 +01:00
|
|
|
ctx->reorder->stream_prepare = stream_prepare_cb_wrapper;
|
2020-07-28 04:36:44 +02:00
|
|
|
ctx->reorder->stream_commit = stream_commit_cb_wrapper;
|
|
|
|
ctx->reorder->stream_change = stream_change_cb_wrapper;
|
|
|
|
ctx->reorder->stream_message = stream_message_cb_wrapper;
|
|
|
|
ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
|
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* To support two-phase logical decoding, we require
|
|
|
|
* begin_prepare/prepare/commit-prepare/abort-prepare callbacks. The
|
|
|
|
* filter_prepare callback is optional. We however enable two-phase
|
|
|
|
* logical decoding when at least one of the methods is enabled so that we
|
|
|
|
* can easily identify missing methods.
|
|
|
|
*
|
|
|
|
* We decide it here, but only check it later in the wrappers.
|
|
|
|
*/
|
|
|
|
ctx->twophase = (ctx->callbacks.begin_prepare_cb != NULL) ||
|
|
|
|
(ctx->callbacks.prepare_cb != NULL) ||
|
|
|
|
(ctx->callbacks.commit_prepared_cb != NULL) ||
|
|
|
|
(ctx->callbacks.rollback_prepared_cb != NULL) ||
|
|
|
|
(ctx->callbacks.stream_prepare_cb != NULL) ||
|
|
|
|
(ctx->callbacks.filter_prepare_cb != NULL);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback to support decoding at prepare time.
|
|
|
|
*/
|
|
|
|
ctx->reorder->begin_prepare = begin_prepare_cb_wrapper;
|
|
|
|
ctx->reorder->prepare = prepare_cb_wrapper;
|
|
|
|
ctx->reorder->commit_prepared = commit_prepared_cb_wrapper;
|
|
|
|
ctx->reorder->rollback_prepared = rollback_prepared_cb_wrapper;
|
|
|
|
|
2023-02-08 03:28:25 +01:00
|
|
|
/*
|
|
|
|
* Callback to support updating progress during sending data of a
|
|
|
|
* transaction (and its subtransactions) to the output plugin.
|
|
|
|
*/
|
|
|
|
ctx->reorder->update_progress_txn = update_progress_txn_cb_wrapper;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ctx->out = makeStringInfo();
|
|
|
|
ctx->prepare_write = prepare_write;
|
|
|
|
ctx->write = do_write;
|
2017-05-12 11:50:56 +02:00
|
|
|
ctx->update_progress = update_progress;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
ctx->output_plugin_options = output_plugin_options;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
ctx->fast_forward = fast_forward;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
MemoryContextSwitchTo(old_context);
|
|
|
|
|
|
|
|
return ctx;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a new decoding context, for a new logical slot.
|
|
|
|
*
|
2019-04-05 19:52:45 +02:00
|
|
|
* plugin -- contains the name of the output plugin
|
|
|
|
* output_plugin_options -- contains options passed to the output plugin
|
2020-03-17 20:13:18 +01:00
|
|
|
* need_full_snapshot -- if true, must obtain a snapshot able to read all
|
|
|
|
* tables; if false, one that can read only catalogs is acceptable.
|
2019-04-05 19:52:45 +02:00
|
|
|
* restart_lsn -- if given as invalid, it's this routine's responsibility to
|
|
|
|
* mark WAL as reserved by setting a convenient restart_lsn for the slot.
|
|
|
|
* Otherwise, we set for decoding to start from the given LSN without
|
|
|
|
* marking WAL reserved beforehand. In that scenario, it's up to the
|
|
|
|
* caller to guarantee that WAL remains available.
|
2020-05-08 21:30:34 +02:00
|
|
|
* xl_routine -- XLogReaderRoutine for underlying XLogReader
|
|
|
|
* prepare_write, do_write, update_progress --
|
2019-04-05 19:52:45 +02:00
|
|
|
* callbacks that perform the use-case dependent, actual, work.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*
|
|
|
|
* Needs to be called while in a memory context that's at least as long lived
|
2014-03-05 19:00:22 +01:00
|
|
|
* as the decoding context because further memory contexts will be created
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* inside it.
|
|
|
|
*
|
|
|
|
* Returns an initialized decoding context after calling the output plugin's
|
|
|
|
* startup function.
|
|
|
|
*/
|
|
|
|
LogicalDecodingContext *
|
2020-08-08 07:31:52 +02:00
|
|
|
CreateInitDecodingContext(const char *plugin,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
List *output_plugin_options,
|
Preserve required !catalog tuples while computing initial decoding snapshot.
The logical decoding machinery already preserved all the required
catalog tuples, which is sufficient in the course of normal logical
decoding, but did not guarantee that non-catalog tuples were preserved
during computation of the initial snapshot when creating a slot over
the replication protocol.
This could cause a corrupted initial snapshot being exported. The
time window for issues is usually not terribly large, but on a busy
server it's perfectly possible to it hit it. Ongoing decoding is not
affected by this bug.
To avoid increased overhead for the SQL API, only retain additional
tuples when a logical slot is being created over the replication
protocol. To do so this commit changes the signature of
CreateInitDecodingContext(), but it seems unlikely that it's being
used in an extension, so that's probably ok.
In a drive-by fix, fix handling of
ReplicationSlotsComputeRequiredXmin's already_locked argument, which
should only apply to ProcArrayLock, not ReplicationSlotControlLock.
Reported-By: Erik Rijkers
Analyzed-By: Petr Jelinek
Author: Petr Jelinek, heavily editorialized by Andres Freund
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/9a897b86-46e1-9915-ee4c-da02e4ff6a95@2ndquadrant.com
Backport: 9.4, where logical decoding was introduced.
2017-04-24 05:41:29 +02:00
|
|
|
bool need_full_snapshot,
|
2019-04-05 19:52:45 +02:00
|
|
|
XLogRecPtr restart_lsn,
|
2021-05-10 06:00:53 +02:00
|
|
|
XLogReaderRoutine *xl_routine,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalOutputPluginWriterPrepareWrite prepare_write,
|
2017-05-12 11:50:56 +02:00
|
|
|
LogicalOutputPluginWriterWrite do_write,
|
|
|
|
LogicalOutputPluginWriterUpdateProgress update_progress)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
TransactionId xmin_horizon = InvalidTransactionId;
|
|
|
|
ReplicationSlot *slot;
|
2020-08-10 18:51:31 +02:00
|
|
|
NameData plugin_name;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalDecodingContext *ctx;
|
|
|
|
MemoryContext old_context;
|
|
|
|
|
2023-04-08 11:20:01 +02:00
|
|
|
/*
|
2023-05-19 23:24:48 +02:00
|
|
|
* On a standby, this check is also required while creating the slot.
|
|
|
|
* Check the comments in the function.
|
2023-04-08 11:20:01 +02:00
|
|
|
*/
|
|
|
|
CheckLogicalDecodingRequirements();
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* shorter lines... */
|
|
|
|
slot = MyReplicationSlot;
|
|
|
|
|
|
|
|
/* first some sanity checks that are unlikely to be violated */
|
|
|
|
if (slot == NULL)
|
2015-05-20 15:18:11 +02:00
|
|
|
elog(ERROR, "cannot perform logical decoding without an acquired slot");
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
if (plugin == NULL)
|
|
|
|
elog(ERROR, "cannot initialize logical decoding without a specified plugin");
|
|
|
|
|
|
|
|
/* Make sure the passed slot is suitable. These are user facing errors. */
|
2015-08-11 12:32:48 +02:00
|
|
|
if (SlotIsPhysical(slot))
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2014-11-12 02:00:58 +01:00
|
|
|
errmsg("cannot use physical replication slot for logical decoding")));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
if (slot->data.database != MyDatabaseId)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("replication slot \"%s\" was not created in this database",
|
|
|
|
NameStr(slot->data.name))));
|
|
|
|
|
|
|
|
if (IsTransactionState() &&
|
|
|
|
GetTopTransactionIdIfAny() != InvalidTransactionId)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_ACTIVE_SQL_TRANSACTION),
|
|
|
|
errmsg("cannot create logical replication slot in transaction that has performed writes")));
|
|
|
|
|
2020-08-10 18:51:31 +02:00
|
|
|
/*
|
|
|
|
* Register output plugin name with slot. We need the mutex to avoid
|
|
|
|
* concurrent reading of a partially copied string. But we don't want any
|
|
|
|
* complicated code while holding a spinlock, so do namestrcpy() outside.
|
|
|
|
*/
|
|
|
|
namestrcpy(&plugin_name, plugin);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
SpinLockAcquire(&slot->mutex);
|
2020-08-10 18:51:31 +02:00
|
|
|
slot->data.plugin = plugin_name;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
SpinLockRelease(&slot->mutex);
|
|
|
|
|
2019-04-05 19:52:45 +02:00
|
|
|
if (XLogRecPtrIsInvalid(restart_lsn))
|
|
|
|
ReplicationSlotReserveWal();
|
|
|
|
else
|
|
|
|
{
|
|
|
|
SpinLockAcquire(&slot->mutex);
|
|
|
|
slot->data.restart_lsn = restart_lsn;
|
|
|
|
SpinLockRelease(&slot->mutex);
|
|
|
|
}
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* ----
|
|
|
|
* This is a bit tricky: We need to determine a safe xmin horizon to start
|
|
|
|
* decoding from, to avoid starting from a running xacts record referring
|
|
|
|
* to xids whose rows have been vacuumed or pruned
|
|
|
|
* already. GetOldestSafeDecodingTransactionId() returns such a value, but
|
2014-05-23 14:16:09 +02:00
|
|
|
* without further interlock its return value might immediately be out of
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* date.
|
|
|
|
*
|
|
|
|
* So we have to acquire the ProcArrayLock to prevent computation of new
|
|
|
|
* xmin horizons by other backends, get the safe decoding xid, and inform
|
|
|
|
* the slot machinery about the new limit. Once that's done the
|
2014-03-17 12:47:28 +01:00
|
|
|
* ProcArrayLock can be released as the slot machinery now is
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* protecting against vacuum.
|
Preserve required !catalog tuples while computing initial decoding snapshot.
The logical decoding machinery already preserved all the required
catalog tuples, which is sufficient in the course of normal logical
decoding, but did not guarantee that non-catalog tuples were preserved
during computation of the initial snapshot when creating a slot over
the replication protocol.
This could cause a corrupted initial snapshot being exported. The
time window for issues is usually not terribly large, but on a busy
server it's perfectly possible to it hit it. Ongoing decoding is not
affected by this bug.
To avoid increased overhead for the SQL API, only retain additional
tuples when a logical slot is being created over the replication
protocol. To do so this commit changes the signature of
CreateInitDecodingContext(), but it seems unlikely that it's being
used in an extension, so that's probably ok.
In a drive-by fix, fix handling of
ReplicationSlotsComputeRequiredXmin's already_locked argument, which
should only apply to ProcArrayLock, not ReplicationSlotControlLock.
Reported-By: Erik Rijkers
Analyzed-By: Petr Jelinek
Author: Petr Jelinek, heavily editorialized by Andres Freund
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/9a897b86-46e1-9915-ee4c-da02e4ff6a95@2ndquadrant.com
Backport: 9.4, where logical decoding was introduced.
2017-04-24 05:41:29 +02:00
|
|
|
*
|
|
|
|
* Note that, temporarily, the data, not just the catalog, xmin has to be
|
|
|
|
* reserved if a data snapshot is to be exported. Otherwise the initial
|
|
|
|
* data snapshot created here is not guaranteed to be valid. After that
|
|
|
|
* the data xmin doesn't need to be managed anymore and the global xmin
|
|
|
|
* should be recomputed. As we are fine with losing the pegged data xmin
|
|
|
|
* after crash - no chance a snapshot would get exported anymore - we can
|
|
|
|
* get away with just setting the slot's
|
|
|
|
* effective_xmin. ReplicationSlotRelease will reset it again.
|
|
|
|
*
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* ----
|
|
|
|
*/
|
|
|
|
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
|
|
|
|
|
2017-08-06 05:52:53 +02:00
|
|
|
xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot);
|
Preserve required !catalog tuples while computing initial decoding snapshot.
The logical decoding machinery already preserved all the required
catalog tuples, which is sufficient in the course of normal logical
decoding, but did not guarantee that non-catalog tuples were preserved
during computation of the initial snapshot when creating a slot over
the replication protocol.
This could cause a corrupted initial snapshot being exported. The
time window for issues is usually not terribly large, but on a busy
server it's perfectly possible to it hit it. Ongoing decoding is not
affected by this bug.
To avoid increased overhead for the SQL API, only retain additional
tuples when a logical slot is being created over the replication
protocol. To do so this commit changes the signature of
CreateInitDecodingContext(), but it seems unlikely that it's being
used in an extension, so that's probably ok.
In a drive-by fix, fix handling of
ReplicationSlotsComputeRequiredXmin's already_locked argument, which
should only apply to ProcArrayLock, not ReplicationSlotControlLock.
Reported-By: Erik Rijkers
Analyzed-By: Petr Jelinek
Author: Petr Jelinek, heavily editorialized by Andres Freund
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/9a897b86-46e1-9915-ee4c-da02e4ff6a95@2ndquadrant.com
Backport: 9.4, where logical decoding was introduced.
2017-04-24 05:41:29 +02:00
|
|
|
|
2018-06-01 20:30:55 +02:00
|
|
|
SpinLockAcquire(&slot->mutex);
|
Preserve required !catalog tuples while computing initial decoding snapshot.
The logical decoding machinery already preserved all the required
catalog tuples, which is sufficient in the course of normal logical
decoding, but did not guarantee that non-catalog tuples were preserved
during computation of the initial snapshot when creating a slot over
the replication protocol.
This could cause a corrupted initial snapshot being exported. The
time window for issues is usually not terribly large, but on a busy
server it's perfectly possible to it hit it. Ongoing decoding is not
affected by this bug.
To avoid increased overhead for the SQL API, only retain additional
tuples when a logical slot is being created over the replication
protocol. To do so this commit changes the signature of
CreateInitDecodingContext(), but it seems unlikely that it's being
used in an extension, so that's probably ok.
In a drive-by fix, fix handling of
ReplicationSlotsComputeRequiredXmin's already_locked argument, which
should only apply to ProcArrayLock, not ReplicationSlotControlLock.
Reported-By: Erik Rijkers
Analyzed-By: Petr Jelinek
Author: Petr Jelinek, heavily editorialized by Andres Freund
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/9a897b86-46e1-9915-ee4c-da02e4ff6a95@2ndquadrant.com
Backport: 9.4, where logical decoding was introduced.
2017-04-24 05:41:29 +02:00
|
|
|
slot->effective_catalog_xmin = xmin_horizon;
|
|
|
|
slot->data.catalog_xmin = xmin_horizon;
|
|
|
|
if (need_full_snapshot)
|
|
|
|
slot->effective_xmin = xmin_horizon;
|
2018-06-01 20:30:55 +02:00
|
|
|
SpinLockRelease(&slot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
ReplicationSlotsComputeRequiredXmin(true);
|
|
|
|
|
|
|
|
LWLockRelease(ProcArrayLock);
|
|
|
|
|
|
|
|
ReplicationSlotMarkDirty();
|
|
|
|
ReplicationSlotSave();
|
|
|
|
|
2019-04-05 19:52:45 +02:00
|
|
|
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
|
2018-08-01 23:39:07 +02:00
|
|
|
need_full_snapshot, false,
|
2021-05-10 06:00:53 +02:00
|
|
|
xl_routine, prepare_write, do_write,
|
2018-01-17 12:38:34 +01:00
|
|
|
update_progress);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* call output plugin initialization callback */
|
|
|
|
old_context = MemoryContextSwitchTo(ctx->context);
|
|
|
|
if (ctx->callbacks.startup_cb != NULL)
|
|
|
|
startup_cb_wrapper(ctx, &ctx->options, true);
|
|
|
|
MemoryContextSwitchTo(old_context);
|
|
|
|
|
2021-03-03 02:58:43 +01:00
|
|
|
/*
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
* We allow decoding of prepared transactions when the two_phase is
|
|
|
|
* enabled at the time of slot creation, or when the two_phase option is
|
|
|
|
* given at the streaming start, provided the plugin supports all the
|
|
|
|
* callbacks for two-phase.
|
2021-03-03 02:58:43 +01:00
|
|
|
*/
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
ctx->twophase &= slot->data.two_phase;
|
2021-03-03 02:58:43 +01:00
|
|
|
|
2018-03-21 14:13:24 +01:00
|
|
|
ctx->reorder->output_rewrites = ctx->options.receive_rewrites;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
return ctx;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a new decoding context, for a logical slot that has previously been
|
|
|
|
* used already.
|
|
|
|
*
|
2016-05-02 21:04:29 +02:00
|
|
|
* start_lsn
|
|
|
|
* The LSN at which to start decoding. If InvalidXLogRecPtr, restart
|
|
|
|
* from the slot's confirmed_flush; otherwise, start from the specified
|
|
|
|
* location (but move it forwards to confirmed_flush if it's older than
|
|
|
|
* that, see below).
|
|
|
|
*
|
|
|
|
* output_plugin_options
|
2018-07-19 20:15:44 +02:00
|
|
|
* options passed to the output plugin.
|
|
|
|
*
|
|
|
|
* fast_forward
|
|
|
|
* bypass the generation of logical changes.
|
2016-05-02 21:04:29 +02:00
|
|
|
*
|
2020-05-08 21:30:34 +02:00
|
|
|
* xl_routine
|
|
|
|
* XLogReaderRoutine used by underlying xlogreader
|
|
|
|
*
|
|
|
|
* prepare_write, do_write, update_progress
|
2016-05-02 21:04:29 +02:00
|
|
|
* callbacks that have to be filled to perform the use-case dependent,
|
|
|
|
* actual work.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*
|
|
|
|
* Needs to be called while in a memory context that's at least as long lived
|
2014-03-05 19:00:22 +01:00
|
|
|
* as the decoding context because further memory contexts will be created
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* inside it.
|
|
|
|
*
|
|
|
|
* Returns an initialized decoding context after calling the output plugin's
|
|
|
|
* startup function.
|
|
|
|
*/
|
|
|
|
LogicalDecodingContext *
|
|
|
|
CreateDecodingContext(XLogRecPtr start_lsn,
|
|
|
|
List *output_plugin_options,
|
2018-01-17 12:38:34 +01:00
|
|
|
bool fast_forward,
|
2021-05-10 06:00:53 +02:00
|
|
|
XLogReaderRoutine *xl_routine,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
LogicalOutputPluginWriterPrepareWrite prepare_write,
|
2017-05-12 11:50:56 +02:00
|
|
|
LogicalOutputPluginWriterWrite do_write,
|
|
|
|
LogicalOutputPluginWriterUpdateProgress update_progress)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx;
|
|
|
|
ReplicationSlot *slot;
|
|
|
|
MemoryContext old_context;
|
|
|
|
|
|
|
|
/* shorter lines... */
|
|
|
|
slot = MyReplicationSlot;
|
|
|
|
|
|
|
|
/* first some sanity checks that are unlikely to be violated */
|
|
|
|
if (slot == NULL)
|
2015-05-20 15:18:11 +02:00
|
|
|
elog(ERROR, "cannot perform logical decoding without an acquired slot");
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* make sure the passed slot is suitable, these are user facing errors */
|
2015-08-11 12:32:48 +02:00
|
|
|
if (SlotIsPhysical(slot))
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2020-01-30 17:32:04 +01:00
|
|
|
errmsg("cannot use physical replication slot for logical decoding")));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
if (slot->data.database != MyDatabaseId)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2020-01-30 17:32:04 +01:00
|
|
|
errmsg("replication slot \"%s\" was not created in this database",
|
|
|
|
NameStr(slot->data.name))));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2023-04-08 07:19:05 +02:00
|
|
|
/*
|
|
|
|
* Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid
|
|
|
|
* "cannot get changes" wording in this errmsg because that'd be
|
|
|
|
* confusingly ambiguous about no changes being available when called from
|
|
|
|
* pg_logical_slot_get_changes_guts().
|
|
|
|
*/
|
|
|
|
if (MyReplicationSlot->data.invalidated == RS_INVAL_WAL_REMOVED)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("can no longer get changes from replication slot \"%s\"",
|
|
|
|
NameStr(MyReplicationSlot->data.name)),
|
|
|
|
errdetail("This slot has been invalidated because it exceeded the maximum reserved size.")));
|
|
|
|
|
2023-04-08 07:40:27 +02:00
|
|
|
if (MyReplicationSlot->data.invalidated != RS_INVAL_NONE)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("can no longer get changes from replication slot \"%s\"",
|
|
|
|
NameStr(MyReplicationSlot->data.name)),
|
|
|
|
errdetail("This slot has been invalidated because it was conflicting with recovery.")));
|
|
|
|
|
2023-04-08 07:19:05 +02:00
|
|
|
Assert(MyReplicationSlot->data.invalidated == RS_INVAL_NONE);
|
|
|
|
Assert(MyReplicationSlot->data.restart_lsn != InvalidXLogRecPtr);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
if (start_lsn == InvalidXLogRecPtr)
|
|
|
|
{
|
|
|
|
/* continue from last position */
|
|
|
|
start_lsn = slot->data.confirmed_flush;
|
|
|
|
}
|
|
|
|
else if (start_lsn < slot->data.confirmed_flush)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* It might seem like we should error out in this case, but it's
|
|
|
|
* pretty common for a client to acknowledge a LSN it doesn't have to
|
|
|
|
* do anything for, and thus didn't store persistently, because the
|
|
|
|
* xlog records didn't result in anything relevant for logical
|
|
|
|
* decoding. Clients have to be able to do that to support synchronous
|
|
|
|
* replication.
|
2021-07-30 23:59:19 +02:00
|
|
|
*
|
|
|
|
* Starting at a different LSN than requested might not catch certain
|
|
|
|
* kinds of client errors; so the client may wish to check that
|
|
|
|
* confirmed_flush_lsn matches its expectations.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*/
|
2021-07-30 23:59:19 +02:00
|
|
|
elog(LOG, "%X/%X has been already streamed, forwarding to %X/%X",
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(start_lsn),
|
|
|
|
LSN_FORMAT_ARGS(slot->data.confirmed_flush));
|
2015-08-05 13:26:01 +02:00
|
|
|
|
|
|
|
start_lsn = slot->data.confirmed_flush;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
ctx = StartupDecodingContext(output_plugin_options,
|
2017-04-28 00:28:24 +02:00
|
|
|
start_lsn, InvalidTransactionId, false,
|
2021-05-10 06:00:53 +02:00
|
|
|
fast_forward, xl_routine, prepare_write,
|
|
|
|
do_write, update_progress);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* call output plugin initialization callback */
|
|
|
|
old_context = MemoryContextSwitchTo(ctx->context);
|
|
|
|
if (ctx->callbacks.startup_cb != NULL)
|
2014-03-31 19:03:18 +02:00
|
|
|
startup_cb_wrapper(ctx, &ctx->options, false);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
MemoryContextSwitchTo(old_context);
|
|
|
|
|
2021-03-03 02:58:43 +01:00
|
|
|
/*
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
* We allow decoding of prepared transactions when the two_phase is
|
|
|
|
* enabled at the time of slot creation, or when the two_phase option is
|
|
|
|
* given at the streaming start, provided the plugin supports all the
|
|
|
|
* callbacks for two-phase.
|
2021-03-03 02:58:43 +01:00
|
|
|
*/
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
ctx->twophase &= (slot->data.two_phase || ctx->twophase_opt_given);
|
|
|
|
|
|
|
|
/* Mark slot to allow two_phase decoding if not already marked */
|
|
|
|
if (ctx->twophase && !slot->data.two_phase)
|
|
|
|
{
|
2023-01-12 05:40:33 +01:00
|
|
|
SpinLockAcquire(&slot->mutex);
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
slot->data.two_phase = true;
|
|
|
|
slot->data.two_phase_at = start_lsn;
|
2023-01-12 05:40:33 +01:00
|
|
|
SpinLockRelease(&slot->mutex);
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
ReplicationSlotMarkDirty();
|
|
|
|
ReplicationSlotSave();
|
|
|
|
SnapBuildSetTwoPhaseAt(ctx->snapshot_builder, start_lsn);
|
|
|
|
}
|
2021-03-03 02:58:43 +01:00
|
|
|
|
2018-03-21 14:13:24 +01:00
|
|
|
ctx->reorder->output_rewrites = ctx->options.receive_rewrites;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ereport(LOG,
|
2014-11-12 02:00:58 +01:00
|
|
|
(errmsg("starting logical decoding for slot \"%s\"",
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
NameStr(slot->data.name)),
|
2018-03-22 22:33:10 +01:00
|
|
|
errdetail("Streaming transactions committing after %X/%X, reading WAL from %X/%X.",
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(slot->data.confirmed_flush),
|
|
|
|
LSN_FORMAT_ARGS(slot->data.restart_lsn))));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
return ctx;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2015-05-20 15:18:11 +02:00
|
|
|
* Returns true if a consistent initial decoding snapshot has been built.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*/
|
|
|
|
bool
|
|
|
|
DecodingContextReady(LogicalDecodingContext *ctx)
|
|
|
|
{
|
|
|
|
return SnapBuildCurrentState(ctx->snapshot_builder) == SNAPBUILD_CONSISTENT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Read from the decoding slot, until it is ready to start extracting changes.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
|
|
|
|
{
|
2018-06-01 20:30:55 +02:00
|
|
|
ReplicationSlot *slot = ctx->slot;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* Initialize from where to start reading WAL. */
|
2020-01-26 10:39:00 +01:00
|
|
|
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
elog(DEBUG1, "searching for logical decoding starting point, starting at %X/%X",
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(slot->data.restart_lsn));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* Wait for a consistent starting point */
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
XLogRecord *record;
|
|
|
|
char *err = NULL;
|
|
|
|
|
|
|
|
/* the read_page callback waits for new WAL */
|
2021-05-10 06:00:53 +02:00
|
|
|
record = XLogReadRecord(ctx->reader, &err);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
if (err)
|
2021-11-10 04:00:33 +01:00
|
|
|
elog(ERROR, "could not find logical decoding starting point: %s", err);
|
Revamp the WAL record format.
Each WAL record now carries information about the modified relation and
block(s) in a standardized format. That makes it easier to write tools that
need that information, like pg_rewind, prefetching the blocks to speed up
recovery, etc.
There's a whole new API for building WAL records, replacing the XLogRecData
chains used previously. The new API consists of XLogRegister* functions,
which are called for each buffer and chunk of data that is added to the
record. The new API also gives more control over when a full-page image is
written, by passing flags to the XLogRegisterBuffer function.
This also simplifies the XLogReadBufferForRedo() calls. The function can dig
the relation and block number from the WAL record, so they no longer need to
be passed as arguments.
For the convenience of redo routines, XLogReader now disects each WAL record
after reading it, copying the main data part and the per-block data into
MAXALIGNed buffers. The data chunks are not aligned within the WAL record,
but the redo routines can assume that the pointers returned by XLogRecGet*
functions are. Redo routines are now passed the XLogReaderState, which
contains the record in the already-disected format, instead of the plain
XLogRecord.
The new record format also makes the fixed size XLogRecord header smaller,
by removing the xl_len field. The length of the "main data" portion is now
stored at the end of the WAL record, and there's a separate header after
XLogRecord for it. The alignment padding at the end of XLogRecord is also
removed. This compansates for the fact that the new format would otherwise
be more bulky than the old format.
Reviewed by Andres Freund, Amit Kapila, Michael Paquier, Alvaro Herrera,
Fujii Masao.
2014-11-20 16:56:26 +01:00
|
|
|
if (!record)
|
2021-11-10 04:00:33 +01:00
|
|
|
elog(ERROR, "could not find logical decoding starting point");
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
Revamp the WAL record format.
Each WAL record now carries information about the modified relation and
block(s) in a standardized format. That makes it easier to write tools that
need that information, like pg_rewind, prefetching the blocks to speed up
recovery, etc.
There's a whole new API for building WAL records, replacing the XLogRecData
chains used previously. The new API consists of XLogRegister* functions,
which are called for each buffer and chunk of data that is added to the
record. The new API also gives more control over when a full-page image is
written, by passing flags to the XLogRegisterBuffer function.
This also simplifies the XLogReadBufferForRedo() calls. The function can dig
the relation and block number from the WAL record, so they no longer need to
be passed as arguments.
For the convenience of redo routines, XLogReader now disects each WAL record
after reading it, copying the main data part and the per-block data into
MAXALIGNed buffers. The data chunks are not aligned within the WAL record,
but the redo routines can assume that the pointers returned by XLogRecGet*
functions are. Redo routines are now passed the XLogReaderState, which
contains the record in the already-disected format, instead of the plain
XLogRecord.
The new record format also makes the fixed size XLogRecord header smaller,
by removing the xl_len field. The length of the "main data" portion is now
stored at the end of the WAL record, and there's a separate header after
XLogRecord for it. The alignment padding at the end of XLogRecord is also
removed. This compansates for the fact that the new format would otherwise
be more bulky than the old format.
Reviewed by Andres Freund, Amit Kapila, Michael Paquier, Alvaro Herrera,
Fujii Masao.
2014-11-20 16:56:26 +01:00
|
|
|
LogicalDecodingProcessRecord(ctx, ctx->reader);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* only continue till we found a consistent spot */
|
|
|
|
if (DecodingContextReady(ctx))
|
|
|
|
break;
|
2014-06-29 17:08:04 +02:00
|
|
|
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
|
2018-06-01 20:30:55 +02:00
|
|
|
SpinLockAcquire(&slot->mutex);
|
|
|
|
slot->data.confirmed_flush = ctx->reader->EndRecPtr;
|
Add support for prepared transactions to built-in logical replication.
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:
* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.
* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.
We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.
The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.
We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.
Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
2021-07-14 04:03:50 +02:00
|
|
|
if (slot->data.two_phase)
|
|
|
|
slot->data.two_phase_at = ctx->reader->EndRecPtr;
|
2018-06-01 20:30:55 +02:00
|
|
|
SpinLockRelease(&slot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Free a previously allocated decoding context, invoking the shutdown
|
|
|
|
* callback if necessary.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
FreeDecodingContext(LogicalDecodingContext *ctx)
|
|
|
|
{
|
|
|
|
if (ctx->callbacks.shutdown_cb != NULL)
|
|
|
|
shutdown_cb_wrapper(ctx);
|
|
|
|
|
|
|
|
ReorderBufferFree(ctx->reorder);
|
|
|
|
FreeSnapshotBuilder(ctx->snapshot_builder);
|
|
|
|
XLogReaderFree(ctx->reader);
|
|
|
|
MemoryContextDelete(ctx->context);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Prepare a write using the context's output routine.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write)
|
|
|
|
{
|
|
|
|
if (!ctx->accept_writes)
|
|
|
|
elog(ERROR, "writes are only accepted in commit, begin and change callbacks");
|
|
|
|
|
|
|
|
ctx->prepare_write(ctx, ctx->write_location, ctx->write_xid, last_write);
|
|
|
|
ctx->prepared_write = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Perform a write using the context's output routine.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
|
|
|
|
{
|
|
|
|
if (!ctx->prepared_write)
|
|
|
|
elog(ERROR, "OutputPluginPrepareWrite needs to be called before OutputPluginWrite");
|
|
|
|
|
|
|
|
ctx->write(ctx, ctx->write_location, ctx->write_xid, last_write);
|
|
|
|
ctx->prepared_write = false;
|
|
|
|
}
|
|
|
|
|
2017-05-12 11:50:56 +02:00
|
|
|
/*
|
|
|
|
* Update progress tracking (if supported).
|
|
|
|
*/
|
|
|
|
void
|
Skip empty transactions for logical replication.
The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty. This can happen because
transaction doesn't contain changes from the selected publications or all
the changes got filtered. It is a waste of CPU cycles and network
bandwidth to build/transmit these empty transactions.
This patch addresses the above problem by postponing the BEGIN message
until the first change is sent. While processing a COMMIT message, if
there was no other change for that transaction, do not send the COMMIT
message. This allows us to skip sending BEGIN/COMMIT messages for empty
transactions.
When skipping empty transactions in synchronous replication mode, we send
a keepalive message to avoid delaying such transactions.
Author: Ajin Cherian, Hou Zhijie, Euler Taveira
Reviewed-by: Peter Smith, Takamichi Osumi, Shi Yu, Masahiko Sawada, Greg Nancarrow, Vignesh C, Amit Kapila
Discussion: https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
2022-03-30 04:11:05 +02:00
|
|
|
OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
|
|
|
|
bool skipped_xact)
|
2017-05-12 11:50:56 +02:00
|
|
|
{
|
|
|
|
if (!ctx->update_progress)
|
|
|
|
return;
|
|
|
|
|
Skip empty transactions for logical replication.
The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty. This can happen because
transaction doesn't contain changes from the selected publications or all
the changes got filtered. It is a waste of CPU cycles and network
bandwidth to build/transmit these empty transactions.
This patch addresses the above problem by postponing the BEGIN message
until the first change is sent. While processing a COMMIT message, if
there was no other change for that transaction, do not send the COMMIT
message. This allows us to skip sending BEGIN/COMMIT messages for empty
transactions.
When skipping empty transactions in synchronous replication mode, we send
a keepalive message to avoid delaying such transactions.
Author: Ajin Cherian, Hou Zhijie, Euler Taveira
Reviewed-by: Peter Smith, Takamichi Osumi, Shi Yu, Masahiko Sawada, Greg Nancarrow, Vignesh C, Amit Kapila
Discussion: https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
2022-03-30 04:11:05 +02:00
|
|
|
ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
|
|
|
|
skipped_xact);
|
2017-05-12 11:50:56 +02:00
|
|
|
}
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Load the output plugin, lookup its output plugin init function, and check
|
|
|
|
* that it provides the required callbacks.
|
|
|
|
*/
|
|
|
|
static void
|
2020-08-08 07:31:52 +02:00
|
|
|
LoadOutputPlugin(OutputPluginCallbacks *callbacks, const char *plugin)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
LogicalOutputPluginInit plugin_init;
|
|
|
|
|
|
|
|
plugin_init = (LogicalOutputPluginInit)
|
|
|
|
load_external_function(plugin, "_PG_output_plugin_init", false, NULL);
|
|
|
|
|
|
|
|
if (plugin_init == NULL)
|
|
|
|
elog(ERROR, "output plugins have to declare the _PG_output_plugin_init symbol");
|
|
|
|
|
|
|
|
/* ask the output plugin to fill the callback struct */
|
|
|
|
plugin_init(callbacks);
|
|
|
|
|
|
|
|
if (callbacks->begin_cb == NULL)
|
|
|
|
elog(ERROR, "output plugins have to register a begin callback");
|
|
|
|
if (callbacks->change_cb == NULL)
|
|
|
|
elog(ERROR, "output plugins have to register a change callback");
|
|
|
|
if (callbacks->commit_cb == NULL)
|
|
|
|
elog(ERROR, "output plugins have to register a commit callback");
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
output_plugin_error_callback(void *arg)
|
|
|
|
{
|
|
|
|
LogicalErrorCallbackState *state = (LogicalErrorCallbackState *) arg;
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* not all callbacks have an associated LSN */
|
|
|
|
if (state->report_location != InvalidXLogRecPtr)
|
|
|
|
errcontext("slot \"%s\", output plugin \"%s\", in the %s callback, associated LSN %X/%X",
|
|
|
|
NameStr(state->ctx->slot->data.name),
|
|
|
|
NameStr(state->ctx->slot->data.plugin),
|
|
|
|
state->callback_name,
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(state->report_location));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
else
|
|
|
|
errcontext("slot \"%s\", output plugin \"%s\", in the %s callback",
|
|
|
|
NameStr(state->ctx->slot->data.name),
|
|
|
|
NameStr(state->ctx->slot->data.plugin),
|
|
|
|
state->callback_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
startup_cb_wrapper(LogicalDecodingContext *ctx, OutputPluginOptions *opt, bool is_init)
|
|
|
|
{
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "startup";
|
|
|
|
state.report_location = InvalidXLogRecPtr;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = false;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.startup_cb(ctx, opt, is_init);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
shutdown_cb_wrapper(LogicalDecodingContext *ctx)
|
|
|
|
{
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "shutdown";
|
|
|
|
state.report_location = InvalidXLogRecPtr;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = false;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.shutdown_cb(ctx);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Callbacks for ReorderBuffer which add in some more information and then call
|
|
|
|
* output_plugin.h plugins.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
begin_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "begin";
|
|
|
|
state.report_location = txn->first_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->first_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.begin_cb(ctx, txn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "commit";
|
|
|
|
state.report_location = txn->final_lsn; /* beginning of commit record */
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn; /* points to the end of the record */
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.commit_cb(ctx, txn, commit_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
/*
|
|
|
|
* The functionality of begin_prepare is quite similar to begin with the
|
|
|
|
* exception that this will have gid (global transaction id) information which
|
|
|
|
* can be used by plugin. Now, we thought about extending the existing begin
|
|
|
|
* but that would break the replication protocol and additionally this looks
|
|
|
|
* cleaner.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when two-phase commits are supported */
|
|
|
|
Assert(ctx->twophase);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "begin_prepare";
|
|
|
|
state.report_location = txn->first_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->first_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the plugin supports two-phase commits then begin prepare callback is
|
|
|
|
* mandatory
|
|
|
|
*/
|
|
|
|
if (ctx->callbacks.begin_prepare_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical replication at prepare time requires a %s callback",
|
|
|
|
"begin_prepare_cb")));
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.begin_prepare_cb(ctx, txn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when two-phase commits are supported */
|
|
|
|
Assert(ctx->twophase);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "prepare";
|
|
|
|
state.report_location = txn->final_lsn; /* beginning of prepare record */
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn; /* points to the end of the record */
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the plugin supports two-phase commits then prepare callback is
|
|
|
|
* mandatory
|
|
|
|
*/
|
|
|
|
if (ctx->callbacks.prepare_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical replication at prepare time requires a %s callback",
|
|
|
|
"prepare_cb")));
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.prepare_cb(ctx, txn, prepare_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when two-phase commits are supported */
|
|
|
|
Assert(ctx->twophase);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "commit_prepared";
|
|
|
|
state.report_location = txn->final_lsn; /* beginning of commit record */
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn; /* points to the end of the record */
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the plugin support two-phase commits then commit prepared callback
|
|
|
|
* is mandatory
|
|
|
|
*/
|
|
|
|
if (ctx->callbacks.commit_prepared_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical replication at prepare time requires a %s callback",
|
|
|
|
"commit_prepared_cb")));
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_end_lsn,
|
|
|
|
TimestampTz prepare_time)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when two-phase commits are supported */
|
|
|
|
Assert(ctx->twophase);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "rollback_prepared";
|
|
|
|
state.report_location = txn->final_lsn; /* beginning of commit record */
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn; /* points to the end of the record */
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the plugin support two-phase commits then rollback prepared callback
|
|
|
|
* is mandatory
|
|
|
|
*/
|
|
|
|
if (ctx->callbacks.rollback_prepared_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical replication at prepare time requires a %s callback",
|
|
|
|
"rollback_prepared_cb")));
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.rollback_prepared_cb(ctx, txn, prepare_end_lsn,
|
|
|
|
prepare_time);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
static void
|
|
|
|
change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
Relation relation, ReorderBufferChange *change)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "change";
|
|
|
|
state.report_location = change->lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this change's lsn so replies from clients can give an up-to-date
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* answer. This won't ever be enough (and shouldn't be!) to confirm
|
|
|
|
* receipt of this transaction, but it might allow another transaction's
|
|
|
|
* commit to be confirmed with one message.
|
|
|
|
*/
|
|
|
|
ctx->write_location = change->lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ctx->callbacks.change_cb(ctx, txn, relation, change);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2018-04-07 17:17:56 +02:00
|
|
|
static void
|
|
|
|
truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
int nrelations, Relation relations[], ReorderBufferChange *change)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
if (!ctx->callbacks.truncate_cb)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "truncate";
|
|
|
|
state.report_location = change->lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this change's lsn so replies from clients can give an up-to-date
|
2018-04-07 17:17:56 +02:00
|
|
|
* answer. This won't ever be enough (and shouldn't be!) to confirm
|
|
|
|
* receipt of this transaction, but it might allow another transaction's
|
|
|
|
* commit to be confirmed with one message.
|
|
|
|
*/
|
|
|
|
ctx->write_location = change->lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
2018-04-07 17:17:56 +02:00
|
|
|
ctx->callbacks.truncate_cb(ctx, txn, nrelations, relations, change);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
bool
|
2021-03-30 07:04:43 +02:00
|
|
|
filter_prepare_cb_wrapper(LogicalDecodingContext *ctx, TransactionId xid,
|
|
|
|
const char *gid)
|
2020-12-30 11:47:26 +01:00
|
|
|
{
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
bool ret;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "filter_prepare";
|
|
|
|
state.report_location = InvalidXLogRecPtr;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = false;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
2021-03-30 07:04:43 +02:00
|
|
|
ret = ctx->callbacks.filter_prepare_cb(ctx, xid, gid);
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
bool
|
|
|
|
filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
|
|
|
|
{
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
bool ret;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
2015-12-18 18:17:35 +01:00
|
|
|
state.callback_name = "filter_by_origin";
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
state.report_location = InvalidXLogRecPtr;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = false;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ret = ctx->callbacks.filter_by_origin_cb(ctx, origin_id);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-04-06 11:05:41 +02:00
|
|
|
static void
|
|
|
|
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn, bool transactional,
|
|
|
|
const char *prefix, Size message_size, const char *message)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
2018-01-17 12:38:34 +01:00
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
2016-04-06 11:05:41 +02:00
|
|
|
if (ctx->callbacks.message_cb == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "message";
|
|
|
|
state.report_location = message_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
|
|
|
|
ctx->write_location = message_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
2016-04-06 11:05:41 +02:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.message_cb(ctx, txn, message_lsn, transactional, prefix,
|
|
|
|
message_size, message);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
static void
|
|
|
|
stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr first_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_start";
|
|
|
|
state.report_location = first_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this message's lsn so replies from clients can give an
|
|
|
|
* up-to-date answer. This won't ever be enough (and shouldn't be!) to
|
|
|
|
* confirm receipt of this transaction, but it might allow another
|
|
|
|
* transaction's commit to be confirmed with one message.
|
2020-07-28 04:36:44 +02:00
|
|
|
*/
|
|
|
|
ctx->write_location = first_lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/* in streaming mode, stream_start_cb is required */
|
|
|
|
if (ctx->callbacks.stream_start_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming requires a %s callback",
|
|
|
|
"stream_start_cb")));
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
ctx->callbacks.stream_start_cb(ctx, txn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr last_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_stop";
|
|
|
|
state.report_location = last_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this message's lsn so replies from clients can give an
|
|
|
|
* up-to-date answer. This won't ever be enough (and shouldn't be!) to
|
|
|
|
* confirm receipt of this transaction, but it might allow another
|
|
|
|
* transaction's commit to be confirmed with one message.
|
2020-07-28 04:36:44 +02:00
|
|
|
*/
|
|
|
|
ctx->write_location = last_lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/* in streaming mode, stream_stop_cb is required */
|
|
|
|
if (ctx->callbacks.stream_stop_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming requires a %s callback",
|
|
|
|
"stream_stop_cb")));
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
ctx->callbacks.stream_stop_cb(ctx, txn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr abort_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_abort";
|
|
|
|
state.report_location = abort_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = abort_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
/* in streaming mode, stream_abort_cb is required */
|
|
|
|
if (ctx->callbacks.stream_abort_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming requires a %s callback",
|
|
|
|
"stream_abort_cb")));
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2020-12-30 11:47:26 +01:00
|
|
|
static void
|
|
|
|
stream_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr prepare_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We're only supposed to call this when streaming and two-phase commits
|
|
|
|
* are supported.
|
|
|
|
*/
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
Assert(ctx->twophase);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_prepare";
|
|
|
|
state.report_location = txn->final_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
/* in streaming mode with two-phase commits, stream_prepare_cb is required */
|
|
|
|
if (ctx->callbacks.stream_prepare_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming at prepare time requires a %s callback",
|
|
|
|
"stream_prepare_cb")));
|
2020-12-30 11:47:26 +01:00
|
|
|
|
|
|
|
ctx->callbacks.stream_prepare_cb(ctx, txn, prepare_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
static void
|
|
|
|
stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr commit_lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_commit";
|
|
|
|
state.report_location = txn->final_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
ctx->write_location = txn->end_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = true;
|
2020-07-28 04:36:44 +02:00
|
|
|
|
2021-01-06 04:26:19 +01:00
|
|
|
/* in streaming mode, stream_commit_cb is required */
|
2020-07-28 04:36:44 +02:00
|
|
|
if (ctx->callbacks.stream_commit_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming requires a %s callback",
|
|
|
|
"stream_commit_cb")));
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
ctx->callbacks.stream_commit_cb(ctx, txn, commit_lsn);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
Relation relation, ReorderBufferChange *change)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_change";
|
|
|
|
state.report_location = change->lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this change's lsn so replies from clients can give an up-to-date
|
2020-07-28 04:36:44 +02:00
|
|
|
* answer. This won't ever be enough (and shouldn't be!) to confirm
|
|
|
|
* receipt of this transaction, but it might allow another transaction's
|
|
|
|
* commit to be confirmed with one message.
|
|
|
|
*/
|
|
|
|
ctx->write_location = change->lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
/* in streaming mode, stream_change_cb is required */
|
|
|
|
if (ctx->callbacks.stream_change_cb == NULL)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
2021-06-27 09:41:16 +02:00
|
|
|
errmsg("logical streaming requires a %s callback",
|
|
|
|
"stream_change_cb")));
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn, bool transactional,
|
|
|
|
const char *prefix, Size message_size, const char *message)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* this callback is optional */
|
|
|
|
if (ctx->callbacks.stream_message_cb == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_message";
|
|
|
|
state.report_location = message_lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
|
|
|
|
ctx->write_location = message_lsn;
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
2020-07-28 04:36:44 +02:00
|
|
|
|
|
|
|
/* do the actual work: call callback */
|
|
|
|
ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
|
|
|
|
message_size, message);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
int nrelations, Relation relations[],
|
|
|
|
ReorderBufferChange *change)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* We're only supposed to call this when streaming is supported. */
|
|
|
|
Assert(ctx->streaming);
|
|
|
|
|
|
|
|
/* this callback is optional */
|
|
|
|
if (!ctx->callbacks.stream_truncate_cb)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "stream_truncate";
|
|
|
|
state.report_location = change->lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = true;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
2021-07-28 03:31:24 +02:00
|
|
|
* Report this change's lsn so replies from clients can give an up-to-date
|
2020-07-28 04:36:44 +02:00
|
|
|
* answer. This won't ever be enough (and shouldn't be!) to confirm
|
|
|
|
* receipt of this transaction, but it might allow another transaction's
|
|
|
|
* commit to be confirmed with one message.
|
|
|
|
*/
|
|
|
|
ctx->write_location = change->lsn;
|
|
|
|
|
2022-05-11 07:41:44 +02:00
|
|
|
ctx->end_xact = false;
|
|
|
|
|
2020-07-28 04:36:44 +02:00
|
|
|
ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
2023-02-08 03:28:25 +01:00
|
|
|
static void
|
|
|
|
update_progress_txn_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr lsn)
|
|
|
|
{
|
|
|
|
LogicalDecodingContext *ctx = cache->private_data;
|
|
|
|
LogicalErrorCallbackState state;
|
|
|
|
ErrorContextCallback errcallback;
|
|
|
|
|
|
|
|
Assert(!ctx->fast_forward);
|
|
|
|
|
|
|
|
/* Push callback + info on the error context stack */
|
|
|
|
state.ctx = ctx;
|
|
|
|
state.callback_name = "update_progress_txn";
|
|
|
|
state.report_location = lsn;
|
|
|
|
errcallback.callback = output_plugin_error_callback;
|
|
|
|
errcallback.arg = (void *) &state;
|
|
|
|
errcallback.previous = error_context_stack;
|
|
|
|
error_context_stack = &errcallback;
|
|
|
|
|
|
|
|
/* set output state */
|
|
|
|
ctx->accept_writes = false;
|
|
|
|
ctx->write_xid = txn->xid;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Report this change's lsn so replies from clients can give an up-to-date
|
|
|
|
* answer. This won't ever be enough (and shouldn't be!) to confirm
|
|
|
|
* receipt of this transaction, but it might allow another transaction's
|
|
|
|
* commit to be confirmed with one message.
|
|
|
|
*/
|
|
|
|
ctx->write_location = lsn;
|
|
|
|
|
|
|
|
ctx->end_xact = false;
|
|
|
|
|
|
|
|
OutputPluginUpdateProgress(ctx, false);
|
|
|
|
|
|
|
|
/* Pop the error context stack */
|
|
|
|
error_context_stack = errcallback.previous;
|
|
|
|
}
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Set the required catalog xmin horizon for historic snapshots in the current
|
|
|
|
* replication slot.
|
|
|
|
*
|
|
|
|
* Note that in the most cases, we won't be able to immediately use the xmin
|
2016-05-02 21:04:29 +02:00
|
|
|
* to increase the xmin horizon: we need to wait till the client has confirmed
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* receiving current_lsn with LogicalConfirmReceivedLocation().
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalIncreaseXminForSlot(XLogRecPtr current_lsn, TransactionId xmin)
|
|
|
|
{
|
|
|
|
bool updated_xmin = false;
|
|
|
|
ReplicationSlot *slot;
|
2021-09-07 04:37:11 +02:00
|
|
|
bool got_new_xmin = false;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
slot = MyReplicationSlot;
|
|
|
|
|
|
|
|
Assert(slot != NULL);
|
|
|
|
|
|
|
|
SpinLockAcquire(&slot->mutex);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* don't overwrite if we already have a newer xmin. This can happen if we
|
|
|
|
* restart decoding in a slot.
|
|
|
|
*/
|
|
|
|
if (TransactionIdPrecedesOrEquals(xmin, slot->data.catalog_xmin))
|
|
|
|
{
|
|
|
|
}
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* If the client has already confirmed up to this lsn, we directly can
|
|
|
|
* mark this as accepted. This can happen if we restart decoding in a
|
|
|
|
* slot.
|
|
|
|
*/
|
|
|
|
else if (current_lsn <= slot->data.confirmed_flush)
|
|
|
|
{
|
|
|
|
slot->candidate_catalog_xmin = xmin;
|
|
|
|
slot->candidate_xmin_lsn = current_lsn;
|
|
|
|
|
|
|
|
/* our candidate can directly be used */
|
|
|
|
updated_xmin = true;
|
|
|
|
}
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Only increase if the previous values have been applied, otherwise we
|
|
|
|
* might never end up updating if the receiver acks too slowly.
|
|
|
|
*/
|
|
|
|
else if (slot->candidate_xmin_lsn == InvalidXLogRecPtr)
|
|
|
|
{
|
|
|
|
slot->candidate_catalog_xmin = xmin;
|
|
|
|
slot->candidate_xmin_lsn = current_lsn;
|
2021-09-07 04:37:11 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Log new xmin at an appropriate log level after releasing the
|
|
|
|
* spinlock.
|
|
|
|
*/
|
|
|
|
got_new_xmin = true;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
SpinLockRelease(&slot->mutex);
|
|
|
|
|
2021-09-07 04:37:11 +02:00
|
|
|
if (got_new_xmin)
|
|
|
|
elog(DEBUG1, "got new catalog xmin %u at %X/%X", xmin,
|
|
|
|
LSN_FORMAT_ARGS(current_lsn));
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/* candidate already valid with the current flush position, apply */
|
|
|
|
if (updated_xmin)
|
|
|
|
LogicalConfirmReceivedLocation(slot->data.confirmed_flush);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Mark the minimal LSN (restart_lsn) we need to read to replay all
|
|
|
|
* transactions that have not yet committed at current_lsn.
|
|
|
|
*
|
2018-07-02 06:30:12 +02:00
|
|
|
* Just like LogicalIncreaseXminForSlot this only takes effect when the
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* client has confirmed to have received current_lsn.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalIncreaseRestartDecodingForSlot(XLogRecPtr current_lsn, XLogRecPtr restart_lsn)
|
|
|
|
{
|
|
|
|
bool updated_lsn = false;
|
|
|
|
ReplicationSlot *slot;
|
|
|
|
|
|
|
|
slot = MyReplicationSlot;
|
|
|
|
|
|
|
|
Assert(slot != NULL);
|
|
|
|
Assert(restart_lsn != InvalidXLogRecPtr);
|
|
|
|
Assert(current_lsn != InvalidXLogRecPtr);
|
|
|
|
|
|
|
|
SpinLockAcquire(&slot->mutex);
|
|
|
|
|
|
|
|
/* don't overwrite if have a newer restart lsn */
|
|
|
|
if (restart_lsn <= slot->data.restart_lsn)
|
|
|
|
{
|
|
|
|
}
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* We might have already flushed far enough to directly accept this lsn,
|
|
|
|
* in this case there is no need to check for existing candidate LSNs
|
|
|
|
*/
|
|
|
|
else if (current_lsn <= slot->data.confirmed_flush)
|
|
|
|
{
|
|
|
|
slot->candidate_restart_valid = current_lsn;
|
|
|
|
slot->candidate_restart_lsn = restart_lsn;
|
|
|
|
|
|
|
|
/* our candidate can directly be used */
|
|
|
|
updated_lsn = true;
|
|
|
|
}
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Only increase if the previous values have been applied, otherwise we
|
|
|
|
* might never end up updating if the receiver acks too slowly. A missed
|
|
|
|
* value here will just cause some extra effort after reconnecting.
|
|
|
|
*/
|
|
|
|
if (slot->candidate_restart_valid == InvalidXLogRecPtr)
|
|
|
|
{
|
|
|
|
slot->candidate_restart_valid = current_lsn;
|
|
|
|
slot->candidate_restart_lsn = restart_lsn;
|
2020-06-04 03:17:49 +02:00
|
|
|
SpinLockRelease(&slot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
elog(DEBUG1, "got new restart lsn %X/%X at %X/%X",
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(restart_lsn),
|
|
|
|
LSN_FORMAT_ARGS(current_lsn));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2020-06-04 03:17:49 +02:00
|
|
|
XLogRecPtr candidate_restart_lsn;
|
|
|
|
XLogRecPtr candidate_restart_valid;
|
|
|
|
XLogRecPtr confirmed_flush;
|
|
|
|
|
|
|
|
candidate_restart_lsn = slot->candidate_restart_lsn;
|
|
|
|
candidate_restart_valid = slot->candidate_restart_valid;
|
|
|
|
confirmed_flush = slot->data.confirmed_flush;
|
|
|
|
SpinLockRelease(&slot->mutex);
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
elog(DEBUG1, "failed to increase restart lsn: proposed %X/%X, after %X/%X, current candidate %X/%X, current after %X/%X, flushed up to %X/%X",
|
2021-02-23 10:14:38 +01:00
|
|
|
LSN_FORMAT_ARGS(restart_lsn),
|
|
|
|
LSN_FORMAT_ARGS(current_lsn),
|
|
|
|
LSN_FORMAT_ARGS(candidate_restart_lsn),
|
|
|
|
LSN_FORMAT_ARGS(candidate_restart_valid),
|
|
|
|
LSN_FORMAT_ARGS(confirmed_flush));
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/* candidates are already valid with the current flush position, apply */
|
|
|
|
if (updated_lsn)
|
|
|
|
LogicalConfirmReceivedLocation(slot->data.confirmed_flush);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2016-07-18 18:46:57 +02:00
|
|
|
* Handle a consumer's confirmation having received all changes up to lsn.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalConfirmReceivedLocation(XLogRecPtr lsn)
|
|
|
|
{
|
|
|
|
Assert(lsn != InvalidXLogRecPtr);
|
|
|
|
|
|
|
|
/* Do an unlocked check for candidate_lsn first. */
|
|
|
|
if (MyReplicationSlot->candidate_xmin_lsn != InvalidXLogRecPtr ||
|
|
|
|
MyReplicationSlot->candidate_restart_valid != InvalidXLogRecPtr)
|
|
|
|
{
|
|
|
|
bool updated_xmin = false;
|
|
|
|
bool updated_restart = false;
|
|
|
|
|
2015-10-06 21:45:02 +02:00
|
|
|
SpinLockAcquire(&MyReplicationSlot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2015-10-06 21:45:02 +02:00
|
|
|
MyReplicationSlot->data.confirmed_flush = lsn;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2016-05-02 21:04:29 +02:00
|
|
|
/* if we're past the location required for bumping xmin, do so */
|
2015-10-06 21:45:02 +02:00
|
|
|
if (MyReplicationSlot->candidate_xmin_lsn != InvalidXLogRecPtr &&
|
|
|
|
MyReplicationSlot->candidate_xmin_lsn <= lsn)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We have to write the changed xmin to disk *before* we change
|
|
|
|
* the in-memory value, otherwise after a crash we wouldn't know
|
|
|
|
* that some catalog tuples might have been removed already.
|
|
|
|
*
|
|
|
|
* Ensure that by first writing to ->xmin and only update
|
|
|
|
* ->effective_xmin once the new state is synced to disk. After a
|
|
|
|
* crash ->effective_xmin is set to ->xmin.
|
|
|
|
*/
|
2015-10-06 21:45:02 +02:00
|
|
|
if (TransactionIdIsValid(MyReplicationSlot->candidate_catalog_xmin) &&
|
|
|
|
MyReplicationSlot->data.catalog_xmin != MyReplicationSlot->candidate_catalog_xmin)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
2015-10-06 21:45:02 +02:00
|
|
|
MyReplicationSlot->data.catalog_xmin = MyReplicationSlot->candidate_catalog_xmin;
|
|
|
|
MyReplicationSlot->candidate_catalog_xmin = InvalidTransactionId;
|
|
|
|
MyReplicationSlot->candidate_xmin_lsn = InvalidXLogRecPtr;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
updated_xmin = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-06 21:45:02 +02:00
|
|
|
if (MyReplicationSlot->candidate_restart_valid != InvalidXLogRecPtr &&
|
|
|
|
MyReplicationSlot->candidate_restart_valid <= lsn)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
2015-10-06 21:45:02 +02:00
|
|
|
Assert(MyReplicationSlot->candidate_restart_lsn != InvalidXLogRecPtr);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2015-10-06 21:45:02 +02:00
|
|
|
MyReplicationSlot->data.restart_lsn = MyReplicationSlot->candidate_restart_lsn;
|
|
|
|
MyReplicationSlot->candidate_restart_lsn = InvalidXLogRecPtr;
|
|
|
|
MyReplicationSlot->candidate_restart_valid = InvalidXLogRecPtr;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
updated_restart = true;
|
|
|
|
}
|
|
|
|
|
2015-10-06 21:45:02 +02:00
|
|
|
SpinLockRelease(&MyReplicationSlot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2016-05-02 21:04:29 +02:00
|
|
|
/* first write new xmin to disk, so we know what's up after a crash */
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
if (updated_xmin || updated_restart)
|
|
|
|
{
|
|
|
|
ReplicationSlotMarkDirty();
|
|
|
|
ReplicationSlotSave();
|
|
|
|
elog(DEBUG1, "updated xmin: %u restart: %u", updated_xmin, updated_restart);
|
|
|
|
}
|
2014-05-06 18:12:18 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
/*
|
|
|
|
* Now the new xmin is safely on disk, we can let the global value
|
|
|
|
* advance. We do not take ProcArrayLock or similar since we only
|
|
|
|
* advance xmin here and there's not much harm done by a concurrent
|
|
|
|
* computation missing that.
|
|
|
|
*/
|
|
|
|
if (updated_xmin)
|
|
|
|
{
|
2015-10-06 21:45:02 +02:00
|
|
|
SpinLockAcquire(&MyReplicationSlot->mutex);
|
|
|
|
MyReplicationSlot->effective_catalog_xmin = MyReplicationSlot->data.catalog_xmin;
|
|
|
|
SpinLockRelease(&MyReplicationSlot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
ReplicationSlotsComputeRequiredXmin(false);
|
|
|
|
ReplicationSlotsComputeRequiredLSN();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2015-10-06 21:45:02 +02:00
|
|
|
SpinLockAcquire(&MyReplicationSlot->mutex);
|
|
|
|
MyReplicationSlot->data.confirmed_flush = lsn;
|
|
|
|
SpinLockRelease(&MyReplicationSlot->mutex);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
|
|
|
}
|
Implement streaming mode in ReorderBuffer.
Instead of serializing the transaction to disk after reaching the
logical_decoding_work_mem limit in memory, we consume the changes we have
in memory and invoke stream API methods added by commit 45fdc9738b.
However, sometimes if we have incomplete toast or speculative insert we
spill to the disk because we can't generate the complete tuple and stream.
And, as soon as we get the complete tuple we stream the transaction
including the serialized changes.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages at each command end. These
features are added by commits 0bead9af48 and c55040ccd0 respectively.
Now that we can stream in-progress transactions, the concurrent aborts
may cause failures when the output plugin consults catalogs (both system
and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend or WALSender
decoding a specific uncommitted transaction. The decoding logic on the
receipt of such a sqlerrcode aborts the decoding of the current
transaction and continue with the decoding of other transactions.
We have ReorderBufferTXN pointer in each ReorderBufferChange by which we
know which xact it belongs to. The output plugin can use this to decide
which changes to discard in case of stream_abort_cb (e.g. when a subxact
gets discarded).
We also provide a new option via SQL APIs to fetch the changes being
streamed.
Author: Dilip Kumar, Tomas Vondra, Amit Kapila, Nikhil Sontakke
Reviewed-by: Amit Kapila, Kuntal Ghosh, Ajin Cherian
Tested-by: Neha Sharma, Mahendra Singh Thalor and Ajin Cherian
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
2020-08-08 04:04:39 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Clear logical streaming state during (sub)transaction abort.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
ResetLogicalStreamingState(void)
|
|
|
|
{
|
|
|
|
CheckXidAlive = InvalidTransactionId;
|
|
|
|
bsysscan = false;
|
|
|
|
}
|
2020-10-08 05:39:08 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Report stats for a slot.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
UpdateDecodingStats(LogicalDecodingContext *ctx)
|
|
|
|
{
|
|
|
|
ReorderBuffer *rb = ctx->reorder;
|
2021-04-27 05:39:11 +02:00
|
|
|
PgStat_StatReplSlotEntry repSlotStat;
|
2020-10-08 05:39:08 +02:00
|
|
|
|
2021-04-16 04:04:43 +02:00
|
|
|
/* Nothing to do if we don't have any replication stats to be sent. */
|
|
|
|
if (rb->spillBytes <= 0 && rb->streamBytes <= 0 && rb->totalBytes <= 0)
|
2020-10-08 05:39:08 +02:00
|
|
|
return;
|
|
|
|
|
2021-04-16 04:04:43 +02:00
|
|
|
elog(DEBUG2, "UpdateDecodingStats: updating stats %p %lld %lld %lld %lld %lld %lld %lld %lld",
|
2020-10-08 05:39:08 +02:00
|
|
|
rb,
|
|
|
|
(long long) rb->spillTxns,
|
|
|
|
(long long) rb->spillCount,
|
2020-10-29 04:41:51 +01:00
|
|
|
(long long) rb->spillBytes,
|
|
|
|
(long long) rb->streamTxns,
|
|
|
|
(long long) rb->streamCount,
|
2021-04-16 04:04:43 +02:00
|
|
|
(long long) rb->streamBytes,
|
|
|
|
(long long) rb->totalTxns,
|
|
|
|
(long long) rb->totalBytes);
|
2020-10-08 05:39:08 +02:00
|
|
|
|
2021-04-14 05:25:03 +02:00
|
|
|
repSlotStat.spill_txns = rb->spillTxns;
|
|
|
|
repSlotStat.spill_count = rb->spillCount;
|
|
|
|
repSlotStat.spill_bytes = rb->spillBytes;
|
|
|
|
repSlotStat.stream_txns = rb->streamTxns;
|
|
|
|
repSlotStat.stream_count = rb->streamCount;
|
|
|
|
repSlotStat.stream_bytes = rb->streamBytes;
|
2021-04-16 04:04:43 +02:00
|
|
|
repSlotStat.total_txns = rb->totalTxns;
|
|
|
|
repSlotStat.total_bytes = rb->totalBytes;
|
2021-04-14 05:25:03 +02:00
|
|
|
|
2022-04-07 03:26:17 +02:00
|
|
|
pgstat_report_replslot(ctx->slot, &repSlotStat);
|
2021-04-16 04:04:43 +02:00
|
|
|
|
2020-10-08 05:39:08 +02:00
|
|
|
rb->spillTxns = 0;
|
|
|
|
rb->spillCount = 0;
|
|
|
|
rb->spillBytes = 0;
|
2020-10-29 04:41:51 +01:00
|
|
|
rb->streamTxns = 0;
|
|
|
|
rb->streamCount = 0;
|
|
|
|
rb->streamBytes = 0;
|
2021-04-16 04:04:43 +02:00
|
|
|
rb->totalTxns = 0;
|
|
|
|
rb->totalBytes = 0;
|
2020-10-08 05:39:08 +02:00
|
|
|
}
|