2014-03-18 18:20:01 +01:00
|
|
|
<!-- doc/src/sgml/logicaldecoding.sgml -->
|
|
|
|
<chapter id="logicaldecoding">
|
|
|
|
<title>Logical Decoding</title>
|
|
|
|
<indexterm zone="logicaldecoding">
|
|
|
|
<primary>Logical Decoding</primary>
|
|
|
|
</indexterm>
|
|
|
|
<para>
|
|
|
|
PostgreSQL provides infrastructure to stream the modifications performed
|
2016-04-25 02:44:22 +02:00
|
|
|
via SQL to external consumers. This functionality can be used for a
|
2014-03-18 18:20:01 +01:00
|
|
|
variety of purposes, including replication solutions and auditing.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Changes are sent out in streams identified by logical replication slots.
|
|
|
|
Each stream outputs each change exactly once.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The format in which those changes are streamed is determined by the output
|
2014-10-31 03:52:21 +01:00
|
|
|
plugin used. An example plugin is provided in the PostgreSQL distribution.
|
|
|
|
Additional plugins can be
|
2014-03-18 18:20:01 +01:00
|
|
|
written to extend the choice of available formats without modifying any
|
|
|
|
core code.
|
|
|
|
Every output plugin has access to each individual new row produced
|
|
|
|
by <command>INSERT</command> and the new row version created
|
|
|
|
by <command>UPDATE</command>. Availability of old row versions for
|
2014-03-19 14:13:05 +01:00
|
|
|
<command>UPDATE</command> and <command>DELETE</command> depends on
|
2014-10-31 03:52:21 +01:00
|
|
|
the configured replica identity (see <xref linkend="SQL-CREATETABLE-REPLICA-IDENTITY">).
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Changes can be consumed either using the streaming replication protocol
|
|
|
|
(see <xref linkend="protocol-replication"> and
|
|
|
|
<xref linkend="logicaldecoding-walsender">), or by calling functions
|
|
|
|
via SQL (see <xref linkend="logicaldecoding-sql">). It is also possible
|
|
|
|
to write additional methods of consuming the output of a replication slot
|
|
|
|
without modifying core code
|
|
|
|
(see <xref linkend="logicaldecoding-writer">).
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<sect1 id="logicaldecoding-example">
|
2014-10-31 03:52:21 +01:00
|
|
|
<title>Logical Decoding Examples</title>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
The following example demonstrates controlling logical decoding using the
|
|
|
|
SQL interface.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
Before you can use logical decoding, you must set
|
2014-03-19 14:13:05 +01:00
|
|
|
<xref linkend="guc-wal-level"> to <literal>logical</literal> and
|
2014-10-31 03:52:21 +01:00
|
|
|
<xref linkend="guc-max-replication-slots"> to at least 1. Then, you
|
|
|
|
should connect to the target database (in the example
|
2014-03-18 18:20:01 +01:00
|
|
|
below, <literal>postgres</literal>) as a superuser.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
postgres=# -- Create a slot named 'regression_slot' using the output plugin 'test_decoding'
|
|
|
|
postgres=# SELECT * FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
|
2014-06-05 16:29:20 +02:00
|
|
|
slot_name | xlog_position
|
2014-03-18 18:20:01 +01:00
|
|
|
-----------------+---------------
|
|
|
|
regression_slot | 0/16B1970
|
|
|
|
(1 row)
|
|
|
|
|
2015-08-10 13:28:18 +02:00
|
|
|
postgres=# SELECT slot_name, plugin, slot_type, database, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;
|
|
|
|
slot_name | plugin | slot_type | database | active | restart_lsn | confirmed_flush_lsn
|
|
|
|
-----------------+---------------+-----------+----------+--------+-------------+-----------------
|
|
|
|
regression_slot | test_decoding | logical | postgres | f | 0/16A4408 | 0/16A4440
|
2014-03-18 18:20:01 +01:00
|
|
|
(1 row)
|
|
|
|
|
|
|
|
postgres=# -- There are no changes to see yet
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
----------+-----+------
|
|
|
|
(0 rows)
|
|
|
|
|
|
|
|
postgres=# CREATE TABLE data(id serial primary key, data text);
|
|
|
|
CREATE TABLE
|
|
|
|
|
|
|
|
postgres=# -- DDL isn't replicated, so all you'll see is the transaction
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
-----------+-----+------------
|
|
|
|
0/16D5D48 | 688 | BEGIN 688
|
|
|
|
0/16E0380 | 688 | COMMIT 688
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
postgres=# -- Once changes are read, they're consumed and not emitted
|
|
|
|
postgres=# -- in a subsequent call:
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
----------+-----+------
|
|
|
|
(0 rows)
|
|
|
|
|
|
|
|
postgres=# BEGIN;
|
|
|
|
postgres=# INSERT INTO data(data) VALUES('1');
|
|
|
|
postgres=# INSERT INTO data(data) VALUES('2');
|
|
|
|
postgres=# COMMIT;
|
|
|
|
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
-----------+-----+-----------------------------------------------
|
|
|
|
0/16E0478 | 689 | BEGIN 689
|
2014-03-18 19:57:49 +01:00
|
|
|
0/16E0478 | 689 | table public.data: INSERT: id[integer]:1 data[text]:'1'
|
|
|
|
0/16E0580 | 689 | table public.data: INSERT: id[integer]:2 data[text]:'2'
|
2014-03-18 18:20:01 +01:00
|
|
|
0/16E0650 | 689 | COMMIT 689
|
|
|
|
(4 rows)
|
|
|
|
|
|
|
|
postgres=# INSERT INTO data(data) VALUES('3');
|
|
|
|
|
|
|
|
postgres=# -- You can also peek ahead in the change stream without consuming changes
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
-----------+-----+-----------------------------------------------
|
|
|
|
0/16E09C0 | 690 | BEGIN 690
|
2014-03-18 19:57:49 +01:00
|
|
|
0/16E09C0 | 690 | table public.data: INSERT: id[integer]:3 data[text]:'3'
|
2014-03-18 18:20:01 +01:00
|
|
|
0/16E0B90 | 690 | COMMIT 690
|
|
|
|
(3 rows)
|
|
|
|
|
2014-07-29 05:07:54 +02:00
|
|
|
postgres=# -- The next call to pg_logical_slot_peek_changes() returns the same changes again
|
2014-03-18 18:20:01 +01:00
|
|
|
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL);
|
|
|
|
location | xid | data
|
|
|
|
-----------+-----+-----------------------------------------------
|
|
|
|
0/16E09C0 | 690 | BEGIN 690
|
2014-03-18 19:57:49 +01:00
|
|
|
0/16E09C0 | 690 | table public.data: INSERT: id[integer]:3 data[text]:'3'
|
2014-03-18 18:20:01 +01:00
|
|
|
0/16E0B90 | 690 | COMMIT 690
|
|
|
|
(3 rows)
|
|
|
|
|
|
|
|
postgres=# -- options can be passed to output plugin, to influence the formatting
|
|
|
|
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-timestamp', 'on');
|
|
|
|
location | xid | data
|
|
|
|
-----------+-----+-----------------------------------------------
|
|
|
|
0/16E09C0 | 690 | BEGIN 690
|
2014-03-18 19:57:49 +01:00
|
|
|
0/16E09C0 | 690 | table public.data: INSERT: id[integer]:3 data[text]:'3'
|
2014-03-18 18:20:01 +01:00
|
|
|
0/16E0B90 | 690 | COMMIT 690 (at 2014-02-27 16:41:51.863092+01)
|
|
|
|
(3 rows)
|
|
|
|
|
|
|
|
postgres=# -- Remember to destroy a slot you no longer need to stop it consuming
|
|
|
|
postgres=# -- server resources:
|
|
|
|
postgres=# SELECT pg_drop_replication_slot('regression_slot');
|
|
|
|
pg_drop_replication_slot
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
(1 row)
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
The following example shows how logical decoding is controlled over the
|
|
|
|
streaming replication protocol, using the
|
|
|
|
program <xref linkend="app-pgrecvlogical"> included in the PostgreSQL
|
|
|
|
distribution. This requires that client authentication is set up to allow
|
|
|
|
replication connections
|
|
|
|
(see <xref linkend="streaming-replication-authentication">) and
|
2016-03-08 03:52:30 +01:00
|
|
|
that <varname>max_wal_senders</varname> is set sufficiently high to allow
|
|
|
|
an additional connection.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-10-31 03:52:21 +01:00
|
|
|
$ pg_recvlogical -d postgres --slot test --create-slot
|
|
|
|
$ pg_recvlogical -d postgres --slot test --start -f -
|
2014-10-31 13:11:06 +01:00
|
|
|
<keycombo action="simul"><keycap>Control</><keycap>Z</></>
|
2014-10-31 03:52:21 +01:00
|
|
|
$ psql -d postgres -c "INSERT INTO data(data) VALUES('4');"
|
|
|
|
$ fg
|
2014-03-18 18:20:01 +01:00
|
|
|
BEGIN 693
|
2014-03-18 19:57:49 +01:00
|
|
|
table public.data: INSERT: id[integer]:4 data[text]:'4'
|
2014-03-18 18:20:01 +01:00
|
|
|
COMMIT 693
|
2014-10-31 03:52:21 +01:00
|
|
|
<keycombo action="simul"><keycap>Control</><keycap>C</></>
|
|
|
|
$ pg_recvlogical -d postgres --slot test --drop-slot
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-explanation">
|
|
|
|
<title>Logical Decoding Concepts</title>
|
|
|
|
<sect2>
|
2014-05-07 03:28:58 +02:00
|
|
|
<title>Logical Decoding</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<indexterm>
|
|
|
|
<primary>Logical Decoding</primary>
|
|
|
|
</indexterm>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-03-24 07:42:13 +01:00
|
|
|
Logical decoding is the process of extracting all persistent changes
|
2014-03-18 18:20:01 +01:00
|
|
|
to a database's tables into a coherent, easy to understand format which
|
|
|
|
can be interpreted without detailed knowledge of the database's internal
|
|
|
|
state.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
In <productname>PostgreSQL</productname>, logical decoding is implemented
|
|
|
|
by decoding the contents of the <link linkend="wal">write-ahead
|
|
|
|
log</link>, which describe changes on a storage level, into an
|
|
|
|
application-specific form such as a stream of tuples or SQL statements.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2016-08-11 21:09:24 +02:00
|
|
|
<sect2 id="logicaldecoding-replication-slots">
|
2014-04-22 21:22:10 +02:00
|
|
|
<title>Replication Slots</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<indexterm>
|
2014-04-22 21:22:10 +02:00
|
|
|
<primary>replication slot</primary>
|
|
|
|
<secondary>logical replication</secondary>
|
2014-03-18 18:20:01 +01:00
|
|
|
</indexterm>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
In the context of logical replication, a slot represents a stream of
|
2014-10-31 03:52:21 +01:00
|
|
|
changes that can be replayed to a client in the order they were made on
|
2014-03-18 18:20:01 +01:00
|
|
|
the origin server. Each slot streams a sequence of changes from a single
|
|
|
|
database, sending each change exactly once (except when peeking forward
|
|
|
|
in the stream).
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<note>
|
2014-03-19 14:13:05 +01:00
|
|
|
<para><productname>PostgreSQL</productname> also has streaming replication slots
|
2014-03-18 18:20:01 +01:00
|
|
|
(see <xref linkend="streaming-replication">), but they are used somewhat
|
|
|
|
differently there.
|
|
|
|
</para>
|
|
|
|
</note>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
A replication slot has an identifier that is unique across all databases
|
2014-03-18 18:20:01 +01:00
|
|
|
in a <productname>PostgreSQL</productname> cluster. Slots persist
|
|
|
|
independently of the connection using them and are crash-safe.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
Multiple independent slots may exist for a single database. Each slot has
|
|
|
|
its own state, allowing different consumers to receive changes from
|
|
|
|
different points in the database change stream. For most applications, a
|
|
|
|
separate slot will be required for each consumer.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
A logical replication slot knows nothing about the state of the
|
|
|
|
receiver(s). It's even possible to have multiple different receivers using
|
|
|
|
the same slot at different times; they'll just get the changes following
|
|
|
|
on from when the last receiver stopped consuming them. Only one receiver
|
|
|
|
may consume changes from a slot at any given time.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
Replication slots persist across crashes and know nothing about the state
|
|
|
|
of their consumer(s). They will prevent removal of required resources
|
|
|
|
even when there is no connection using them. This consumes storage
|
|
|
|
because neither required WAL nor required rows from the system catalogs
|
2014-10-31 03:52:21 +01:00
|
|
|
can be removed by <command>VACUUM</command> as long as they are required by a replication
|
|
|
|
slot. So if a slot is no longer required it should be dropped.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
</sect2>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect2>
|
|
|
|
<title>Output Plugins</title>
|
|
|
|
<para>
|
|
|
|
Output plugins transform the data from the write-ahead log's internal
|
|
|
|
representation into the format the consumer of a replication slot desires.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect2>
|
|
|
|
<title>Exported Snapshots</title>
|
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
When a new replication slot is created using the streaming replication interface,
|
|
|
|
a snapshot is exported
|
|
|
|
(see <xref linkend="functions-snapshot-synchronization">), which will show
|
2014-03-18 18:20:01 +01:00
|
|
|
exactly the state of the database after which all changes will be
|
|
|
|
included in the change stream. This can be used to create a new replica by
|
|
|
|
using <link linkend="sql-set-transaction"><literal>SET TRANSACTION
|
|
|
|
SNAPSHOT</literal></link> to read the state of the database at the moment
|
|
|
|
the slot was created. This transaction can then be used to dump the
|
2014-10-31 03:52:21 +01:00
|
|
|
database's state at that point in time, which afterwards can be updated
|
2014-03-18 19:57:49 +01:00
|
|
|
using the slot's contents without losing any changes.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-walsender">
|
|
|
|
<title>Streaming Replication Protocol Interface</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
The commands
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem>
|
2016-01-29 04:14:56 +01:00
|
|
|
<para><literal>CREATE_REPLICATION_SLOT <replaceable>slot_name</replaceable> LOGICAL <replaceable>output_plugin</replaceable></literal></para>
|
2014-10-31 03:52:21 +01:00
|
|
|
</listitem>
|
|
|
|
|
|
|
|
<listitem>
|
|
|
|
<para><literal>DROP_REPLICATION_SLOT <replaceable>slot_name</replaceable></literal></para>
|
|
|
|
</listitem>
|
|
|
|
|
|
|
|
<listitem>
|
2016-01-29 04:14:56 +01:00
|
|
|
<para><literal>START_REPLICATION SLOT <replaceable>slot_name</replaceable> LOGICAL ...</literal></para>
|
2014-10-31 03:52:21 +01:00
|
|
|
</listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
are used to create, drop, and stream changes from a replication
|
|
|
|
slot, respectively. These commands are only available over a replication
|
2014-03-18 18:20:01 +01:00
|
|
|
connection; they cannot be used via SQL.
|
2014-10-31 03:52:21 +01:00
|
|
|
See <xref linkend="protocol-replication"> for details on these commands.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-10-31 03:52:21 +01:00
|
|
|
The command <xref linkend="app-pgrecvlogical"> can be used to control
|
2015-02-12 01:16:32 +01:00
|
|
|
logical decoding over a streaming replication connection. (It uses
|
2014-10-31 03:52:21 +01:00
|
|
|
these commands internally.)
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-sql">
|
|
|
|
<title>Logical Decoding <acronym>SQL</acronym> Interface</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
See <xref linkend="functions-replication"> for detailed documentation on
|
|
|
|
the SQL-level API for interacting with logical decoding.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
Synchronous replication (see <xref linkend="synchronous-replication">) is
|
2014-10-31 13:11:06 +01:00
|
|
|
only supported on replication slots used over the streaming replication interface. The
|
2014-03-18 18:20:01 +01:00
|
|
|
function interface and additional, non-core interfaces do not support
|
|
|
|
synchronous replication.
|
|
|
|
</para>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-catalogs">
|
2014-10-31 03:52:21 +01:00
|
|
|
<title>System Catalogs Related to Logical Decoding</title>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2016-05-05 18:33:12 +02:00
|
|
|
The <link linkend="view-pg-replication-slots"><structname>pg_replication_slots</structname></link>
|
2014-03-18 18:20:01 +01:00
|
|
|
view and the
|
|
|
|
<link linkend="monitoring-stats-views-table"><structname>pg_stat_replication</structname></link>
|
|
|
|
view provide information about the current state of replication slots and
|
2014-10-31 03:52:21 +01:00
|
|
|
streaming replication connections respectively. These views apply to both physical and
|
2014-03-18 18:20:01 +01:00
|
|
|
logical replication.
|
|
|
|
</para>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-output-plugin">
|
|
|
|
<title>Logical Decoding Output Plugins</title>
|
|
|
|
<para>
|
|
|
|
An example output plugin can be found in the
|
|
|
|
<link linkend="test-decoding">
|
|
|
|
<filename>contrib/test_decoding</filename>
|
|
|
|
</link>
|
|
|
|
subdirectory of the PostgreSQL source tree.
|
|
|
|
</para>
|
|
|
|
<sect2 id="logicaldecoding-output-init">
|
|
|
|
<title>Initialization Function</title>
|
|
|
|
<indexterm zone="logicaldecoding">
|
|
|
|
<primary>_PG_output_plugin_init</primary>
|
|
|
|
</indexterm>
|
|
|
|
<para>
|
|
|
|
An output plugin is loaded by dynamically loading a shared library with
|
2014-07-17 04:20:15 +02:00
|
|
|
the output plugin's name as the library base name. The normal library
|
2014-03-18 18:20:01 +01:00
|
|
|
search path is used to locate the library. To provide the required output
|
|
|
|
plugin callbacks and to indicate that the library is actually an output
|
|
|
|
plugin it needs to provide a function named
|
|
|
|
<function>_PG_output_plugin_init</function>. This function is passed a
|
|
|
|
struct that needs to be filled with the callback function pointers for
|
|
|
|
individual actions.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef struct OutputPluginCallbacks
|
|
|
|
{
|
|
|
|
LogicalDecodeStartupCB startup_cb;
|
|
|
|
LogicalDecodeBeginCB begin_cb;
|
|
|
|
LogicalDecodeChangeCB change_cb;
|
|
|
|
LogicalDecodeCommitCB commit_cb;
|
2016-04-06 11:05:41 +02:00
|
|
|
LogicalDecodeMessageCB message_cb;
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
|
2014-03-18 18:20:01 +01:00
|
|
|
LogicalDecodeShutdownCB shutdown_cb;
|
|
|
|
} OutputPluginCallbacks;
|
2014-07-08 17:39:07 +02:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalOutputPluginInit)(struct OutputPluginCallbacks *cb);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
The <function>begin_cb</function>, <function>change_cb</function>
|
|
|
|
and <function>commit_cb</function> callbacks are required,
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
while <function>startup_cb</function>,
|
|
|
|
<function>filter_by_origin_cb</function>
|
2014-03-18 18:20:01 +01:00
|
|
|
and <function>shutdown_cb</function> are optional.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
|
|
|
<sect2 id="logicaldecoding-capabilities">
|
|
|
|
<title>Capabilities</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
To decode, format and output changes, output plugins can use most of the
|
|
|
|
backend's normal infrastructure, including calling output functions. Read
|
|
|
|
only access to relations is permitted as long as only relations are
|
|
|
|
accessed that either have been created by <command>initdb</command> in
|
2014-03-18 19:57:49 +01:00
|
|
|
the <literal>pg_catalog</literal> schema, or have been marked as user
|
2014-03-18 18:20:01 +01:00
|
|
|
provided catalog tables using
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
|
|
|
|
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-10-31 03:52:21 +01:00
|
|
|
Any actions leading to transaction ID assignment are prohibited. That, among others,
|
|
|
|
includes writing to tables, performing DDL changes, and
|
2014-03-18 18:20:01 +01:00
|
|
|
calling <literal>txid_current()</literal>.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2014-10-01 13:13:59 +02:00
|
|
|
<sect2 id="logicaldecoding-output-mode">
|
|
|
|
<title>Output Modes</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-10-01 13:13:59 +02:00
|
|
|
<para>
|
|
|
|
Output plugin callbacks can pass data to the consumer in nearly arbitrary
|
|
|
|
formats. For some use cases, like viewing the changes via SQL, returning
|
2014-10-31 03:52:21 +01:00
|
|
|
data in a data type that can contain arbitrary data (e.g., <type>bytea</type>) is
|
2014-10-01 13:13:59 +02:00
|
|
|
cumbersome. If the output plugin only outputs textual data in the
|
2014-10-31 03:52:21 +01:00
|
|
|
server's encoding, it can declare that by
|
2014-10-01 13:13:59 +02:00
|
|
|
setting <literal>OutputPluginOptions.output_mode</>
|
|
|
|
to <literal>OUTPUT_PLUGIN_TEXTUAL_OUTPUT</> instead
|
|
|
|
of <literal>OUTPUT_PLUGIN_BINARY_OUTPUT</> in
|
|
|
|
the <link linkend="logicaldecoding-output-plugin-startup">startup
|
2014-10-31 03:52:21 +01:00
|
|
|
callback</>. In that case, all the data has to be in the server's encoding
|
|
|
|
so that a <type>text</> datum can contain it. This is checked in assertion-enabled
|
2014-10-01 13:13:59 +02:00
|
|
|
builds.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect2 id="logicaldecoding-output-plugin-callbacks">
|
|
|
|
<title>Output Plugin Callbacks</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
An output plugin gets notified about changes that are happening via
|
|
|
|
various callbacks it needs to provide.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2014-07-17 04:20:15 +02:00
|
|
|
Concurrent transactions are decoded in commit order, and only changes
|
|
|
|
belonging to a specific transaction are decoded between
|
2014-03-18 18:20:01 +01:00
|
|
|
the <literal>begin</literal> and <literal>commit</literal>
|
|
|
|
callbacks. Transactions that were rolled back explicitly or implicitly
|
|
|
|
never get
|
2014-10-31 03:52:21 +01:00
|
|
|
decoded. Successful savepoints are
|
2014-03-18 18:20:01 +01:00
|
|
|
folded into the transaction containing them in the order they were
|
2014-03-18 19:57:49 +01:00
|
|
|
executed within that transaction.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
Only transactions that have already safely been flushed to disk will be
|
2014-10-31 03:52:21 +01:00
|
|
|
decoded. That can lead to a <command>COMMIT</command> not immediately being decoded in a
|
2014-03-18 18:20:01 +01:00
|
|
|
directly following <literal>pg_logical_slot_get_changes()</literal>
|
|
|
|
when <varname>synchronous_commit</varname> is set
|
|
|
|
to <literal>off</literal>.
|
|
|
|
</para>
|
|
|
|
</note>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-startup">
|
|
|
|
<title>Startup Callback</title>
|
|
|
|
<para>
|
|
|
|
The optional <function>startup_cb</function> callback is called whenever
|
2014-03-18 19:57:49 +01:00
|
|
|
a replication slot is created or asked to stream changes, independent
|
2014-03-18 18:20:01 +01:00
|
|
|
of the number of changes that are ready to be put out.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalDecodeStartupCB) (
|
|
|
|
struct LogicalDecodingContext *ctx,
|
|
|
|
OutputPluginOptions *options,
|
|
|
|
bool is_init
|
|
|
|
);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 19:57:49 +01:00
|
|
|
The <literal>is_init</literal> parameter will be true when the
|
2014-03-18 18:20:01 +01:00
|
|
|
replication slot is being created and false
|
|
|
|
otherwise. <parameter>options</parameter> points to a struct of options
|
|
|
|
that output plugins can set:
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef struct OutputPluginOptions
|
|
|
|
{
|
|
|
|
OutputPluginOutputType output_type;
|
|
|
|
} OutputPluginOptions;
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
<literal>output_type</literal> has to either be set to
|
|
|
|
<literal>OUTPUT_PLUGIN_TEXTUAL_OUTPUT</literal>
|
2014-10-01 13:13:59 +02:00
|
|
|
or <literal>OUTPUT_PLUGIN_BINARY_OUTPUT</literal>. See also
|
2015-02-04 11:00:09 +01:00
|
|
|
<xref linkend="logicaldecoding-output-mode">.
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The startup callback should validate the options present in
|
|
|
|
<literal>ctx->output_plugin_options</literal>. If the output plugin
|
|
|
|
needs to have a state, it can
|
|
|
|
use <literal>ctx->output_plugin_private</literal> to store it.
|
|
|
|
</para>
|
|
|
|
</sect3>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-shutdown">
|
|
|
|
<title>Shutdown Callback</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The optional <function>shutdown_cb</function> callback is called
|
|
|
|
whenever a formerly active replication slot is not used anymore and can
|
|
|
|
be used to deallocate resources private to the output plugin. The slot
|
|
|
|
isn't necessarily being dropped, streaming is just being stopped.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalDecodeShutdownCB) (
|
|
|
|
struct LogicalDecodingContext *ctx
|
|
|
|
);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
</sect3>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-begin">
|
|
|
|
<title>Transaction Begin Callback</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The required <function>begin_cb</function> callback is called whenever a
|
2014-07-17 04:20:15 +02:00
|
|
|
start of a committed transaction has been decoded. Aborted transactions
|
2014-03-18 18:20:01 +01:00
|
|
|
and their contents never get decoded.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalDecodeBeginCB) (
|
|
|
|
struct LogicalDecodingContext *,
|
|
|
|
ReorderBufferTXN *txn
|
|
|
|
);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
The <parameter>txn</parameter> parameter contains meta information about
|
2014-07-17 04:20:15 +02:00
|
|
|
the transaction, like the time stamp at which it has been committed and
|
2014-03-18 18:20:01 +01:00
|
|
|
its XID.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
</sect3>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-commit">
|
|
|
|
<title>Transaction End Callback</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The required <function>commit_cb</function> callback is called whenever
|
|
|
|
a transaction commit has been
|
|
|
|
decoded. The <function>change_cb</function> callbacks for all modified
|
|
|
|
rows will have been called before this, if there have been any modified
|
|
|
|
rows.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalDecodeCommitCB) (
|
|
|
|
struct LogicalDecodingContext *,
|
|
|
|
ReorderBufferTXN *txn
|
|
|
|
);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
</sect3>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-change">
|
2014-10-31 03:52:21 +01:00
|
|
|
<title>Change Callback</title>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The required <function>change_cb</function> callback is called for every
|
|
|
|
individual row modification inside a transaction, may it be
|
2014-10-31 03:52:21 +01:00
|
|
|
an <command>INSERT</command>, <command>UPDATE</command>,
|
2014-03-18 18:20:01 +01:00
|
|
|
or <command>DELETE</command>. Even if the original command modified
|
2014-07-17 04:20:15 +02:00
|
|
|
several rows at once the callback will be called individually for each
|
2014-03-18 18:20:01 +01:00
|
|
|
row.
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
typedef void (*LogicalDecodeChangeCB) (
|
|
|
|
struct LogicalDecodingContext *ctx,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
Relation relation,
|
|
|
|
ReorderBufferChange *change
|
|
|
|
);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
The <parameter>ctx</parameter> and <parameter>txn</parameter> parameters
|
|
|
|
have the same contents as for the <function>begin_cb</function>
|
|
|
|
and <function>commit_cb</function> callbacks, but additionally the
|
|
|
|
relation descriptor <parameter>relation</parameter> points to the
|
|
|
|
relation the row belongs to and a struct
|
|
|
|
<parameter>change</parameter> describing the row modification are passed
|
|
|
|
in.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
Only changes in user defined tables that are not unlogged
|
|
|
|
(see <xref linkend="SQL-CREATETABLE-UNLOGGED">) and not temporary
|
|
|
|
(see <xref linkend="SQL-CREATETABLE-TEMPORARY">) can be extracted using
|
|
|
|
logical decoding.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
</sect3>
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
|
2015-05-15 04:34:28 +02:00
|
|
|
<sect3 id="logicaldecoding-output-plugin-filter-origin">
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
<title>Origin Filter Callback</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The optional <function>filter_by_origin_cb</function> callback
|
2015-09-11 03:22:21 +02:00
|
|
|
is called to determine whether data that has been replayed
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
from <parameter>origin_id</parameter> is of interest to the
|
|
|
|
output plugin.
|
|
|
|
<programlisting>
|
2016-01-11 02:12:27 +01:00
|
|
|
typedef bool (*LogicalDecodeFilterByOriginCB) (
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
struct LogicalDecodingContext *ctx,
|
|
|
|
RepNodeId origin_id
|
|
|
|
);
|
|
|
|
</programlisting>
|
|
|
|
The <parameter>ctx</parameter> parameter has the same contents
|
|
|
|
as for the other callbacks. No information but the origin is
|
|
|
|
available. To signal that changes originating on the passed in
|
|
|
|
node are irrelevant, return true, causing them to be filtered
|
|
|
|
away; false otherwise. The other callbacks will not be called
|
|
|
|
for transactions and changes that have been filtered away.
|
|
|
|
</para>
|
|
|
|
<para>
|
2015-09-11 03:22:21 +02:00
|
|
|
This is useful when implementing cascading or multidirectional
|
Introduce replication progress tracking infrastructure.
When implementing a replication solution ontop of logical decoding, two
related problems exist:
* How to safely keep track of replication progress
* How to change replication behavior, based on the origin of a row;
e.g. to avoid loops in bi-directional replication setups
The solution to these problems, as implemented here, consist out of
three parts:
1) 'replication origins', which identify nodes in a replication setup.
2) 'replication progress tracking', which remembers, for each
replication origin, how far replay has progressed in a efficient and
crash safe manner.
3) The ability to filter out changes performed on the behest of a
replication origin during logical decoding; this allows complex
replication topologies. E.g. by filtering all replayed changes out.
Most of this could also be implemented in "userspace", e.g. by inserting
additional rows contain origin information, but that ends up being much
less efficient and more complicated. We don't want to require various
replication solutions to reimplement logic for this independently. The
infrastructure is intended to be generic enough to be reusable.
This infrastructure also replaces the 'nodeid' infrastructure of commit
timestamps. It is intended to provide all the former capabilities,
except that there's only 2^16 different origins; but now they integrate
with logical decoding. Additionally more functionality is accessible via
SQL. Since the commit timestamp infrastructure has also been introduced
in 9.5 (commit 73c986add) changing the API is not a problem.
For now the number of origins for which the replication progress can be
tracked simultaneously is determined by the max_replication_slots
GUC. That GUC is not a perfect match to configure this, but there
doesn't seem to be sufficient reason to introduce a separate new one.
Bumps both catversion and wal page magic.
Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer
Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer
Discussion: 20150216002155.GI15326@awork2.anarazel.de,
20140923182422.GA15776@alap3.anarazel.de,
20131114172632.GE7522@alap2.anarazel.de
2015-04-29 19:30:53 +02:00
|
|
|
replication solutions. Filtering by the origin allows to
|
|
|
|
prevent replicating the same changes back and forth in such
|
|
|
|
setups. While transactions and changes also carry information
|
|
|
|
about the origin, filtering via this callback is noticeably
|
|
|
|
more efficient.
|
|
|
|
</para>
|
|
|
|
</sect3>
|
2016-04-06 11:05:41 +02:00
|
|
|
|
|
|
|
<sect3 id="logicaldecoding-output-plugin-message">
|
|
|
|
<title>Generic Message Callback</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The optional <function>message_cb</function> callback is called whenever
|
|
|
|
a logical decoding message has been decoded.
|
|
|
|
<programlisting>
|
|
|
|
typedef void (*LogicalDecodeMessageCB) (
|
|
|
|
struct LogicalDecodingContext *,
|
|
|
|
ReorderBufferTXN *txn,
|
|
|
|
XLogRecPtr message_lsn,
|
|
|
|
bool transactional,
|
|
|
|
const char *prefix,
|
|
|
|
Size message_size,
|
|
|
|
const char *message
|
|
|
|
);
|
|
|
|
</programlisting>
|
|
|
|
The <parameter>txn</parameter> parameter contains meta information about
|
|
|
|
the transaction, like the time stamp at which it has been committed and
|
|
|
|
its XID. Note however that it can be NULL when the message is
|
|
|
|
non-transactional and the XID was not assigned yet in the transaction
|
|
|
|
which logged the message. The <parameter>lsn</parameter> has WAL
|
|
|
|
position of the message. The <parameter>transactional</parameter> says
|
|
|
|
if the message was sent as transactional or not.
|
|
|
|
The <parameter>prefix</parameter> is arbitrary null-terminated prefix
|
|
|
|
which can be used for identifying interesting messages for the current
|
|
|
|
plugin. And finally the <parameter>message</parameter> parameter holds
|
|
|
|
the actual message of <parameter>message_size</parameter> size.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Extra care should be taken to ensure that the prefix the output plugin
|
|
|
|
considers interesting is unique. Using name of the extension or the
|
|
|
|
output plugin itself is often a good choice.
|
|
|
|
</para>
|
|
|
|
</sect3>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
</sect2>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect2 id="logicaldecoding-output-plugin-output">
|
2014-10-31 03:52:21 +01:00
|
|
|
<title>Functions for Producing Output</title>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
To actually produce output, output plugins can write data to
|
|
|
|
the <literal>StringInfo</literal> output buffer
|
|
|
|
in <literal>ctx->out</literal> when inside
|
2014-10-31 03:52:21 +01:00
|
|
|
the <function>begin_cb</function>, <function>commit_cb</function>,
|
2014-03-18 18:20:01 +01:00
|
|
|
or <function>change_cb</function> callbacks. Before writing to the output
|
2014-10-31 03:52:21 +01:00
|
|
|
buffer, <function>OutputPluginPrepareWrite(ctx, last_write)</function> has
|
2014-03-18 18:20:01 +01:00
|
|
|
to be called, and after finishing writing to the
|
2014-10-31 03:52:21 +01:00
|
|
|
buffer, <function>OutputPluginWrite(ctx, last_write)</function> has to be
|
2014-03-18 18:20:01 +01:00
|
|
|
called to perform the write. The <parameter>last_write</parameter>
|
|
|
|
indicates whether a particular write was the callback's last write.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
The following example shows how to output data to the consumer of an
|
|
|
|
output plugin:
|
2014-07-08 17:39:07 +02:00
|
|
|
<programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
OutputPluginPrepareWrite(ctx, true);
|
|
|
|
appendStringInfo(ctx->out, "BEGIN %u", txn->xid);
|
|
|
|
OutputPluginWrite(ctx, true);
|
2014-07-08 17:39:07 +02:00
|
|
|
</programlisting>
|
2014-03-18 18:20:01 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-writer">
|
|
|
|
<title>Logical Decoding Output Writers</title>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
|
|
|
It is possible to add more output methods for logical decoding.
|
|
|
|
For details, see
|
|
|
|
<filename>src/backend/replication/logical/logicalfuncs.c</filename>.
|
|
|
|
Essentially, three functions need to be provided: one to read WAL, one to
|
|
|
|
prepare writing output, and one to write the output
|
|
|
|
(see <xref linkend="logicaldecoding-output-plugin-output">).
|
|
|
|
</para>
|
|
|
|
</sect1>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<sect1 id="logicaldecoding-synchronous">
|
2014-10-31 03:52:21 +01:00
|
|
|
<title>Synchronous Replication Support for Logical Decoding</title>
|
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<para>
|
2015-05-17 14:57:30 +02:00
|
|
|
Logical decoding can be used to build
|
2014-03-18 18:20:01 +01:00
|
|
|
<link linkend="synchronous-replication">synchronous
|
|
|
|
replication</link> solutions with the same user interface as synchronous
|
|
|
|
replication for <link linkend="streaming-replication">streaming
|
2014-10-31 03:52:21 +01:00
|
|
|
replication</link>. To do this, the streaming replication interface
|
2014-03-18 18:20:01 +01:00
|
|
|
(see <xref linkend="logicaldecoding-walsender">) must be used to stream out
|
|
|
|
data. Clients have to send <literal>Standby status update (F)</literal>
|
|
|
|
(see <xref linkend="protocol-replication">) messages, just like streaming
|
|
|
|
replication clients do.
|
|
|
|
</para>
|
2014-10-31 03:52:21 +01:00
|
|
|
|
2014-03-18 18:20:01 +01:00
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
A synchronous replica receiving changes via logical decoding will work in
|
|
|
|
the scope of a single database. Since, in contrast to
|
|
|
|
that, <parameter>synchronous_standby_names</parameter> currently is
|
|
|
|
server wide, this means this technique will not work properly if more
|
|
|
|
than one database is actively used.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
</sect1>
|
|
|
|
</chapter>
|