postgresql/src/backend/replication
Peter Eisentraut 721856ff24 Remove distprep
A PostgreSQL release tarball contains a number of prebuilt files, in
particular files produced by bison, flex, perl, and well as html and
man documentation.  We have done this consistent with established
practice at the time to not require these tools for building from a
tarball.  Some of these tools were hard to get, or get the right
version of, from time to time, and shipping the prebuilt output was a
convenience to users.

Now this has at least two problems:

One, we have to make the build system(s) work in two modes: Building
from a git checkout and building from a tarball.  This is pretty
complicated, but it works so far for autoconf/make.  It does not
currently work for meson; you can currently only build with meson from
a git checkout.  Making meson builds work from a tarball seems very
difficult or impossible.  One particular problem is that since meson
requires a separate build directory, we cannot make the build update
files like gram.h in the source tree.  So if you were to build from a
tarball and update gram.y, you will have a gram.h in the source tree
and one in the build tree, but the way things work is that the
compiler will always use the one in the source tree.  So you cannot,
for example, make any gram.y changes when building from a tarball.
This seems impossible to fix in a non-horrible way.

Second, there is increased interest nowadays in precisely tracking the
origin of software.  We can reasonably track contributions into the
git tree, and users can reasonably track the path from a tarball to
packages and downloads and installs.  But what happens between the git
tree and the tarball is obscure and in some cases non-reproducible.

The solution for both of these issues is to get rid of the step that
adds prebuilt files to the tarball.  The tarball now only contains
what is in the git tree (*).  Getting the additional build
dependencies is no longer a problem nowadays, and the complications to
keep these dual build modes working are significant.  And of course we
want to get the meson build system working universally.

This commit removes the make distprep target altogether.  The make
dist target continues to do its job, it just doesn't call distprep
anymore.

(*) - The tarball also contains the INSTALL file that is built at make
dist time, but not by distprep.  This is unchanged for now.

The make maintainer-clean target, whose job it is to remove the
prebuilt files in addition to what make distclean does, is now just an
alias to make distprep.  (In practice, it is probably obsolete given
that git clean is available.)

The following programs are now hard build requirements in configure
(they were already required by meson.build):

- bison
- flex
- perl

Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://www.postgresql.org/message-id/flat/e07408d9-e5f2-d9fd-5672-f53354e9305e@eisentraut.org
2023-11-06 15:18:04 +01:00
..
libpqwalreceiver Remove distprep 2023-11-06 15:18:04 +01:00
logical Add trailing commas to enum definitions 2023-10-26 09:20:54 +02:00
pgoutput Remove distprep 2023-11-06 15:18:04 +01:00
.gitignore Build all Flex files standalone 2022-09-04 12:09:01 +07:00
Makefile Remove distprep 2023-11-06 15:18:04 +01:00
README code: replace 'master' with 'primary' where appropriate. 2020-07-08 12:57:23 -07:00
meson.build Update copyright for 2023 2023-01-02 15:00:37 -05:00
repl_gram.y Update copyright for 2023 2023-01-02 15:00:37 -05:00
repl_scanner.l Handle \v as a whitespace character in parsers 2023-07-06 08:16:24 +09:00
slot.c Migrate logical slots to the new node during an upgrade. 2023-10-26 07:06:55 +05:30
slotfuncs.c Remove unnecessary break in pg_logical_replication_slot_advance() 2023-10-23 10:20:30 +09:00
syncrep.c Pre-beta mechanical code beautification. 2023-05-19 17:24:48 -04:00
syncrep_gram.y Update copyright for 2023 2023-01-02 15:00:37 -05:00
syncrep_scanner.l Update copyright for 2023 2023-01-02 15:00:37 -05:00
walreceiver.c Add trailing commas to enum definitions 2023-10-26 09:20:54 +02:00
walreceiverfuncs.c Update copyright for 2023 2023-01-02 15:00:37 -05:00
walsender.c Use more consistent names for wait event objects and types 2023-09-06 10:04:43 +09:00

README

src/backend/replication/README

Walreceiver - libpqwalreceiver API
----------------------------------

The transport-specific part of walreceiver, responsible for connecting to
the primary server, receiving WAL files and sending messages, is loaded
dynamically to avoid having to link the main server binary with libpq.
The dynamically loaded module is in libpqwalreceiver subdirectory.

The dynamically loaded module implements a set of functions with details
about each one of them provided in src/include/replication/walreceiver.h.

This API should be considered internal at the moment, but we could open it
up for 3rd party replacements of libpqwalreceiver in the future, allowing
pluggable methods for receiving WAL.

Walreceiver IPC
---------------

When the WAL replay in startup process has reached the end of archived WAL,
restorable using restore_command, it starts up the walreceiver process
to fetch more WAL (if streaming replication is configured).

Walreceiver is a postmaster subprocess, so the startup process can't fork it
directly. Instead, it sends a signal to postmaster, asking postmaster to launch
it. Before that, however, startup process fills in WalRcvData->conninfo
and WalRcvData->slotname, and initializes the starting point in
WalRcvData->receiveStart.

As walreceiver receives WAL from the primary server, and writes and flushes
it to disk (in pg_wal), it updates WalRcvData->flushedUpto and signals
the startup process to know how far WAL replay can advance.

Walreceiver sends information about replication progress to the primary server
whenever it either writes or flushes new WAL, or the specified interval elapses.
This is used for reporting purpose.

Walsender IPC
-------------

At shutdown, postmaster handles walsender processes differently from regular
backends. It waits for regular backends to die before writing the
shutdown checkpoint and terminating pgarch and other auxiliary processes, but
that's not desirable for walsenders, because we want the standby servers to
receive all the WAL, including the shutdown checkpoint, before the primary
is shut down. Therefore postmaster treats walsenders like the pgarch process,
and instructs them to terminate at PM_SHUTDOWN_2 phase, after all regular
backends have died and checkpointer has issued the shutdown checkpoint.

When postmaster accepts a connection, it immediately forks a new process
to handle the handshake and authentication, and the process initializes to
become a backend. Postmaster doesn't know if the process becomes a regular
backend or a walsender process at that time - that's indicated in the
connection handshake - so we need some extra signaling to let postmaster
identify walsender processes.

When walsender process starts up, it marks itself as a walsender process in
the PMSignal array. That way postmaster can tell it apart from regular
backends.

Note that no big harm is done if postmaster thinks that a walsender is a
regular backend; it will just terminate the walsender earlier in the shutdown
phase. A walsender will look like a regular backend until it's done with the
initialization and has marked itself in PMSignal array, and at process
termination, after unmarking the PMSignal slot.

Each walsender allocates an entry from the WalSndCtl array, and tracks
information about replication progress. User can monitor them via
statistics views.


Walsender - walreceiver protocol
--------------------------------

See manual.