Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* basebackup_sink.c
|
|
|
|
* Default implementations for bbsink (basebackup sink) callbacks.
|
|
|
|
*
|
2024-01-04 02:49:05 +01:00
|
|
|
* Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group
|
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
*
|
2022-08-10 20:03:23 +02:00
|
|
|
* src/backend/backup/basebackup_sink.c
|
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "postgres.h"
|
|
|
|
|
2022-08-10 20:03:23 +02:00
|
|
|
#include "backup/basebackup_sink.h"
|
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward begin_backup callback.
|
|
|
|
*
|
|
|
|
* Only use this implementation if you want the bbsink you're implementing to
|
2022-03-15 03:29:23 +01:00
|
|
|
* share a buffer with the successor bbsink.
|
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_begin_backup(bbsink *sink)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
Assert(sink->bbs_state != NULL);
|
|
|
|
bbsink_begin_backup(sink->bbs_next, sink->bbs_state,
|
|
|
|
sink->bbs_buffer_length);
|
|
|
|
sink->bbs_buffer = sink->bbs_next->bbs_buffer;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward begin_archive callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_begin_archive(bbsink *sink, const char *archive_name)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_begin_archive(sink->bbs_next, archive_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward archive_contents callback.
|
|
|
|
*
|
2022-03-15 03:29:23 +01:00
|
|
|
* Code that wants to use this should initialize its own bbs_buffer and
|
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
2021-11-05 15:08:30 +01:00
|
|
|
* bbs_buffer_length fields to the values from the successor sink. In cases
|
|
|
|
* where the buffer isn't shared, the data needs to be copied before forwarding
|
|
|
|
* the callback. We don't do try to do that here, because there's really no
|
|
|
|
* reason to have separately allocated buffers containing the same identical
|
|
|
|
* data.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_archive_contents(bbsink *sink, size_t len)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
Assert(sink->bbs_buffer == sink->bbs_next->bbs_buffer);
|
|
|
|
Assert(sink->bbs_buffer_length == sink->bbs_next->bbs_buffer_length);
|
|
|
|
bbsink_archive_contents(sink->bbs_next, len);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward end_archive callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_end_archive(bbsink *sink)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_end_archive(sink->bbs_next);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward begin_manifest callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_begin_manifest(bbsink *sink)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_begin_manifest(sink->bbs_next);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward manifest_contents callback.
|
|
|
|
*
|
|
|
|
* As with the archive_contents callback, it's expected that the buffer is
|
|
|
|
* shared.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_manifest_contents(bbsink *sink, size_t len)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
Assert(sink->bbs_buffer == sink->bbs_next->bbs_buffer);
|
|
|
|
Assert(sink->bbs_buffer_length == sink->bbs_next->bbs_buffer_length);
|
|
|
|
bbsink_manifest_contents(sink->bbs_next, len);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward end_manifest callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_end_manifest(bbsink *sink)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_end_manifest(sink->bbs_next);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward end_backup callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_end_backup(bbsink *sink, XLogRecPtr endptr, TimeLineID endtli)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_end_backup(sink->bbs_next, endptr, endtli);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forward cleanup callback.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
bbsink_forward_cleanup(bbsink *sink)
|
|
|
|
{
|
|
|
|
Assert(sink->bbs_next != NULL);
|
|
|
|
bbsink_cleanup(sink->bbs_next);
|
|
|
|
}
|