2021-05-07 16:56:14 +02:00
|
|
|
|
|
|
|
# Copyright (c) 2021, PostgreSQL Global Development Group
|
|
|
|
|
2020-04-12 17:26:05 +02:00
|
|
|
# Test the behavior of pg_verifybackup when the backup manifest has
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
# problems.
|
|
|
|
|
|
|
|
use strict;
|
|
|
|
use warnings;
|
|
|
|
use Cwd;
|
|
|
|
use Config;
|
|
|
|
use PostgresNode;
|
|
|
|
use TestLib;
|
|
|
|
use Test::More tests => 58;
|
|
|
|
|
|
|
|
my $tempdir = TestLib::tempdir;
|
|
|
|
|
|
|
|
test_bad_manifest(
|
|
|
|
'input string ended unexpectedly',
|
|
|
|
qr/could not parse backup manifest: The input string ended unexpectedly/,
|
|
|
|
<<EOM);
|
|
|
|
{
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected object end', <<EOM);
|
|
|
|
{}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected array start', <<EOM);
|
|
|
|
[]
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('expected version indicator', <<EOM);
|
|
|
|
{"not-expected": 1}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected manifest version', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": "phooey"}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected scalar', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": true}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('unrecognized top-level field', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Oops": 1}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected object start', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": {}}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('missing path name', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [{}]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('both path name and encoded path name', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Encoded-Path": "1234"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected file field', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Oops": 1}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('missing size', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('file size is not an integer', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Size": "Oops"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('could not decode file name', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Encoded-Path": "123", "Size": 0}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_fatal_error('duplicate path name in backup manifest', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Size": 0},
|
|
|
|
{"Path": "x", "Size": 0}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('checksum without algorithm', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Size": 100, "Checksum": "Oops"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_fatal_error('unrecognized checksum algorithm', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Size": 100, "Checksum-Algorithm": "Oops", "Checksum": "00"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_fatal_error('invalid checksum for file', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
|
|
|
|
{"Path": "x", "Size": 100, "Checksum-Algorithm": "CRC32C", "Checksum": "0"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('missing start LSN', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": 1}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('missing end LSN', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": 1, "Start-LSN": "0/0"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('unexpected WAL range field', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Oops": 1}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('missing timeline', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('unexpected object end', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": 1, "Start-LSN": "0/0", "End-LSN": "0/0"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('timeline is not an integer', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": true, "Start-LSN": "0/0", "End-LSN": "0/0"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('could not parse start LSN', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": 1, "Start-LSN": "oops", "End-LSN": "0/0"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
2020-09-14 06:42:07 +02:00
|
|
|
test_parse_error('could not parse end LSN', <<EOM);
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
|
|
|
|
{"Timeline": 1, "Start-LSN": "0/0", "End-LSN": "oops"}
|
|
|
|
]}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
test_parse_error('expected at least 2 lines', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [], "Manifest-Checksum": null}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
my $manifest_without_newline = <<EOM;
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1,
|
|
|
|
"Files": [],
|
|
|
|
"Manifest-Checksum": null}
|
|
|
|
EOM
|
|
|
|
chomp($manifest_without_newline);
|
|
|
|
test_parse_error('last line not newline-terminated',
|
|
|
|
$manifest_without_newline);
|
|
|
|
|
|
|
|
test_fatal_error('invalid manifest checksum', <<EOM);
|
|
|
|
{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [],
|
|
|
|
"Manifest-Checksum": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234567890-"}
|
|
|
|
EOM
|
|
|
|
|
|
|
|
sub test_parse_error
|
|
|
|
{
|
|
|
|
my ($test_name, $manifest_contents) = @_;
|
|
|
|
|
|
|
|
test_bad_manifest($test_name,
|
|
|
|
qr/could not parse backup manifest: $test_name/,
|
|
|
|
$manifest_contents);
|
2020-04-03 21:28:59 +02:00
|
|
|
return;
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
sub test_fatal_error
|
|
|
|
{
|
|
|
|
my ($test_name, $manifest_contents) = @_;
|
|
|
|
|
|
|
|
test_bad_manifest($test_name, qr/fatal: $test_name/, $manifest_contents);
|
2020-04-03 21:28:59 +02:00
|
|
|
return;
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
sub test_bad_manifest
|
|
|
|
{
|
|
|
|
my ($test_name, $regexp, $manifest_contents) = @_;
|
|
|
|
|
|
|
|
open(my $fh, '>', "$tempdir/backup_manifest") || die "open: $!";
|
|
|
|
print $fh $manifest_contents;
|
|
|
|
close($fh);
|
|
|
|
|
2020-04-12 17:26:05 +02:00
|
|
|
command_fails_like([ 'pg_verifybackup', $tempdir ], $regexp, $test_name);
|
2020-04-03 21:28:59 +02:00
|
|
|
return;
|
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.
A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.
Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.
Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
2020-04-03 20:59:47 +02:00
|
|
|
}
|