2022-03-23 14:19:14 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
* compression.c
|
2022-03-23 14:19:14 +01:00
|
|
|
*
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
* Shared code for compression methods and specifications.
|
2022-03-23 14:19:14 +01:00
|
|
|
*
|
|
|
|
* A compression specification specifies the parameters that should be used
|
|
|
|
* when performing compression with a specific algorithm. The simplest
|
|
|
|
* possible compression specification is an integer, which sets the
|
|
|
|
* compression level.
|
|
|
|
*
|
|
|
|
* Otherwise, a compression specification is a comma-separated list of items,
|
|
|
|
* each having the form keyword or keyword=value.
|
|
|
|
*
|
2023-04-06 17:18:38 +02:00
|
|
|
* Currently, the supported keywords are "level", "long", and "workers".
|
2022-03-23 14:19:14 +01:00
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
|
2022-03-23 14:19:14 +01:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
* src/common/compression.c
|
2022-03-23 14:19:14 +01:00
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef FRONTEND
|
|
|
|
#include "postgres.h"
|
|
|
|
#else
|
|
|
|
#include "postgres_fe.h"
|
|
|
|
#endif
|
|
|
|
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
#ifdef USE_ZSTD
|
|
|
|
#include <zstd.h>
|
|
|
|
#endif
|
|
|
|
#ifdef HAVE_LIBZ
|
|
|
|
#include <zlib.h>
|
|
|
|
#endif
|
|
|
|
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
#include "common/compression.h"
|
2022-03-23 14:19:14 +01:00
|
|
|
|
|
|
|
static int expect_integer_value(char *keyword, char *value,
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
pg_compress_specification *result);
|
2023-04-06 17:18:38 +02:00
|
|
|
static bool expect_boolean_value(char *keyword, char *value,
|
|
|
|
pg_compress_specification *result);
|
2022-03-23 14:19:14 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Look up a compression algorithm by name. Returns true and sets *algorithm
|
|
|
|
* if the name is recognized. Otherwise returns false.
|
|
|
|
*/
|
|
|
|
bool
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
parse_compress_algorithm(char *name, pg_compress_algorithm *algorithm)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
|
|
|
if (strcmp(name, "none") == 0)
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
*algorithm = PG_COMPRESSION_NONE;
|
2022-03-23 14:19:14 +01:00
|
|
|
else if (strcmp(name, "gzip") == 0)
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
*algorithm = PG_COMPRESSION_GZIP;
|
2022-03-23 14:19:14 +01:00
|
|
|
else if (strcmp(name, "lz4") == 0)
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
*algorithm = PG_COMPRESSION_LZ4;
|
2022-03-23 14:19:14 +01:00
|
|
|
else if (strcmp(name, "zstd") == 0)
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
*algorithm = PG_COMPRESSION_ZSTD;
|
2022-03-23 14:19:14 +01:00
|
|
|
else
|
|
|
|
return false;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get the human-readable name corresponding to a particular compression
|
|
|
|
* algorithm.
|
|
|
|
*/
|
|
|
|
const char *
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
get_compress_algorithm_name(pg_compress_algorithm algorithm)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
|
|
|
switch (algorithm)
|
|
|
|
{
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
case PG_COMPRESSION_NONE:
|
2022-03-23 14:19:14 +01:00
|
|
|
return "none";
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
case PG_COMPRESSION_GZIP:
|
2022-03-23 14:19:14 +01:00
|
|
|
return "gzip";
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
case PG_COMPRESSION_LZ4:
|
2022-03-23 14:19:14 +01:00
|
|
|
return "lz4";
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
case PG_COMPRESSION_ZSTD:
|
2022-03-23 14:19:14 +01:00
|
|
|
return "zstd";
|
|
|
|
/* no default, to provoke compiler warnings if values are added */
|
|
|
|
}
|
|
|
|
Assert(false);
|
2022-03-23 16:37:12 +01:00
|
|
|
return "???"; /* placate compiler */
|
2022-03-23 14:19:14 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Parse a compression specification for a specified algorithm.
|
|
|
|
*
|
|
|
|
* See the file header comments for a brief description of what a compression
|
|
|
|
* specification is expected to look like.
|
|
|
|
*
|
|
|
|
* On return, all fields of the result object will be initialized.
|
|
|
|
* In particular, result->parse_error will be NULL if no errors occurred
|
|
|
|
* during parsing, and will otherwise contain an appropriate error message.
|
|
|
|
* The caller may free this error message string using pfree, if desired.
|
|
|
|
* Note, however, even if there's no parse error, the string might not make
|
|
|
|
* sense: e.g. for gzip, level=12 is not sensible, but it does parse OK.
|
|
|
|
*
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
* The compression level is assigned by default if not directly specified
|
|
|
|
* by the specification.
|
|
|
|
*
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
* Use validate_compress_specification() to find out whether a compression
|
2022-03-23 14:19:14 +01:00
|
|
|
* specification is semantically sensible.
|
|
|
|
*/
|
|
|
|
void
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
parse_compress_specification(pg_compress_algorithm algorithm, char *specification,
|
|
|
|
pg_compress_specification *result)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
|
|
|
int bare_level;
|
|
|
|
char *bare_level_endp;
|
|
|
|
|
|
|
|
/* Initial setup of result object. */
|
|
|
|
result->algorithm = algorithm;
|
|
|
|
result->options = 0;
|
|
|
|
result->parse_error = NULL;
|
|
|
|
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
/*
|
|
|
|
* Assign a default level depending on the compression method. This may
|
|
|
|
* be enforced later.
|
|
|
|
*/
|
|
|
|
switch (result->algorithm)
|
|
|
|
{
|
|
|
|
case PG_COMPRESSION_NONE:
|
|
|
|
result->level = 0;
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_LZ4:
|
|
|
|
#ifdef USE_LZ4
|
|
|
|
result->level = 0; /* fast compression mode */
|
|
|
|
#else
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("this build does not support compression with %s"),
|
|
|
|
"LZ4");
|
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_ZSTD:
|
|
|
|
#ifdef USE_ZSTD
|
|
|
|
result->level = ZSTD_CLEVEL_DEFAULT;
|
|
|
|
#else
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("this build does not support compression with %s"),
|
|
|
|
"ZSTD");
|
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_GZIP:
|
|
|
|
#ifdef HAVE_LIBZ
|
|
|
|
result->level = Z_DEFAULT_COMPRESSION;
|
|
|
|
#else
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("this build does not support compression with %s"),
|
|
|
|
"gzip");
|
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2022-03-23 14:19:14 +01:00
|
|
|
/* If there is no specification, we're done already. */
|
|
|
|
if (specification == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* As a special case, the specification can be a bare integer. */
|
|
|
|
bare_level = strtol(specification, &bare_level_endp, 10);
|
|
|
|
if (specification != bare_level_endp && *bare_level_endp == '\0')
|
|
|
|
{
|
|
|
|
result->level = bare_level;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Look for comma-separated keyword or keyword=value entries. */
|
|
|
|
while (1)
|
|
|
|
{
|
|
|
|
char *kwstart;
|
|
|
|
char *kwend;
|
|
|
|
char *vstart;
|
|
|
|
char *vend;
|
|
|
|
int kwlen;
|
|
|
|
int vlen;
|
|
|
|
bool has_value;
|
|
|
|
char *keyword;
|
|
|
|
char *value;
|
|
|
|
|
|
|
|
/* Figure start, end, and length of next keyword and any value. */
|
|
|
|
kwstart = kwend = specification;
|
|
|
|
while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
|
|
|
|
++kwend;
|
|
|
|
kwlen = kwend - kwstart;
|
|
|
|
if (*kwend != '=')
|
|
|
|
{
|
|
|
|
vstart = vend = NULL;
|
|
|
|
vlen = 0;
|
|
|
|
has_value = false;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
vstart = vend = kwend + 1;
|
|
|
|
while (*vend != '\0' && *vend != ',')
|
|
|
|
++vend;
|
|
|
|
vlen = vend - vstart;
|
|
|
|
has_value = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Reject empty keyword. */
|
|
|
|
if (kwlen == 0)
|
|
|
|
{
|
|
|
|
result->parse_error =
|
|
|
|
pstrdup(_("found empty string where a compression option was expected"));
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Extract keyword and value as separate C strings. */
|
|
|
|
keyword = palloc(kwlen + 1);
|
|
|
|
memcpy(keyword, kwstart, kwlen);
|
|
|
|
keyword[kwlen] = '\0';
|
|
|
|
if (!has_value)
|
|
|
|
value = NULL;
|
|
|
|
else
|
|
|
|
{
|
|
|
|
value = palloc(vlen + 1);
|
|
|
|
memcpy(value, vstart, vlen);
|
|
|
|
value[vlen] = '\0';
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Handle whatever keyword we found. */
|
|
|
|
if (strcmp(keyword, "level") == 0)
|
|
|
|
{
|
|
|
|
result->level = expect_integer_value(keyword, value, result);
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* No need to set a flag in "options", there is a default level
|
|
|
|
* set at least thanks to the logic above.
|
|
|
|
*/
|
2022-03-23 14:19:14 +01:00
|
|
|
}
|
2022-03-30 15:35:14 +02:00
|
|
|
else if (strcmp(keyword, "workers") == 0)
|
|
|
|
{
|
|
|
|
result->workers = expect_integer_value(keyword, value, result);
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
result->options |= PG_COMPRESSION_OPTION_WORKERS;
|
2022-03-30 15:35:14 +02:00
|
|
|
}
|
2023-04-06 17:18:38 +02:00
|
|
|
else if (strcmp(keyword, "long") == 0)
|
|
|
|
{
|
|
|
|
result->long_distance = expect_boolean_value(keyword, value, result);
|
|
|
|
result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
|
|
|
|
}
|
2022-03-23 14:19:14 +01:00
|
|
|
else
|
|
|
|
result->parse_error =
|
2022-09-25 00:38:35 +02:00
|
|
|
psprintf(_("unrecognized compression option: \"%s\""), keyword);
|
2022-03-23 14:19:14 +01:00
|
|
|
|
|
|
|
/* Release memory, just to be tidy. */
|
|
|
|
pfree(keyword);
|
|
|
|
if (value != NULL)
|
|
|
|
pfree(value);
|
|
|
|
|
2022-03-30 21:53:08 +02:00
|
|
|
/*
|
|
|
|
* If we got an error or have reached the end of the string, stop.
|
|
|
|
*
|
|
|
|
* If there is no value, then the end of the keyword might have been
|
|
|
|
* the end of the string. If there is a value, then the end of the
|
|
|
|
* keyword cannot have been the end of the string, but the end of the
|
|
|
|
* value might have been.
|
|
|
|
*/
|
|
|
|
if (result->parse_error != NULL ||
|
|
|
|
(vend == NULL ? *kwend == '\0' : *vend == '\0'))
|
2022-03-23 14:19:14 +01:00
|
|
|
break;
|
|
|
|
|
|
|
|
/* Advance to next entry and loop around. */
|
|
|
|
specification = vend == NULL ? kwend + 1 : vend + 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Parse 'value' as an integer and return the result.
|
|
|
|
*
|
|
|
|
* If parsing fails, set result->parse_error to an appropriate message
|
|
|
|
* and return -1.
|
|
|
|
*/
|
|
|
|
static int
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
expect_integer_value(char *keyword, char *value, pg_compress_specification *result)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
|
|
|
int ivalue;
|
|
|
|
char *ivalue_endp;
|
|
|
|
|
|
|
|
if (value == NULL)
|
|
|
|
{
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("compression option \"%s\" requires a value"),
|
|
|
|
keyword);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
ivalue = strtol(value, &ivalue_endp, 10);
|
|
|
|
if (ivalue_endp == value || *ivalue_endp != '\0')
|
|
|
|
{
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("value for compression option \"%s\" must be an integer"),
|
|
|
|
keyword);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
return ivalue;
|
|
|
|
}
|
|
|
|
|
2023-04-06 17:18:38 +02:00
|
|
|
/*
|
|
|
|
* Parse 'value' as a boolean and return the result.
|
|
|
|
*
|
|
|
|
* If parsing fails, set result->parse_error to an appropriate message
|
|
|
|
* and return -1. The caller must check result->parse_error to determine if
|
|
|
|
* the call was successful.
|
|
|
|
*
|
|
|
|
* Valid values are: yes, no, on, off, 1, 0.
|
|
|
|
*
|
|
|
|
* Inspired by ParseVariableBool().
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
|
|
|
|
{
|
|
|
|
if (value == NULL)
|
|
|
|
return true;
|
|
|
|
|
|
|
|
if (pg_strcasecmp(value, "yes") == 0)
|
|
|
|
return true;
|
|
|
|
if (pg_strcasecmp(value, "on") == 0)
|
|
|
|
return true;
|
|
|
|
if (pg_strcasecmp(value, "1") == 0)
|
|
|
|
return true;
|
|
|
|
|
|
|
|
if (pg_strcasecmp(value, "no") == 0)
|
|
|
|
return false;
|
|
|
|
if (pg_strcasecmp(value, "off") == 0)
|
|
|
|
return false;
|
|
|
|
if (pg_strcasecmp(value, "0") == 0)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
result->parse_error =
|
|
|
|
psprintf(_("value for compression option \"%s\" must be a boolean"),
|
|
|
|
keyword);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2022-03-23 14:19:14 +01:00
|
|
|
/*
|
|
|
|
* Returns NULL if the compression specification string was syntactically
|
|
|
|
* valid and semantically sensible. Otherwise, returns an error message.
|
|
|
|
*
|
|
|
|
* Does not test whether this build of PostgreSQL supports the requested
|
|
|
|
* compression method.
|
|
|
|
*/
|
|
|
|
char *
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
validate_compress_specification(pg_compress_specification *spec)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
int min_level = 1;
|
|
|
|
int max_level = 1;
|
|
|
|
int default_level = 0;
|
|
|
|
|
2022-03-23 14:19:14 +01:00
|
|
|
/* If it didn't even parse OK, it's definitely no good. */
|
|
|
|
if (spec->parse_error != NULL)
|
|
|
|
return spec->parse_error;
|
|
|
|
|
|
|
|
/*
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
* Check that the algorithm expects a compression level and it is within
|
|
|
|
* the legal range for the algorithm.
|
2022-03-23 14:19:14 +01:00
|
|
|
*/
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
switch (spec->algorithm)
|
2022-03-23 14:19:14 +01:00
|
|
|
{
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
case PG_COMPRESSION_GZIP:
|
2022-03-23 14:19:14 +01:00
|
|
|
max_level = 9;
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
#ifdef HAVE_LIBZ
|
|
|
|
default_level = Z_DEFAULT_COMPRESSION;
|
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_LZ4:
|
2022-03-23 14:19:14 +01:00
|
|
|
max_level = 12;
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
default_level = 0; /* fast mode */
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_ZSTD:
|
|
|
|
#ifdef USE_ZSTD
|
2022-09-22 13:02:40 +02:00
|
|
|
max_level = ZSTD_maxCLevel();
|
|
|
|
min_level = ZSTD_minCLevel();
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
default_level = ZSTD_CLEVEL_DEFAULT;
|
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
case PG_COMPRESSION_NONE:
|
|
|
|
if (spec->level != 0)
|
|
|
|
return psprintf(_("compression algorithm \"%s\" does not accept a compression level"),
|
|
|
|
get_compress_algorithm_name(spec->algorithm));
|
|
|
|
break;
|
2022-03-23 14:19:14 +01:00
|
|
|
}
|
|
|
|
|
Simplify handling of compression level with compression specifications
PG_COMPRESSION_OPTION_LEVEL is removed from the compression
specification logic, and instead the compression level is always
assigned with each library's default if nothing is directly given. This
centralizes the checks on the compression methods supported by a given
build, and always assigns a default compression level when parsing a
compression specification. This results in complaining at an earlier
stage than previously if a build supports a compression method or not,
aka when parsing a specification in the backend or the frontend, and not
when processing it. zstd, lz4 and zlib are able to handle in their
respective routines setting up the compression level the case of a
default value, hence the backend or frontend code (pg_receivewal or
pg_basebackup) has now no need to know what the default compression
level should be if nothing is specified: the logic is now done so as the
specification parsing assigns it. It can also be enforced by passing
down a "level" set to the default value, that the backend will accept
(the replication protocol is for example able to handle a command like
BASE_BACKUP (COMPRESSION_DETAIL 'gzip:level=-1')).
This code simplification fixes an issue with pg_basebackup --gzip
introduced by ffd5365, where the tarball of the streamed WAL segments
would be created as of pg_wal.tar.gz with uncompressed contents, while
the intention is to compress the segments with gzip at a default level.
The origin of the confusion comes from the handling of the default
compression level of gzip (-1 or Z_DEFAULT_COMPRESSION) and the value of
0 was getting assigned, which is what walmethods.c would consider
as equivalent to no compression when streaming WAL segments with its tar
methods. Assigning always the compression level removes the confusion
of some code paths considering a value of 0 set in a specification as
either no compression or a default compression level.
Note that 010_pg_basebackup.pl has to be adjusted to skip a few tests
where the shape of the compression detail string for client and
server-side compression was checked using gzip. This is a result of the
code simplification, as gzip specifications cannot be used if a build
does not support it.
Reported-by: Tom Lane
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us
Backpatch-through: 15
2022-09-14 05:16:57 +02:00
|
|
|
if ((spec->level < min_level || spec->level > max_level) &&
|
|
|
|
spec->level != default_level)
|
|
|
|
return psprintf(_("compression algorithm \"%s\" expects a compression level between %d and %d (default at %d)"),
|
|
|
|
get_compress_algorithm_name(spec->algorithm),
|
|
|
|
min_level, max_level, default_level);
|
|
|
|
|
2022-03-30 15:35:14 +02:00
|
|
|
/*
|
|
|
|
* Of the compression algorithms that we currently support, only zstd
|
|
|
|
* allows parallel workers.
|
|
|
|
*/
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
if ((spec->options & PG_COMPRESSION_OPTION_WORKERS) != 0 &&
|
|
|
|
(spec->algorithm != PG_COMPRESSION_ZSTD))
|
2022-03-30 15:35:14 +02:00
|
|
|
{
|
|
|
|
return psprintf(_("compression algorithm \"%s\" does not accept a worker count"),
|
Rename backup_compression.{c,h} to compression.{c,h}
Compression option handling (level, algorithm or even workers) can be
used across several parts of the system and not only base backups.
Structures, objects and routines are renamed in consequence, to remove
the concept of base backups from this part of the code making this
change straight-forward.
pg_receivewal, that has gained support for LZ4 since babbbb5, will make
use of this infrastructure for its set of compression options, bringing
more consistency with pg_basebackup. This cleanup needs to be done
before releasing a beta of 15. pg_dump is a potential future target, as
well, and adding more compression options to it may happen in 16~.
Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12 06:38:54 +02:00
|
|
|
get_compress_algorithm_name(spec->algorithm));
|
2022-03-30 15:35:14 +02:00
|
|
|
}
|
|
|
|
|
2023-04-06 17:18:38 +02:00
|
|
|
/*
|
|
|
|
* Of the compression algorithms that we currently support, only zstd
|
|
|
|
* supports long-distance mode.
|
|
|
|
*/
|
|
|
|
if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
|
|
|
|
(spec->algorithm != PG_COMPRESSION_ZSTD))
|
|
|
|
{
|
|
|
|
return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
|
|
|
|
get_compress_algorithm_name(spec->algorithm));
|
|
|
|
}
|
|
|
|
|
2022-03-23 14:19:14 +01:00
|
|
|
return NULL;
|
|
|
|
}
|
Refactor code parsing compression option values (-Z/--compress)
This commit moves the code in charge of deparsing the method and detail
strings fed later to parse_compress_specification() to a common routine,
where the backward-compatible case of only an integer being found (N
= 0 => "none", N > 1 => gzip at level N) is handled.
Note that this has a side-effect for pg_basebackup, as we now attempt to
detect "server-" and "client-" before checking for the integer-only
pre-14 grammar, where values like server-N and client-N (without the
follow-up detail string) are now valid rather than failing because of an
unsupported method name. Past grammars are still handled the same way,
but these flavors are now authorized, and would now switch to consider N
= 0 as no compression and N > 1 as gzip with the compression level used
as N, with the caller still controlling if the compression method should
be done server-side, client-side or is unspecified. The documentation
of pg_basebackup is updated to reflect that.
This benefits other code paths that would like to rely on the same logic
as pg_basebackup and pg_receivewal with option values used for
compression specifications, one area discussed lately being pg_dump.
Author: Georgios Kokolatos, Michael Paquier
Discussion: https://postgr.es/m/O4mutIrCES8ZhlXJiMvzsivT7ztAMja2lkdL1LJx6O5f22I2W8PBIeLKz7mDLwxHoibcnRAYJXm1pH4tyUNC4a8eDzLn22a6Pb1S74Niexg=@pm.me
2022-11-30 01:34:32 +01:00
|
|
|
|
|
|
|
#ifdef FRONTEND
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Basic parsing of a value specified through a command-line option, commonly
|
|
|
|
* -Z/--compress.
|
|
|
|
*
|
|
|
|
* The parsing consists of a METHOD:DETAIL string fed later to
|
|
|
|
* parse_compress_specification(). This only extracts METHOD and DETAIL.
|
|
|
|
* If only an integer is found, the method is implied by the value specified.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
parse_compress_options(const char *option, char **algorithm, char **detail)
|
|
|
|
{
|
|
|
|
char *sep;
|
|
|
|
char *endp;
|
|
|
|
long result;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check whether the compression specification consists of a bare integer.
|
|
|
|
*
|
|
|
|
* For backward-compatibility, assume "none" if the integer found is zero
|
|
|
|
* and "gzip" otherwise.
|
|
|
|
*/
|
|
|
|
result = strtol(option, &endp, 10);
|
|
|
|
if (*endp == '\0')
|
|
|
|
{
|
|
|
|
if (result == 0)
|
|
|
|
{
|
|
|
|
*algorithm = pstrdup("none");
|
|
|
|
*detail = NULL;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
*algorithm = pstrdup("gzip");
|
|
|
|
*detail = pstrdup(option);
|
|
|
|
}
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check whether there is a compression detail following the algorithm
|
|
|
|
* name.
|
|
|
|
*/
|
|
|
|
sep = strchr(option, ':');
|
|
|
|
if (sep == NULL)
|
|
|
|
{
|
|
|
|
*algorithm = pstrdup(option);
|
|
|
|
*detail = NULL;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
char *alg;
|
|
|
|
|
|
|
|
alg = palloc((sep - option) + 1);
|
|
|
|
memcpy(alg, option, sep - option);
|
|
|
|
alg[sep - option] = '\0';
|
|
|
|
|
|
|
|
*algorithm = alg;
|
|
|
|
*detail = pstrdup(sep + 1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif /* FRONTEND */
|