Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
\set HIDE_TOAST_COMPRESSION false
|
2021-03-21 21:26:44 +01:00
|
|
|
-- ensure we get stable results regardless of installation's default
|
|
|
|
|
SET default_toast_compression = 'pglz';
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
-- test creating table with compression method
|
|
|
|
|
CREATE TABLE cmdata(f1 text COMPRESSION pglz);
|
|
|
|
|
CREATE INDEX idx ON cmdata(f1);
|
|
|
|
|
INSERT INTO cmdata VALUES(repeat('1234567890', 1000));
|
|
|
|
|
\d+ cmdata
|
|
|
|
|
Table "public.cmdata"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | pglz | |
|
|
|
|
|
Indexes:
|
|
|
|
|
"idx" btree (f1)
|
|
|
|
|
|
|
|
|
|
CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
|
|
|
|
|
INSERT INTO cmdata1 VALUES(repeat('1234567890', 1004));
|
|
|
|
|
\d+ cmdata1
|
|
|
|
|
Table "public.cmdata1"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | lz4 | |
|
|
|
|
|
|
|
|
|
|
-- verify stored compression method in the data
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- decompress data slice
|
|
|
|
|
SELECT SUBSTR(f1, 200, 5) FROM cmdata;
|
|
|
|
|
substr
|
|
|
|
|
--------
|
|
|
|
|
01234
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
|
|
|
|
|
substr
|
|
|
|
|
----------------------------------------------------
|
|
|
|
|
01234567890123456789012345678901234567890123456789
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- copy with table creation
|
|
|
|
|
SELECT * INTO cmmove1 FROM cmdata;
|
|
|
|
|
\d+ cmmove1
|
|
|
|
|
Table "public.cmmove1"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | pglz | |
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmmove1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- copy to existing table
|
|
|
|
|
CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
|
|
|
|
|
INSERT INTO cmmove3 SELECT * FROM cmdata;
|
|
|
|
|
INSERT INTO cmmove3 SELECT * FROM cmdata1;
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmmove3;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
-- test LIKE INCLUDING COMPRESSION
|
|
|
|
|
CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | lz4 | |
|
|
|
|
|
|
|
|
|
|
DROP TABLE cmdata2;
|
|
|
|
|
-- try setting compression for incompressible data type
|
|
|
|
|
CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
|
|
|
|
|
ERROR: column data type integer does not support compression
|
|
|
|
|
-- update using datum from different table
|
|
|
|
|
CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
|
|
|
|
|
INSERT INTO cmmove2 VALUES (repeat('1234567890', 1004));
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmmove2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmmove2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- test externally stored compressed data
|
|
|
|
|
CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
|
|
|
|
|
'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
|
|
|
|
|
CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
|
|
|
|
|
INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000);
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
INSERT INTO cmdata1 SELECT large_val() || repeat('a', 4000);
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT SUBSTR(f1, 200, 5) FROM cmdata1;
|
|
|
|
|
substr
|
|
|
|
|
--------
|
|
|
|
|
01234
|
|
|
|
|
8f14e
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
|
|
|
|
|
substr
|
|
|
|
|
--------
|
|
|
|
|
8f14e
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
DROP TABLE cmdata2;
|
|
|
|
|
--test column type update varlena/non-varlena
|
|
|
|
|
CREATE TABLE cmdata2 (f1 int);
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
|
|
|
|
|
f1 | integer | | | | plain | | |
|
|
|
|
|
|
|
|
|
|
ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar;
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+-------------------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | character varying | | | | extended | pglz | |
|
|
|
|
|
|
|
|
|
|
ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer;
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
|
|
|
|
|
f1 | integer | | | | plain | | |
|
|
|
|
|
|
|
|
|
|
--changing column storage should not impact the compression method
|
|
|
|
|
--but the data should not be compressed
|
|
|
|
|
ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar;
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+-------------------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | character varying | | | | extended | pglz | |
|
|
|
|
|
|
|
|
|
|
ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain;
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+-------------------+-----------+----------+---------+---------+-------------+--------------+-------------
|
|
|
|
|
f1 | character varying | | | | plain | pglz | |
|
|
|
|
|
|
|
|
|
|
INSERT INTO cmdata2 VALUES (repeat('123456789', 800));
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- test compression with materialized view
|
|
|
|
|
CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
|
|
|
|
|
\d+ mv
|
|
|
|
|
Materialized view "public.mv"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
x | text | | | | extended | pglz | |
|
|
|
|
|
View definition:
|
|
|
|
|
SELECT cmdata1.f1 AS x
|
|
|
|
|
FROM cmdata1;
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(x) FROM mv;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
-- test compression with partition
|
|
|
|
|
CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
|
|
|
|
|
CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
|
|
|
|
|
CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
|
|
|
|
|
ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
|
|
|
|
|
INSERT INTO cmpart VALUES (repeat('123456789', 1004));
|
|
|
|
|
INSERT INTO cmpart VALUES (repeat('123456789', 4004));
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmpart1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmpart2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
-- test compression with inheritence, error
|
|
|
|
|
CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
|
|
|
|
|
NOTICE: merging multiple inherited definitions of column "f1"
|
|
|
|
|
ERROR: column "f1" has a compression method conflict
|
|
|
|
|
DETAIL: pglz versus lz4
|
|
|
|
|
CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
|
|
|
|
|
NOTICE: merging column "f1" with inherited definition
|
|
|
|
|
ERROR: column "f1" has a compression method conflict
|
|
|
|
|
DETAIL: pglz versus lz4
|
|
|
|
|
-- test default_toast_compression GUC
|
|
|
|
|
SET default_toast_compression = '';
|
|
|
|
|
ERROR: invalid value for parameter "default_toast_compression": ""
|
2021-03-24 17:36:08 +01:00
|
|
|
HINT: Available values: pglz, lz4.
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
SET default_toast_compression = 'I do not exist compression';
|
|
|
|
|
ERROR: invalid value for parameter "default_toast_compression": "I do not exist compression"
|
2021-03-24 17:36:08 +01:00
|
|
|
HINT: Available values: pglz, lz4.
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
SET default_toast_compression = 'lz4';
|
|
|
|
|
DROP TABLE cmdata2;
|
|
|
|
|
CREATE TABLE cmdata2 (f1 text);
|
|
|
|
|
\d+ cmdata2
|
|
|
|
|
Table "public.cmdata2"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | lz4 | |
|
|
|
|
|
|
2021-03-21 21:26:44 +01:00
|
|
|
SET default_toast_compression = 'pglz';
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
-- test alter compression method
|
|
|
|
|
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4;
|
|
|
|
|
INSERT INTO cmdata VALUES (repeat('123456789', 4004));
|
|
|
|
|
\d+ cmdata
|
|
|
|
|
Table "public.cmdata"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
f1 | text | | | | extended | lz4 | |
|
|
|
|
|
Indexes:
|
|
|
|
|
"idx" btree (f1)
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
-- test alter compression method for the materialized view
|
|
|
|
|
ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4;
|
|
|
|
|
\d+ mv
|
|
|
|
|
Materialized view "public.mv"
|
|
|
|
|
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
|
|
|
|
|
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
|
|
|
|
|
x | text | | | | extended | lz4 | |
|
|
|
|
|
View definition:
|
|
|
|
|
SELECT cmdata1.f1 AS x
|
|
|
|
|
FROM cmdata1;
|
|
|
|
|
|
|
|
|
|
-- test alter compression method for the partitioned table
|
|
|
|
|
ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
|
|
|
|
|
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
|
|
|
|
|
-- new data should be compressed with the current compression method
|
|
|
|
|
INSERT INTO cmpart VALUES (repeat('123456789', 1004));
|
|
|
|
|
INSERT INTO cmpart VALUES (repeat('123456789', 4004));
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmpart1;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
pglz
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmpart2;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
--vacuum full to recompress the data
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
pglz
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
VACUUM FULL cmdata;
|
|
|
|
|
SELECT pg_column_compression(f1) FROM cmdata;
|
|
|
|
|
pg_column_compression
|
|
|
|
|
-----------------------
|
|
|
|
|
lz4
|
|
|
|
|
lz4
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
2021-03-26 00:55:32 +01:00
|
|
|
-- test expression index
|
|
|
|
|
DROP TABLE cmdata2;
|
|
|
|
|
CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4);
|
|
|
|
|
CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2));
|
|
|
|
|
INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM
|
|
|
|
|
generate_series(1, 50) g), VERSION());
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
-- check data is ok
|
|
|
|
|
SELECT length(f1) FROM cmdata;
|
|
|
|
|
length
|
|
|
|
|
--------
|
|
|
|
|
10000
|
|
|
|
|
36036
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT length(f1) FROM cmdata1;
|
|
|
|
|
length
|
|
|
|
|
--------
|
|
|
|
|
10040
|
|
|
|
|
12449
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
|
|
|
|
SELECT length(f1) FROM cmmove1;
|
|
|
|
|
length
|
|
|
|
|
--------
|
|
|
|
|
10000
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
SELECT length(f1) FROM cmmove2;
|
|
|
|
|
length
|
|
|
|
|
--------
|
|
|
|
|
10040
|
|
|
|
|
(1 row)
|
|
|
|
|
|
|
|
|
|
SELECT length(f1) FROM cmmove3;
|
|
|
|
|
length
|
|
|
|
|
--------
|
|
|
|
|
10000
|
|
|
|
|
10040
|
|
|
|
|
(2 rows)
|
|
|
|
|
|
2021-03-22 15:57:08 +01:00
|
|
|
CREATE TABLE badcompresstbl (a text COMPRESSION I_Do_Not_Exist_Compression); -- fails
|
|
|
|
|
ERROR: invalid compression method "i_do_not_exist_compression"
|
|
|
|
|
CREATE TABLE badcompresstbl (a text);
|
|
|
|
|
ALTER TABLE badcompresstbl ALTER a SET COMPRESSION I_Do_Not_Exist_Compression; -- fails
|
|
|
|
|
ERROR: invalid compression method "i_do_not_exist_compression"
|
|
|
|
|
DROP TABLE badcompresstbl;
|
Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.
In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.
Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.
It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT. It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data. These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project. However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.
Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored. More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.
Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 20:10:38 +01:00
|
|
|
\set HIDE_TOAST_COMPRESSION true
|
