postgresql/doc/src/sgml/ref
Robert Haas bbe0a81db6 Allow configurable LZ4 TOAST compression.
There is now a per-column COMPRESSION option which can be set to pglz
(the default, and the only option in up until now) or lz4. Or, if you
like, you can set the new default_toast_compression GUC to lz4, and
then that will be the default for new table columns for which no value
is specified. We don't have lz4 support in the PostgreSQL code, so
to use lz4 compression, PostgreSQL must be built --with-lz4.

In general, TOAST compression means compression of individual column
values, not the whole tuple, and those values can either be compressed
inline within the tuple or compressed and then stored externally in
the TOAST table, so those properties also apply to this feature.

Prior to this commit, a TOAST pointer has two unused bits as part of
the va_extsize field, and a compessed datum has two unused bits as
part of the va_rawsize field. These bits are unused because the length
of a varlena is limited to 1GB; we now use them to indicate the
compression type that was used. This means we only have bit space for
2 more built-in compresison types, but we could work around that
problem, if necessary, by introducing a new vartag_external value for
any further types we end up wanting to add. Hopefully, it won't be
too important to offer a wide selection of algorithms here, since
each one we add not only takes more coding but also adds a build
dependency for every packager. Nevertheless, it seems worth doing
at least this much, because LZ4 gets better compression than PGLZ
with less CPU usage.

It's possible for LZ4-compressed datums to leak into composite type
values stored on disk, just as it is for PGLZ. It's also possible for
LZ4-compressed attributes to be copied into a different table via SQL
commands such as CREATE TABLE AS or INSERT .. SELECT.  It would be
expensive to force such values to be decompressed, so PostgreSQL has
never done so. For the same reasons, we also don't force recompression
of already-compressed values even if the target table prefers a
different compression method than was used for the source data.  These
architectural decisions are perhaps arguable but revisiting them is
well beyond the scope of what seemed possible to do as part of this
project.  However, it's relatively cheap to recompress as part of
VACUUM FULL or CLUSTER, so this commit adjusts those commands to do
so, if the configured compression method of the table happens not to
match what was used for some column value stored therein.

Dilip Kumar. The original patches on which this work was based were
written by Ildus Kurbangaliev, and those were patches were based on
even earlier work by Nikita Glukhov, but the design has since changed
very substantially, since allow a potentially large number of
compression methods that could be added and dropped on a running
system proved too problematic given some of the architectural issues
mentioned above; the choice of which specific compression method to
add first is now different; and a lot of the code has been heavily
refactored.  More recently, Justin Przyby helped quite a bit with
testing and reviewing and this version also includes some code
contributions from him. Other design input and review from Tomas
Vondra, Álvaro Herrera, Andres Freund, Oleg Bartunov, Alexander
Korotkov, and me.

Discussion: http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain
Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKebw%40mail.gmail.com
2021-03-19 15:10:38 -04:00
..
abort.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
allfiles.sgml Add pg_amcheck, a CLI for contrib/amcheck. 2021-03-12 13:00:01 -05:00
alter_aggregate.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_collation.sgml Remove pg_collation.collversion. 2020-11-03 00:44:59 +13:00
alter_conversion.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_database.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_default_privileges.sgml Doc: clarify behavior of ALTER DEFAULT PRIVILEGES ... IN SCHEMA. 2019-11-19 14:21:41 -05:00
alter_domain.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_event_trigger.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_extension.sgml Support for OUT parameters in procedures 2020-10-05 09:21:43 +02:00
alter_foreign_data_wrapper.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_foreign_table.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_function.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_group.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_index.sgml doc: Mention NO DEPENDS ON EXTENSION in its supported ALTER commands 2021-02-13 16:06:11 +09:00
alter_language.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_large_object.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_materialized_view.sgml doc: Mention NO DEPENDS ON EXTENSION in its supported ALTER commands 2021-02-13 16:06:11 +09:00
alter_opclass.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_operator.sgml Remove support for postfix (right-unary) operators. 2020-09-17 19:38:05 -04:00
alter_opfamily.sgml Remove support for postfix (right-unary) operators. 2020-09-17 19:38:05 -04:00
alter_policy.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_procedure.sgml doc: Mention NO DEPENDS ON EXTENSION in its supported ALTER commands 2021-02-13 16:06:11 +09:00
alter_publication.sgml Doc: fix description of privileges needed for ALTER PUBLICATION. 2021-01-12 12:52:14 -05:00
alter_role.sgml Allow users with BYPASSRLS to alter their own passwords. 2020-11-03 15:41:32 -05:00
alter_routine.sgml doc: Mention NO DEPENDS ON EXTENSION in its supported ALTER commands 2021-02-13 16:06:11 +09:00
alter_rule.sgml
alter_schema.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_sequence.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_server.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_statistics.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_subscription.sgml Fix some typos, grammar and style in docs and comments 2021-02-24 16:13:17 +09:00
alter_system.sgml
alter_table.sgml Allow configurable LZ4 TOAST compression. 2021-03-19 15:10:38 -04:00
alter_tablespace.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_trigger.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_tsconfig.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_tsdictionary.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_tsparser.sgml
alter_tstemplate.sgml
alter_type.sgml Allow ALTER TYPE to update an existing type's typsubscript value. 2020-12-11 18:58:21 -05:00
alter_user.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
alter_user_mapping.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
alter_view.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
analyze.sgml doc: Add backlinks to progress reporting documentation 2021-03-05 14:58:16 +09:00
begin.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
call.sgml Fix bugs in plpgsql's handling of CALL argument lists. 2018-11-04 13:25:39 -05:00
checkpoint.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
close.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
cluster.sgml doc: Add backlinks to progress reporting documentation 2021-03-05 14:58:16 +09:00
clusterdb.sgml Fix connection string handling in src/bin/scripts/ programs. 2020-10-19 19:03:46 -04:00
comment.sgml Support for OUT parameters in procedures 2020-10-05 09:21:43 +02:00
commit.sgml Fix behavior of AND CHAIN outside of explicit transaction blocks 2019-09-08 16:23:03 +02:00
commit_prepared.sgml doc: Remove unused title ids 2020-03-13 15:45:37 +01:00
copy.sgml doc: Add backlinks to progress reporting documentation 2021-03-05 14:58:16 +09:00
create_access_method.sgml tableam: basic documentation. 2019-04-03 17:40:29 -07:00
create_aggregate.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_cast.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_collation.sgml Remove pg_collation.collversion. 2020-11-03 00:44:59 +13:00
create_conversion.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
create_database.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_domain.sgml Doc: document that we expect CHECK constraint conditions to be immutable. 2018-12-07 16:40:58 -05:00
create_event_trigger.sgml doc: add commas after 'i.e.' and 'e.g.' 2020-08-31 18:33:37 -04:00
create_extension.sgml Make contrib modules' installation scripts more secure. 2020-08-10 10:44:42 -04:00
create_foreign_data_wrapper.sgml
create_foreign_table.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_function.sgml Doc: improve introductory information about procedures. 2021-03-10 11:33:50 -05:00
create_group.sgml Doc: sync CREATE GROUP syntax synopsis with CREATE ROLE. 2020-04-10 10:44:09 -04:00
create_index.sgml Doc: B-Tree only has one additional parameter. 2021-03-10 22:10:36 -08:00
create_language.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_materialized_view.sgml Remove INSERT privilege check at table creation of CTAS and matview 2020-11-21 19:45:30 +09:00
create_opclass.sgml Remove support for postfix (right-unary) operators. 2020-09-17 19:38:05 -04:00
create_operator.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_opfamily.sgml
create_policy.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
create_procedure.sgml Doc: improve introductory information about procedures. 2021-03-10 11:33:50 -05:00
create_publication.sgml doc: Fix man page whitespace issues 2020-06-07 14:54:28 +02:00
create_role.sgml Doc: get rid of <foreignphrase> tags. 2021-03-10 12:38:43 -05:00
create_rule.sgml
create_schema.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
create_sequence.sgml doc: Use proper em and en dashes 2019-10-25 20:39:41 +02:00
create_server.sgml docs: fix spacing around "if not exists" brackets 2018-03-29 21:25:39 -04:00
create_statistics.sgml doc: add commas after 'i.e.' and 'e.g.' 2020-08-31 18:33:37 -04:00
create_subscription.sgml doc: Fix some grammar and inconsistencies 2020-09-10 15:50:19 +09:00
create_table.sgml Allow configurable LZ4 TOAST compression. 2021-03-19 15:10:38 -04:00
create_table_as.sgml Remove INSERT privilege check at table creation of CTAS and matview 2020-11-21 19:45:30 +09:00
create_tablespace.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
create_transform.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_trigger.sgml Provide the OR REPLACE option for CREATE TRIGGER. 2020-11-14 17:05:34 -05:00
create_tsconfig.sgml
create_tsdictionary.sgml
create_tsparser.sgml
create_tstemplate.sgml
create_type.sgml Fix some typos, grammar and style in docs and comments 2021-02-24 16:13:17 +09:00
create_user.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
create_user_mapping.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
create_view.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
createdb.sgml Fix connection string handling in src/bin/scripts/ programs. 2020-10-19 19:03:46 -04:00
createuser.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
deallocate.sgml
declare.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
delete.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
discard.sgml Fix order of steps in DISCARD ALL documentation 2019-06-11 12:22:11 -04:00
do.sgml doc: Remove unused title ids 2020-03-13 15:45:37 +01:00
drop_access_method.sgml
drop_aggregate.sgml
drop_cast.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
drop_collation.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
drop_conversion.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
drop_database.sgml doc: Trim trailing whitespace 2020-06-07 13:24:40 +02:00
drop_domain.sgml
drop_event_trigger.sgml
drop_extension.sgml
drop_foreign_data_wrapper.sgml
drop_foreign_table.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
drop_function.sgml doc: Fix man page whitespace issues 2020-06-07 14:54:28 +02:00
drop_group.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_index.sgml Fix some typos, grammar and style in docs and comments 2021-02-24 16:13:17 +09:00
drop_language.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_materialized_view.sgml
drop_opclass.sgml
drop_operator.sgml Remove support for postfix (right-unary) operators. 2020-09-17 19:38:05 -04:00
drop_opfamily.sgml
drop_owned.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_policy.sgml
drop_procedure.sgml Support for OUT parameters in procedures 2020-10-05 09:21:43 +02:00
drop_publication.sgml
drop_role.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_routine.sgml doc: Fix man page whitespace issues 2020-06-07 14:54:28 +02:00
drop_rule.sgml
drop_schema.sgml
drop_sequence.sgml
drop_server.sgml
drop_statistics.sgml
drop_subscription.sgml Allow multiple xacts during table sync in logical replication. 2021-02-12 07:41:51 +05:30
drop_table.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_tablespace.sgml
drop_transform.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
drop_trigger.sgml
drop_tsconfig.sgml
drop_tsdictionary.sgml
drop_tsparser.sgml
drop_tstemplate.sgml
drop_type.sgml
drop_user.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
drop_user_mapping.sgml Allow CURRENT_ROLE where CURRENT_USER is accepted 2020-09-17 11:40:08 +02:00
drop_view.sgml
dropdb.sgml Fix connection string handling in src/bin/scripts/ programs. 2020-10-19 19:03:46 -04:00
dropuser.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
ecpg-ref.sgml Fix our getopt_long's behavior for a command line argument of just "-". 2020-03-23 11:58:00 -04:00
end.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
execute.sgml Doc: fix "Unresolved ID reference" warnings, clean up man page cross-refs. 2020-05-11 14:15:55 -04:00
explain.sgml Fix typos and grammar in docs and comments 2020-12-24 17:05:49 +09:00
fetch.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
grant.sgml Allow GRANTED BY clause in normal GRANT and REVOKE statements 2021-01-30 09:45:11 +01:00
import_foreign_schema.sgml
initdb.sgml Add documentation chapter about checksums 2021-01-17 15:31:23 +01:00
insert.sgml Doc: fix "Unresolved ID reference" warnings, clean up man page cross-refs. 2020-05-11 14:15:55 -04:00
listen.sgml Doc: improve discussion of race conditions involved in LISTEN. 2019-11-24 18:03:39 -05:00
load.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
lock.sgml Revert "Accept relations of any kind in LOCK TABLE". 2020-11-06 16:17:56 -05:00
move.sgml
notify.sgml Use a hash table to de-duplicate NOTIFY events faster. 2019-08-15 12:22:12 -04:00
pg_amcheck.sgml Add pg_amcheck, a CLI for contrib/amcheck. 2021-03-12 13:00:01 -05:00
pg_basebackup.sgml Prevent buffer overrun in read_tablespace_map(). 2021-03-17 16:10:37 -04:00
pg_checksums.sgml doc: remove unnecessary blank before command option text 2020-12-03 11:33:24 -05:00
pg_config-ref.sgml Add missing <application> tags in application doc <refentrytitle>s 2020-07-10 16:51:29 +02:00
pg_controldata.sgml doc: remove unnecessary blank before command option text 2020-12-03 11:33:24 -05:00
pg_ctl-ref.sgml Revert "Add key management system" (978f869b99) & later commits 2020-12-27 21:37:42 -05:00
pg_dump.sgml Doc: clarify behavior of back-half options in pg_dump. 2021-01-13 13:30:04 -05:00
pg_dumpall.sgml doc: fix spelling "connction" to "connection" 2020-11-10 19:18:35 -05:00
pg_isready.sgml doc: libpq connection options can override command-line flags 2020-10-02 22:19:31 -04:00
pg_receivewal.sgml doc: fix spelling "connction" to "connection" 2020-11-10 19:18:35 -05:00
pg_recvlogical.sgml doc: libpq connection options can override command-line flags 2020-10-02 22:19:31 -04:00
pg_resetwal.sgml doc: remove unnecessary blank before command option text 2020-12-03 11:33:24 -05:00
pg_restore.sgml doc: libpq connection options can override command-line flags 2020-10-02 22:19:31 -04:00
pg_rewind.sgml Allow pg_rewind to use a standby server as the source system. 2020-11-12 14:52:24 +02:00
pg_verifybackup.sgml Fix typos and grammar in docs and comments 2020-12-24 17:05:49 +09:00
pg_waldump.sgml pg_waldump: Add a --quiet option. 2020-04-02 20:25:04 -04:00
pgarchivecleanup.sgml Retire pg_standby. 2021-01-29 14:09:41 +13:00
pgbench.sgml Add libpq pipeline mode support to pgbench 2021-03-15 18:33:03 -03:00
pgtestfsync.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
pgtesttiming.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
pgupgrade.sgml Revert "Add key management system" (978f869b99) & later commits 2020-12-27 21:37:42 -05:00
postgres-ref.sgml Revert "Add key management system" (978f869b99) & later commits 2020-12-27 21:37:42 -05:00
postmaster.sgml doc: Remove unused ids 2020-03-07 14:04:09 +01:00
prepare.sgml Fix typos and grammar in docs and comments 2020-12-24 17:05:49 +09:00
prepare_transaction.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
psql-ref.sgml Allow configurable LZ4 TOAST compression. 2021-03-19 15:10:38 -04:00
reassign_owned.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
refresh_materialized_view.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
reindex.sgml doc: Add backlinks to progress reporting documentation 2021-03-05 14:58:16 +09:00
reindexdb.sgml Add --tablespace option to reindexdb 2021-03-03 10:14:21 +09:00
release_savepoint.sgml
reset.sgml
revoke.sgml Allow GRANTED BY clause in normal GRANT and REVOKE statements 2021-01-30 09:45:11 +01:00
rollback.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
rollback_prepared.sgml doc: Remove unused title ids 2020-03-13 15:45:37 +01:00
rollback_to.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
savepoint.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
security_label.sgml Support for OUT parameters in procedures 2020-10-05 09:21:43 +02:00
select.sgml Implement GROUP BY DISTINCT 2021-03-18 18:22:18 +01:00
select_into.sgml doc: Clarify status of SELECT INTO on reference page 2021-01-30 11:21:32 +01:00
set.sgml Doc: improve cross-references for SET/SHOW. 2021-01-29 10:46:14 -05:00
set_constraints.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
set_role.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
set_session_auth.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
set_transaction.sgml doc: make ref/*.sgml file header comment layout consistent 2020-05-15 08:52:24 -04:00
show.sgml Doc: improve cross-references for SET/SHOW. 2021-01-29 10:46:14 -05:00
start_transaction.sgml Improve <xref> vs. <command> formatting in the documentation 2020-10-03 16:40:02 +02:00
truncate.sgml Fix TRUNCATE doc: ALTER SEQUENCE RESTART is now transactional. 2020-10-19 19:02:25 +03:00
unlisten.sgml
update.sgml Doc: fix "Unresolved ID reference" warnings, clean up man page cross-refs. 2020-05-11 14:15:55 -04:00
vacuum.sgml doc: Add backlinks to progress reporting documentation 2021-03-05 14:58:16 +09:00
vacuumdb.sgml Add option PROCESS_TOAST to VACUUM 2021-02-09 14:13:57 +09:00
values.sgml Doc: fix "Unresolved ID reference" warnings, clean up man page cross-refs. 2020-05-11 14:15:55 -04:00