postgresql/contrib
Peter Geoghegan 0d861bbb70 Add deduplication to nbtree.
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method.  The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs.  Deduplication is only applied at
the point where a leaf page split would otherwise be required.  New
posting list tuples are formed by merging together existing duplicate
tuples.  The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed.  Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.

The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach.  Most individual inserts of index tuples have
exactly the same overhead as before.  The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits.  The key space of indexes works in the same way as it has
since commit dd299df8 (the commit that made heap TID a tiebreaker
column).

Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key).  The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective.  This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.

A new index storage parameter (deduplicate_items) controls the use of
deduplication.  The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible.  This decision will be
reviewed at the end of the Postgres 13 beta period.

There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values.  The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely.  Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").

Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.

No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12).  However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from.  This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.

Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
    https://postgr.es/m/55E4051B.7020209@postgrespro.ru
    https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
2020-02-26 13:05:30 -08:00
..
adminpack Remove some dead code in contrib/adminpack/ 2020-02-14 12:38:44 +09:00
amcheck Add deduplication to nbtree. 2020-02-26 13:05:30 -08:00
auth_delay Update copyrights for 2020 2020-01-01 12:21:45 -05:00
auto_explain Update copyrights for 2020 2020-01-01 12:21:45 -05:00
bloom Introduce IndexAM fields for parallel vacuum. 2020-01-15 07:24:14 +05:30
btree_gin Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
btree_gist Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
citext Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
cube Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
dblink Fix compile failure. 2020-02-24 18:43:40 -05:00
dict_int Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
dict_xsyn Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
earthdistance Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
file_fdw Update copyrights for 2020 2020-01-01 12:21:45 -05:00
fuzzystrmatch Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
hstore Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
hstore_plperl Mark some contrib modules as "trusted". 2020-02-13 15:02:35 -05:00
hstore_plpython Split all OBJS style lines in makefiles into one-line-per-entry style. 2019-11-05 14:41:07 -08:00
intagg Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
intarray Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
isn Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
jsonb_plperl Mark some contrib modules as "trusted". 2020-02-13 15:02:35 -05:00
jsonb_plpython In jsonb_plpython.c, suppress warning message from gcc 10. 2020-01-30 18:26:12 -05:00
lo Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
ltree Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
ltree_plpython Split all OBJS style lines in makefiles into one-line-per-entry style. 2019-11-05 14:41:07 -08:00
oid2name Clean up newlines following left parentheses 2020-01-30 13:42:14 -03:00
pageinspect Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
passwordcheck Update copyrights for 2020 2020-01-01 12:21:45 -05:00
pg_buffercache Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pg_freespacemap Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pg_prewarm Remove excess parens in ereport() calls 2020-01-30 13:32:04 -03:00
pg_standby Remove useless "return;" lines 2019-11-28 16:48:37 -03:00
pg_stat_statements Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pg_trgm Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pg_visibility Update copyrights for 2020 2020-01-01 12:21:45 -05:00
pgcrypto Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pgrowlocks Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
pgstattuple Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
postgres_fdw Fix compile failure. 2020-02-24 18:43:40 -05:00
seg Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
sepgsql Update copyrights for 2020 2020-01-01 12:21:45 -05:00
spi Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
sslinfo Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
start-scripts Remove contrib/start-scripts/osx/. 2017-11-17 12:53:20 -05:00
tablefunc Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
tcn Mark some contrib modules as "trusted". 2020-02-13 15:02:35 -05:00
test_decoding Stop demanding that top xact must be seen before subxact in decoding. 2020-02-19 08:15:49 +05:30
tsm_system_rows Mark some contrib modules as "trusted". 2020-02-13 15:02:35 -05:00
tsm_system_time Mark some contrib modules as "trusted". 2020-02-13 15:02:35 -05:00
unaccent Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
uuid-ossp Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
vacuumlo Update copyrights for 2020 2020-01-01 12:21:45 -05:00
xml2 Remove support for upgrading extensions from "unpackaged" state. 2020-02-19 16:59:14 -05:00
Makefile Transforms for jsonb to PL/Perl 2018-04-03 09:47:18 -04:00
README Rename 'gmake' to 'make' in docs and recommended commands 2014-02-12 17:29:19 -05:00
contrib-global.mk Respect TEMP_CONFIG when pg_regress_check and friends are called 2016-02-27 12:28:21 -05:00

README

The PostgreSQL contrib tree
---------------------------

This subtree contains porting tools, analysis utilities, and plug-in
features that are not part of the core PostgreSQL system, mainly
because they address a limited audience or are too experimental to be
part of the main source tree.  This does not preclude their
usefulness.

User documentation for each module appears in the main SGML
documentation.

When building from the source distribution, these modules are not
built automatically, unless you build the "world" target.  You can
also build and install them all by running "make all" and "make
install" in this directory; or to build and install just one selected
module, do the same in that module's subdirectory.

Some directories supply new user-defined functions, operators, or
types.  To make use of one of these modules, after you have installed
the code you need to register the new SQL objects in the database
system by executing a CREATE EXTENSION command.  In a fresh database,
you can simply do

    CREATE EXTENSION module_name;

See the PostgreSQL documentation for more information about this
procedure.