mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-09-30 18:01:19 +02:00
0d861bbb70
Deduplication reduces the storage overhead of duplicates in indexes that
use the standard nbtree index access method. The deduplication process
is applied lazily, after the point where opportunistic deletion of
LP_DEAD-marked index tuples occurs. Deduplication is only applied at
the point where a leaf page split would otherwise be required. New
posting list tuples are formed by merging together existing duplicate
tuples. The physical representation of the items on an nbtree leaf page
is made more space efficient by deduplication, but the logical contents
of the page are not changed. Even unique indexes make use of
deduplication as a way of controlling bloat from duplicates whose TIDs
point to different versions of the same logical table row.
The lazy approach taken by nbtree has significant advantages over a GIN
style eager approach. Most individual inserts of index tuples have
exactly the same overhead as before. The extra overhead of
deduplication is amortized across insertions, just like the overhead of
page splits. The key space of indexes works in the same way as it has
since commit dd299df8
(the commit that made heap TID a tiebreaker
column).
Testing has shown that nbtree deduplication can generally make indexes
with about 10 or 15 tuples for each distinct key value about 2.5X - 4X
smaller, even with single column integer indexes (e.g., an index on a
referencing column that accompanies a foreign key). The final size of
single column nbtree indexes comes close to the final size of a similar
contrib/btree_gin index, at least in cases where GIN's posting list
compression isn't very effective. This can significantly improve
transaction throughput, and significantly reduce the cost of vacuuming
indexes.
A new index storage parameter (deduplicate_items) controls the use of
deduplication. The default setting is 'on', so all new B-Tree indexes
automatically use deduplication where possible. This decision will be
reviewed at the end of the Postgres 13 beta period.
There is a regression of approximately 2% of transaction throughput with
synthetic workloads that consist of append-only inserts into a table
with several non-unique indexes, where all indexes have few or no
repeated values. The underlying issue is that cycles are wasted on
unsuccessful attempts at deduplicating items in non-unique indexes.
There doesn't seem to be a way around it short of disabling
deduplication entirely. Note that deduplication of items in unique
indexes is fairly well targeted in general, which avoids the problem
there (we can use a special heuristic to trigger deduplication passes in
unique indexes, since we're specifically targeting "version bloat").
Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.
No bump in BTREE_VERSION, since the representation of posting list
tuples works in a way that's backwards compatible with version 4 indexes
(i.e. indexes built on PostgreSQL 12). However, users must still
REINDEX a pg_upgrade'd index to use deduplication, regardless of the
Postgres version they've upgraded from. This is the only way to set the
new nbtree metapage flag indicating that deduplication is generally
safe.
Author: Anastasia Lubennikova, Peter Geoghegan
Reviewed-By: Peter Geoghegan, Heikki Linnakangas
Discussion:
https://postgr.es/m/55E4051B.7020209@postgrespro.ru
https://postgr.es/m/4ab6e2db-bcee-f4cf-0916-3a06e6ccbb55@postgrespro.ru
294 lines
9.7 KiB
Plaintext
294 lines
9.7 KiB
Plaintext
<!-- doc/src/sgml/citext.sgml -->
|
|
|
|
<sect1 id="citext" xreflabel="citext">
|
|
<title>citext</title>
|
|
|
|
<indexterm zone="citext">
|
|
<primary>citext</primary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
The <filename>citext</filename> module provides a case-insensitive
|
|
character string type, <type>citext</type>. Essentially, it internally calls
|
|
<function>lower</function> when comparing values. Otherwise, it behaves almost
|
|
exactly like <type>text</type>.
|
|
</para>
|
|
|
|
<tip>
|
|
<para>
|
|
Consider using <firstterm>nondeterministic collations</firstterm> (see
|
|
<xref linkend="collation-nondeterministic"/>) instead of this module. They
|
|
can be used for case-insensitive comparisons, accent-insensitive
|
|
comparisons, and other combinations, and they handle more Unicode special
|
|
cases correctly.
|
|
</para>
|
|
</tip>
|
|
|
|
<para>
|
|
This module is considered <quote>trusted</quote>, that is, it can be
|
|
installed by non-superusers who have <literal>CREATE</literal> privilege
|
|
on the current database.
|
|
</para>
|
|
|
|
<sect2>
|
|
<title>Rationale</title>
|
|
|
|
<para>
|
|
The standard approach to doing case-insensitive matches
|
|
in <productname>PostgreSQL</productname> has been to use the <function>lower</function>
|
|
function when comparing values, for example
|
|
|
|
<programlisting>
|
|
SELECT * FROM tab WHERE lower(col) = LOWER(?);
|
|
</programlisting>
|
|
</para>
|
|
|
|
<para>
|
|
This works reasonably well, but has a number of drawbacks:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
It makes your SQL statements verbose, and you always have to remember to
|
|
use <function>lower</function> on both the column and the query value.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
It won't use an index, unless you create a functional index using
|
|
<function>lower</function>.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
If you declare a column as <literal>UNIQUE</literal> or <literal>PRIMARY
|
|
KEY</literal>, the implicitly generated index is case-sensitive. So it's
|
|
useless for case-insensitive searches, and it won't enforce
|
|
uniqueness case-insensitively.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>
|
|
The <type>citext</type> data type allows you to eliminate calls
|
|
to <function>lower</function> in SQL queries, and allows a primary key to
|
|
be case-insensitive. <type>citext</type> is locale-aware, just
|
|
like <type>text</type>, which means that the matching of upper case and
|
|
lower case characters is dependent on the rules of
|
|
the database's <literal>LC_CTYPE</literal> setting. Again, this behavior is
|
|
identical to the use of <function>lower</function> in queries. But because it's
|
|
done transparently by the data type, you don't have to remember to do
|
|
anything special in your queries.
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>How to Use It</title>
|
|
|
|
<para>
|
|
Here's a simple example of usage:
|
|
|
|
<programlisting>
|
|
CREATE TABLE users (
|
|
nick CITEXT PRIMARY KEY,
|
|
pass TEXT NOT NULL
|
|
);
|
|
|
|
INSERT INTO users VALUES ( 'larry', sha256(random()::text::bytea) );
|
|
INSERT INTO users VALUES ( 'Tom', sha256(random()::text::bytea) );
|
|
INSERT INTO users VALUES ( 'Damian', sha256(random()::text::bytea) );
|
|
INSERT INTO users VALUES ( 'NEAL', sha256(random()::text::bytea) );
|
|
INSERT INTO users VALUES ( 'Bjørn', sha256(random()::text::bytea) );
|
|
|
|
SELECT * FROM users WHERE nick = 'Larry';
|
|
</programlisting>
|
|
|
|
The <command>SELECT</command> statement will return one tuple, even though
|
|
the <structfield>nick</structfield> column was set to <literal>larry</literal> and the query
|
|
was for <literal>Larry</literal>.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>String Comparison Behavior</title>
|
|
|
|
<para>
|
|
<type>citext</type> performs comparisons by converting each string to lower
|
|
case (as though <function>lower</function> were called) and then comparing the
|
|
results normally. Thus, for example, two strings are considered equal
|
|
if <function>lower</function> would produce identical results for them.
|
|
</para>
|
|
|
|
<para>
|
|
In order to emulate a case-insensitive collation as closely as possible,
|
|
there are <type>citext</type>-specific versions of a number of string-processing
|
|
operators and functions. So, for example, the regular expression
|
|
operators <literal>~</literal> and <literal>~*</literal> exhibit the same behavior when
|
|
applied to <type>citext</type>: they both match case-insensitively.
|
|
The same is true
|
|
for <literal>!~</literal> and <literal>!~*</literal>, as well as for the
|
|
<literal>LIKE</literal> operators <literal>~~</literal> and <literal>~~*</literal>, and
|
|
<literal>!~~</literal> and <literal>!~~*</literal>. If you'd like to match
|
|
case-sensitively, you can cast the operator's arguments to <type>text</type>.
|
|
</para>
|
|
|
|
<para>
|
|
Similarly, all of the following functions perform matching
|
|
case-insensitively if their arguments are <type>citext</type>:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<function>regexp_match()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>regexp_matches()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>regexp_replace()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>regexp_split_to_array()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>regexp_split_to_table()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>replace()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>split_part()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>strpos()</function>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<function>translate()</function>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>
|
|
For the regexp functions, if you want to match case-sensitively, you can
|
|
specify the <quote>c</quote> flag to force a case-sensitive match. Otherwise,
|
|
you must cast to <type>text</type> before using one of these functions if
|
|
you want case-sensitive behavior.
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Limitations</title>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<type>citext</type>'s case-folding behavior depends on
|
|
the <literal>LC_CTYPE</literal> setting of your database. How it compares
|
|
values is therefore determined when the database is created.
|
|
It is not truly
|
|
case-insensitive in the terms defined by the Unicode standard.
|
|
Effectively, what this means is that, as long as you're happy with your
|
|
collation, you should be happy with <type>citext</type>'s comparisons. But
|
|
if you have data in different languages stored in your database, users
|
|
of one language may find their query results are not as expected if the
|
|
collation is for another language.
|
|
</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>
|
|
As of <productname>PostgreSQL</productname> 9.1, you can attach a
|
|
<literal>COLLATE</literal> specification to <type>citext</type> columns or data
|
|
values. Currently, <type>citext</type> operators will honor a non-default
|
|
<literal>COLLATE</literal> specification while comparing case-folded strings,
|
|
but the initial folding to lower case is always done according to the
|
|
database's <literal>LC_CTYPE</literal> setting (that is, as though
|
|
<literal>COLLATE "default"</literal> were given). This may be changed in a
|
|
future release so that both steps follow the input <literal>COLLATE</literal>
|
|
specification.
|
|
</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>
|
|
<type>citext</type> is not as efficient as <type>text</type> because the
|
|
operator functions and the B-tree comparison functions must make copies
|
|
of the data and convert it to lower case for comparisons. Also, only
|
|
<type>text</type> can support B-Tree deduplication. However,
|
|
<type>citext</type> is slightly more efficient than using
|
|
<function>lower</function> to get case-insensitive matching.
|
|
</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>
|
|
<type>citext</type> doesn't help much if you need data to compare
|
|
case-sensitively in some contexts and case-insensitively in other
|
|
contexts. The standard answer is to use the <type>text</type> type and
|
|
manually use the <function>lower</function> function when you need to compare
|
|
case-insensitively; this works all right if case-insensitive comparison
|
|
is needed only infrequently. If you need case-insensitive behavior most
|
|
of the time and case-sensitive infrequently, consider storing the data
|
|
as <type>citext</type> and explicitly casting the column to <type>text</type>
|
|
when you want case-sensitive comparison. In either situation, you will
|
|
need two indexes if you want both types of searches to be fast.
|
|
</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>
|
|
The schema containing the <type>citext</type> operators must be
|
|
in the current <varname>search_path</varname> (typically <literal>public</literal>);
|
|
if it is not, the normal case-sensitive <type>text</type> operators
|
|
will be invoked instead.
|
|
</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>
|
|
The approach of lower-casing strings for comparison does not handle some
|
|
Unicode special cases correctly, for example when one upper-case letter
|
|
has two lower-case letter equivalents. Unicode distinguishes between
|
|
<firstterm>case mapping</firstterm> and <firstterm>case
|
|
folding</firstterm> for this reason. Use nondeterministic collations
|
|
instead of <type>citext</type> to handle that correctly.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Author</title>
|
|
|
|
<para>
|
|
David E. Wheeler <email>david@kineticode.com</email>
|
|
</para>
|
|
|
|
<para>
|
|
Inspired by the original <type>citext</type> module by Donald Fraser.
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|