mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-09-10 05:59:23 +02:00
893 lines
32 KiB
Plaintext
893 lines
32 KiB
Plaintext
<!-- doc/src/sgml/storage.sgml -->
|
|
|
|
<chapter id="storage">
|
|
|
|
<title>Database Physical Storage</title>
|
|
|
|
<para>
|
|
This chapter provides an overview of the physical storage format used by
|
|
<productname>PostgreSQL</productname> databases.
|
|
</para>
|
|
|
|
<sect1 id="storage-file-layout">
|
|
|
|
<title>Database File Layout</title>
|
|
|
|
<para>
|
|
This section describes the storage format at the level of files and
|
|
directories.
|
|
</para>
|
|
|
|
<para>
|
|
Traditionally, the configuration and data files used by a database
|
|
cluster are stored together within the cluster's data
|
|
directory, commonly referred to as <varname>PGDATA</> (after the name of the
|
|
environment variable that can be used to define it). A common location for
|
|
<varname>PGDATA</> is <filename>/var/lib/pgsql/data</>. Multiple clusters,
|
|
managed by different server instances, can exist on the same machine.
|
|
</para>
|
|
|
|
<para>
|
|
The <varname>PGDATA</> directory contains several subdirectories and control
|
|
files, as shown in <xref linkend="pgdata-contents-table">. In addition to
|
|
these required items, the cluster configuration files
|
|
<filename>postgresql.conf</filename>, <filename>pg_hba.conf</filename>, and
|
|
<filename>pg_ident.conf</filename> are traditionally stored in
|
|
<varname>PGDATA</> (although in <productname>PostgreSQL</productname> 8.0 and
|
|
later, it is possible to place them elsewhere).
|
|
</para>
|
|
|
|
<table tocentry="1" id="pgdata-contents-table">
|
|
<title>Contents of <varname>PGDATA</></title>
|
|
<tgroup cols="2">
|
|
<thead>
|
|
<row>
|
|
<entry>
|
|
Item
|
|
</entry>
|
|
<entry>Description</entry>
|
|
</row>
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<row>
|
|
<entry><filename>PG_VERSION</></entry>
|
|
<entry>A file containing the major version number of <productname>PostgreSQL</productname></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>base</></entry>
|
|
<entry>Subdirectory containing per-database subdirectories</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>global</></entry>
|
|
<entry>Subdirectory containing cluster-wide tables, such as
|
|
<structname>pg_database</></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_clog</></entry>
|
|
<entry>Subdirectory containing transaction commit status data</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_dynshmem</></entry>
|
|
<entry>Subdirectory containing files used by the dynamic shared memory
|
|
subsystem</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_multixact</></entry>
|
|
<entry>Subdirectory containing multitransaction status data
|
|
(used for shared row locks)</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_notify</></entry>
|
|
<entry>Subdirectory containing LISTEN/NOTIFY status data</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_serial</></entry>
|
|
<entry>Subdirectory containing information about committed serializable transactions</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_snapshots</></entry>
|
|
<entry>Subdirectory containing exported snapshots</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_stat_tmp</></entry>
|
|
<entry>Subdirectory containing temporary files for the statistics
|
|
subsystem</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_subtrans</></entry>
|
|
<entry>Subdirectory containing subtransaction status data</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_tblspc</></entry>
|
|
<entry>Subdirectory containing symbolic links to tablespaces</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_twophase</></entry>
|
|
<entry>Subdirectory containing state files for prepared transactions</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>pg_xlog</></entry>
|
|
<entry>Subdirectory containing WAL (Write Ahead Log) files</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>postmaster.opts</></entry>
|
|
<entry>A file recording the command-line options the server was
|
|
last started with</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><filename>postmaster.pid</></entry>
|
|
<entry>A lock file recording the current postmaster process ID (PID),
|
|
cluster data directory path,
|
|
postmaster start timestamp,
|
|
port number,
|
|
Unix-domain socket directory path (empty on Windows),
|
|
first valid listen_address (IP address or <literal>*</>, or empty if
|
|
not listening on TCP),
|
|
and shared memory segment ID
|
|
(this file is not present after server shutdown)</entry>
|
|
</row>
|
|
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
For each database in the cluster there is a subdirectory within
|
|
<varname>PGDATA</><filename>/base</>, named after the database's OID in
|
|
<structname>pg_database</>. This subdirectory is the default location
|
|
for the database's files; in particular, its system catalogs are stored
|
|
there.
|
|
</para>
|
|
|
|
<para>
|
|
Each table and index is stored in a separate file. For ordinary relations,
|
|
these files are named after the table or index's <firstterm>filenode</> number,
|
|
which can be found in <structname>pg_class</>.<structfield>relfilenode</>. But
|
|
for temporary relations, the file name is of the form
|
|
<literal>t<replaceable>BBB</>_<replaceable>FFF</></>, where <replaceable>BBB</>
|
|
is the backend ID of the backend which created the file, and <replaceable>FFF</>
|
|
is the filenode number. In either case, in addition to the main file (a/k/a
|
|
main fork), each table and index has a <firstterm>free space map</> (see <xref
|
|
linkend="storage-fsm">), which stores information about free space available in
|
|
the relation. The free space map is stored in a file named with the filenode
|
|
number plus the suffix <literal>_fsm</>. Tables also have a
|
|
<firstterm>visibility map</>, stored in a fork with the suffix <literal>_vm</>,
|
|
to track which pages are known to have no dead tuples. The visibility map is
|
|
described further in <xref linkend="storage-vm">. Unlogged tables and indexes
|
|
have a third fork, known as the initialization fork, which is stored in a fork
|
|
with the suffix <literal>_init</literal> (see <xref linkend="storage-init">).
|
|
</para>
|
|
|
|
<caution>
|
|
<para>
|
|
Note that while a table's filenode often matches its OID, this is
|
|
<emphasis>not</> necessarily the case; some operations, like
|
|
<command>TRUNCATE</>, <command>REINDEX</>, <command>CLUSTER</> and some forms
|
|
of <command>ALTER TABLE</>, can change the filenode while preserving the OID.
|
|
Avoid assuming that filenode and table OID are the same.
|
|
Also, for certain system catalogs including <structname>pg_class</> itself,
|
|
<structname>pg_class</>.<structfield>relfilenode</> contains zero. The
|
|
actual filenode number of these catalogs is stored in a lower-level data
|
|
structure, and can be obtained using the <function>pg_relation_filenode()</>
|
|
function.
|
|
</para>
|
|
</caution>
|
|
|
|
<para>
|
|
When a table or index exceeds 1 GB, it is divided into gigabyte-sized
|
|
<firstterm>segments</>. The first segment's file name is the same as the
|
|
filenode; subsequent segments are named filenode.1, filenode.2, etc.
|
|
This arrangement avoids problems on platforms that have file size limitations.
|
|
(Actually, 1 GB is just the default segment size. The segment size can be
|
|
adjusted using the configuration option <option>--with-segsize</option>
|
|
when building <productname>PostgreSQL</>.)
|
|
In principle, free space map and visibility map forks could require multiple
|
|
segments as well, though this is unlikely to happen in practice.
|
|
</para>
|
|
|
|
<para>
|
|
A table that has columns with potentially large entries will have an
|
|
associated <firstterm>TOAST</> table, which is used for out-of-line storage of
|
|
field values that are too large to keep in the table rows proper.
|
|
<structname>pg_class</>.<structfield>reltoastrelid</> links from a table to
|
|
its <acronym>TOAST</> table, if any.
|
|
See <xref linkend="storage-toast"> for more information.
|
|
</para>
|
|
|
|
<para>
|
|
The contents of tables and indexes are discussed further in
|
|
<xref linkend="storage-page-layout">.
|
|
</para>
|
|
|
|
<para>
|
|
Tablespaces make the scenario more complicated. Each user-defined tablespace
|
|
has a symbolic link inside the <varname>PGDATA</><filename>/pg_tblspc</>
|
|
directory, which points to the physical tablespace directory (i.e., the
|
|
location specified in the tablespace's <command>CREATE TABLESPACE</> command).
|
|
This symbolic link is named after
|
|
the tablespace's OID. Inside the physical tablespace directory there is
|
|
a subdirectory with a name that depends on the <productname>PostgreSQL</>
|
|
server version, such as <literal>PG_9.0_201008051</>. (The reason for using
|
|
this subdirectory is so that successive versions of the database can use
|
|
the same <command>CREATE TABLESPACE</> location value without conflicts.)
|
|
Within the version-specific subdirectory, there is
|
|
a subdirectory for each database that has elements in the tablespace, named
|
|
after the database's OID. Tables and indexes are stored within that
|
|
directory, using the filenode naming scheme.
|
|
The <literal>pg_default</> tablespace is not accessed through
|
|
<filename>pg_tblspc</>, but corresponds to
|
|
<varname>PGDATA</><filename>/base</>. Similarly, the <literal>pg_global</>
|
|
tablespace is not accessed through <filename>pg_tblspc</>, but corresponds to
|
|
<varname>PGDATA</><filename>/global</>.
|
|
</para>
|
|
|
|
<para>
|
|
The <function>pg_relation_filepath()</> function shows the entire path
|
|
(relative to <varname>PGDATA</>) of any relation. It is often useful
|
|
as a substitute for remembering many of the above rules. But keep in
|
|
mind that this function just gives the name of the first segment of the
|
|
main fork of the relation — you may need to append a segment number
|
|
and/or <literal>_fsm</> or <literal>_vm</> to find all the files associated
|
|
with the relation.
|
|
</para>
|
|
|
|
<para>
|
|
Temporary files (for operations such as sorting more data than can fit in
|
|
memory) are created within <varname>PGDATA</><filename>/base/pgsql_tmp</>,
|
|
or within a <filename>pgsql_tmp</> subdirectory of a tablespace directory
|
|
if a tablespace other than <literal>pg_default</> is specified for them.
|
|
The name of a temporary file has the form
|
|
<filename>pgsql_tmp<replaceable>PPP</>.<replaceable>NNN</></filename>,
|
|
where <replaceable>PPP</> is the PID of the owning backend and
|
|
<replaceable>NNN</> distinguishes different temporary files of that backend.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="storage-toast">
|
|
|
|
<title>TOAST</title>
|
|
|
|
<indexterm>
|
|
<primary>TOAST</primary>
|
|
</indexterm>
|
|
<indexterm><primary>sliced bread</><see>TOAST</></indexterm>
|
|
|
|
<para>
|
|
This section provides an overview of <acronym>TOAST</> (The
|
|
Oversized-Attribute Storage Technique).
|
|
</para>
|
|
|
|
<para>
|
|
<productname>PostgreSQL</productname> uses a fixed page size (commonly
|
|
8 kB), and does not allow tuples to span multiple pages. Therefore, it is
|
|
not possible to store very large field values directly. To overcome
|
|
this limitation, large field values are compressed and/or broken up into
|
|
multiple physical rows. This happens transparently to the user, with only
|
|
small impact on most of the backend code. The technique is affectionately
|
|
known as <acronym>TOAST</> (or <quote>the best thing since sliced bread</>).
|
|
</para>
|
|
|
|
<para>
|
|
Only certain data types support <acronym>TOAST</> — there is no need to
|
|
impose the overhead on data types that cannot produce large field values.
|
|
To support <acronym>TOAST</>, a data type must have a variable-length
|
|
(<firstterm>varlena</>) representation, in which the first 32-bit word of any
|
|
stored value contains the total length of the value in bytes (including
|
|
itself). <acronym>TOAST</> does not constrain the rest of the representation.
|
|
All the C-level functions supporting a <acronym>TOAST</>-able data type must
|
|
be careful to handle <acronym>TOAST</>ed input values. (This is normally done
|
|
by invoking <function>PG_DETOAST_DATUM</> before doing anything with an input
|
|
value, but in some cases more efficient approaches are possible.)
|
|
</para>
|
|
|
|
<para>
|
|
<acronym>TOAST</> usurps two bits of the varlena length word (the high-order
|
|
bits on big-endian machines, the low-order bits on little-endian machines),
|
|
thereby limiting the logical size of any value of a <acronym>TOAST</>-able
|
|
data type to 1 GB (2<superscript>30</> - 1 bytes). When both bits are zero,
|
|
the value is an ordinary un-<acronym>TOAST</>ed value of the data type, and
|
|
the remaining bits of the length word give the total datum size (including
|
|
length word) in bytes. When the highest-order or lowest-order bit is set,
|
|
the value has only a single-byte header instead of the normal four-byte
|
|
header, and the remaining bits give the total datum size (including length
|
|
byte) in bytes. As a special case, if the remaining bits are all zero
|
|
(which would be impossible for a self-inclusive length), the value is a
|
|
pointer to out-of-line data stored in a separate TOAST table. (The size of
|
|
a TOAST pointer is given in the second byte of the datum.)
|
|
Values with single-byte headers aren't aligned on any particular
|
|
boundary, either. Lastly, when the highest-order or lowest-order bit is
|
|
clear but the adjacent bit is set, the content of the datum has been
|
|
compressed and must be decompressed before use. In this case the remaining
|
|
bits of the length word give the total size of the compressed datum, not the
|
|
original data. Note that compression is also possible for out-of-line data
|
|
but the varlena header does not tell whether it has occurred —
|
|
the content of the TOAST pointer tells that, instead.
|
|
</para>
|
|
|
|
<para>
|
|
If any of the columns of a table are <acronym>TOAST</>-able, the table will
|
|
have an associated <acronym>TOAST</> table, whose OID is stored in the table's
|
|
<structname>pg_class</>.<structfield>reltoastrelid</> entry. Out-of-line
|
|
<acronym>TOAST</>ed values are kept in the <acronym>TOAST</> table, as
|
|
described in more detail below.
|
|
</para>
|
|
|
|
<para>
|
|
The compression technique used is a fairly simple and very fast member
|
|
of the LZ family of compression techniques. See
|
|
<filename>src/backend/utils/adt/pg_lzcompress.c</> for the details.
|
|
</para>
|
|
|
|
<para>
|
|
Out-of-line values are divided (after compression if used) into chunks of at
|
|
most <symbol>TOAST_MAX_CHUNK_SIZE</> bytes (by default this value is chosen
|
|
so that four chunk rows will fit on a page, making it about 2000 bytes).
|
|
Each chunk is stored
|
|
as a separate row in the <acronym>TOAST</> table for the owning table. Every
|
|
<acronym>TOAST</> table has the columns <structfield>chunk_id</> (an OID
|
|
identifying the particular <acronym>TOAST</>ed value),
|
|
<structfield>chunk_seq</> (a sequence number for the chunk within its value),
|
|
and <structfield>chunk_data</> (the actual data of the chunk). A unique index
|
|
on <structfield>chunk_id</> and <structfield>chunk_seq</> provides fast
|
|
retrieval of the values. A pointer datum representing an out-of-line
|
|
<acronym>TOAST</>ed value therefore needs to store the OID of the
|
|
<acronym>TOAST</> table in which to look and the OID of the specific value
|
|
(its <structfield>chunk_id</>). For convenience, pointer datums also store the
|
|
logical datum size (original uncompressed data length) and actual stored size
|
|
(different if compression was applied). Allowing for the varlena header bytes,
|
|
the total size of a <acronym>TOAST</> pointer datum is therefore 18 bytes
|
|
regardless of the actual size of the represented value.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>TOAST</> code is triggered only
|
|
when a row value to be stored in a table is wider than
|
|
<symbol>TOAST_TUPLE_THRESHOLD</> bytes (normally 2 kB).
|
|
The <acronym>TOAST</> code will compress and/or move
|
|
field values out-of-line until the row value is shorter than
|
|
<symbol>TOAST_TUPLE_TARGET</> bytes (also normally 2 kB)
|
|
or no more gains can be had. During an UPDATE
|
|
operation, values of unchanged fields are normally preserved as-is; so an
|
|
UPDATE of a row with out-of-line values incurs no <acronym>TOAST</> costs if
|
|
none of the out-of-line values change.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>TOAST</> code recognizes four different strategies for storing
|
|
<acronym>TOAST</>-able columns:
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<literal>PLAIN</literal> prevents either compression or
|
|
out-of-line storage; furthermore it disables use of single-byte headers
|
|
for varlena types.
|
|
This is the only possible strategy for
|
|
columns of non-<acronym>TOAST</>-able data types.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<literal>EXTENDED</literal> allows both compression and out-of-line
|
|
storage. This is the default for most <acronym>TOAST</>-able data types.
|
|
Compression will be attempted first, then out-of-line storage if
|
|
the row is still too big.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<literal>EXTERNAL</literal> allows out-of-line storage but not
|
|
compression. Use of <literal>EXTERNAL</literal> will
|
|
make substring operations on wide <type>text</type> and
|
|
<type>bytea</type> columns faster (at the penalty of increased storage
|
|
space) because these operations are optimized to fetch only the
|
|
required parts of the out-of-line value when it is not compressed.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<literal>MAIN</literal> allows compression but not out-of-line
|
|
storage. (Actually, out-of-line storage will still be performed
|
|
for such columns, but only as a last resort when there is no other
|
|
way to make the row small enough to fit on a page.)
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
Each <acronym>TOAST</>-able data type specifies a default strategy for columns
|
|
of that data type, but the strategy for a given table column can be altered
|
|
with <command>ALTER TABLE SET STORAGE</>.
|
|
</para>
|
|
|
|
<para>
|
|
This scheme has a number of advantages compared to a more straightforward
|
|
approach such as allowing row values to span pages. Assuming that queries are
|
|
usually qualified by comparisons against relatively small key values, most of
|
|
the work of the executor will be done using the main row entry. The big values
|
|
of <acronym>TOAST</>ed attributes will only be pulled out (if selected at all)
|
|
at the time the result set is sent to the client. Thus, the main table is much
|
|
smaller and more of its rows fit in the shared buffer cache than would be the
|
|
case without any out-of-line storage. Sort sets shrink also, and sorts will
|
|
more often be done entirely in memory. A little test showed that a table
|
|
containing typical HTML pages and their URLs was stored in about half of the
|
|
raw data size including the <acronym>TOAST</> table, and that the main table
|
|
contained only about 10% of the entire data (the URLs and some small HTML
|
|
pages). There was no run time difference compared to an un-<acronym>TOAST</>ed
|
|
comparison table, in which all the HTML pages were cut down to 7 kB to fit.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="storage-fsm">
|
|
|
|
<title>Free Space Map</title>
|
|
|
|
<indexterm>
|
|
<primary>Free Space Map</primary>
|
|
</indexterm>
|
|
<indexterm><primary>FSM</><see>Free Space Map</></indexterm>
|
|
|
|
<para>
|
|
Each heap and index relation, except for hash indexes, has a Free Space Map
|
|
(FSM) to keep track of available space in the relation. It's stored
|
|
alongside the main relation data in a separate relation fork, named after the
|
|
filenode number of the relation, plus a <literal>_fsm</> suffix. For example,
|
|
if the filenode of a relation is 12345, the FSM is stored in a file called
|
|
<filename>12345_fsm</>, in the same directory as the main relation file.
|
|
</para>
|
|
|
|
<para>
|
|
The Free Space Map is organized as a tree of <acronym>FSM</> pages. The
|
|
bottom level <acronym>FSM</> pages store the free space available on each
|
|
heap (or index) page, using one byte to represent each such page. The upper
|
|
levels aggregate information from the lower levels.
|
|
</para>
|
|
|
|
<para>
|
|
Within each <acronym>FSM</> page is a binary tree, stored in an array with
|
|
one byte per node. Each leaf node represents a heap page, or a lower level
|
|
<acronym>FSM</> page. In each non-leaf node, the higher of its children's
|
|
values is stored. The maximum value in the leaf nodes is therefore stored
|
|
at the root.
|
|
</para>
|
|
|
|
<para>
|
|
See <filename>src/backend/storage/freespace/README</> for more details on
|
|
how the <acronym>FSM</> is structured, and how it's updated and searched.
|
|
The <xref linkend="pgfreespacemap"> module
|
|
can be used to examine the information stored in free space maps.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="storage-vm">
|
|
|
|
<title>Visibility Map</title>
|
|
|
|
<indexterm>
|
|
<primary>Visibility Map</primary>
|
|
</indexterm>
|
|
<indexterm><primary>VM</><see>Visibility Map</></indexterm>
|
|
|
|
<para>
|
|
Each heap relation has a Visibility Map
|
|
(VM) to keep track of which pages contain only tuples that are known to be
|
|
visible to all active transactions. It's stored
|
|
alongside the main relation data in a separate relation fork, named after the
|
|
filenode number of the relation, plus a <literal>_vm</> suffix. For example,
|
|
if the filenode of a relation is 12345, the VM is stored in a file called
|
|
<filename>12345_vm</>, in the same directory as the main relation file.
|
|
Note that indexes do not have VMs.
|
|
</para>
|
|
|
|
<para>
|
|
The visibility map simply stores one bit per heap page. A set bit means
|
|
that all tuples on the page are known to be visible to all transactions.
|
|
This means that the page does not contain any tuples that need to be vacuumed.
|
|
This information can also be used by <firstterm>index-only scans</> to answer
|
|
queries using only the index tuple.
|
|
</para>
|
|
|
|
<para>
|
|
The map is conservative in the sense that we make sure that whenever a bit is
|
|
set, we know the condition is true, but if a bit is not set, it might or
|
|
might not be true. Visibility map bits are only set by vacuum, but are
|
|
cleared by any data-modifying operations on a page.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="storage-init">
|
|
|
|
<title>The Initialization Fork</title>
|
|
|
|
<indexterm>
|
|
<primary>Initialization Fork</primary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
Each unlogged table, and each index on an unlogged table, has an initialization
|
|
fork. The initialization fork is an empty table or index of the appropriate
|
|
type. When an unlogged table must be reset to empty due to a crash, the
|
|
initialization fork is copied over the main fork, and any other forks are
|
|
erased (they will be recreated automatically as needed).
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="storage-page-layout">
|
|
|
|
<title>Database Page Layout</title>
|
|
|
|
<para>
|
|
This section provides an overview of the page format used within
|
|
<productname>PostgreSQL</productname> tables and indexes.<footnote>
|
|
<para>
|
|
Actually, index access methods need not use this page format.
|
|
All the existing index methods do use this basic format,
|
|
but the data kept on index metapages usually doesn't follow
|
|
the item layout rules.
|
|
</para>
|
|
</footnote>
|
|
Sequences and <acronym>TOAST</> tables are formatted just like a regular table.
|
|
</para>
|
|
|
|
<para>
|
|
In the following explanation, a
|
|
<firstterm>byte</firstterm>
|
|
is assumed to contain 8 bits. In addition, the term
|
|
<firstterm>item</firstterm>
|
|
refers to an individual data value that is stored on a page. In a table,
|
|
an item is a row; in an index, an item is an index entry.
|
|
</para>
|
|
|
|
<para>
|
|
Every table and index is stored as an array of <firstterm>pages</> of a
|
|
fixed size (usually 8 kB, although a different page size can be selected
|
|
when compiling the server). In a table, all the pages are logically
|
|
equivalent, so a particular item (row) can be stored in any page. In
|
|
indexes, the first page is generally reserved as a <firstterm>metapage</>
|
|
holding control information, and there can be different types of pages
|
|
within the index, depending on the index access method.
|
|
</para>
|
|
|
|
<para>
|
|
<xref linkend="page-table"> shows the overall layout of a page.
|
|
There are five parts to each page.
|
|
</para>
|
|
|
|
<table tocentry="1" id="page-table">
|
|
<title>Overall Page Layout</title>
|
|
<titleabbrev>Page Layout</titleabbrev>
|
|
<tgroup cols="2">
|
|
<thead>
|
|
<row>
|
|
<entry>
|
|
Item
|
|
</entry>
|
|
<entry>Description</entry>
|
|
</row>
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<row>
|
|
<entry>PageHeaderData</entry>
|
|
<entry>24 bytes long. Contains general information about the page, including
|
|
free space pointers.</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry>ItemIdData</entry>
|
|
<entry>Array of (offset,length) pairs pointing to the actual items.
|
|
4 bytes per item.</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry>Free space</entry>
|
|
<entry>The unallocated space. New item pointers are allocated from the start
|
|
of this area, new items from the end.</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry>Items</entry>
|
|
<entry>The actual items themselves.</entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry>Special space</entry>
|
|
<entry>Index access method specific data. Different methods store different
|
|
data. Empty in ordinary tables.</entry>
|
|
</row>
|
|
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
|
|
The first 24 bytes of each page consists of a page header
|
|
(PageHeaderData). Its format is detailed in <xref
|
|
linkend="pageheaderdata-table">. The first two fields track the most
|
|
recent WAL entry related to this page. Next is a 2-byte field
|
|
containing flag bits. This is followed by three 2-byte integer fields
|
|
(<structfield>pd_lower</structfield>, <structfield>pd_upper</structfield>,
|
|
and <structfield>pd_special</structfield>). These contain byte offsets
|
|
from the page start to the start
|
|
of unallocated space, to the end of unallocated space, and to the start of
|
|
the special space.
|
|
The next 2 bytes of the page header,
|
|
<structfield>pd_pagesize_version</structfield>, store both the page size
|
|
and a version indicator. Beginning with
|
|
<productname>PostgreSQL</productname> 8.3 the version number is 4;
|
|
<productname>PostgreSQL</productname> 8.1 and 8.2 used version number 3;
|
|
<productname>PostgreSQL</productname> 8.0 used version number 2;
|
|
<productname>PostgreSQL</productname> 7.3 and 7.4 used version number 1;
|
|
prior releases used version number 0.
|
|
(The basic page layout and header format has not changed in most of these
|
|
versions, but the layout of heap row headers has.) The page size
|
|
is basically only present as a cross-check; there is no support for having
|
|
more than one page size in an installation.
|
|
The last field is a hint that shows whether pruning the page is likely
|
|
to be profitable: it tracks the oldest un-pruned XMAX on the page.
|
|
|
|
</para>
|
|
|
|
<table tocentry="1" id="pageheaderdata-table">
|
|
<title>PageHeaderData Layout</title>
|
|
<titleabbrev>PageHeaderData Layout</titleabbrev>
|
|
<tgroup cols="4">
|
|
<thead>
|
|
<row>
|
|
<entry>Field</entry>
|
|
<entry>Type</entry>
|
|
<entry>Length</entry>
|
|
<entry>Description</entry>
|
|
</row>
|
|
</thead>
|
|
<tbody>
|
|
<row>
|
|
<entry>pd_lsn</entry>
|
|
<entry>XLogRecPtr</entry>
|
|
<entry>8 bytes</entry>
|
|
<entry>LSN: next byte after last byte of xlog record for last change
|
|
to this page</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_checksum</entry>
|
|
<entry>uint16</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Page checksum</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_flags</entry>
|
|
<entry>uint16</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Flag bits</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_lower</entry>
|
|
<entry>LocationIndex</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Offset to start of free space</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_upper</entry>
|
|
<entry>LocationIndex</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Offset to end of free space</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_special</entry>
|
|
<entry>LocationIndex</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Offset to start of special space</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_pagesize_version</entry>
|
|
<entry>uint16</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>Page size and layout version number information</entry>
|
|
</row>
|
|
<row>
|
|
<entry>pd_prune_xid</entry>
|
|
<entry>TransactionId</entry>
|
|
<entry>4 bytes</entry>
|
|
<entry>Oldest unpruned XMAX on page, or zero if none</entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
All the details can be found in
|
|
<filename>src/include/storage/bufpage.h</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Following the page header are item identifiers
|
|
(<type>ItemIdData</type>), each requiring four bytes.
|
|
An item identifier contains a byte-offset to
|
|
the start of an item, its length in bytes, and a few attribute bits
|
|
which affect its interpretation.
|
|
New item identifiers are allocated
|
|
as needed from the beginning of the unallocated space.
|
|
The number of item identifiers present can be determined by looking at
|
|
<structfield>pd_lower</>, which is increased to allocate a new identifier.
|
|
Because an item
|
|
identifier is never moved until it is freed, its index can be used on a
|
|
long-term basis to reference an item, even when the item itself is moved
|
|
around on the page to compact free space. In fact, every pointer to an
|
|
item (<type>ItemPointer</type>, also known as
|
|
<type>CTID</type>) created by
|
|
<productname>PostgreSQL</productname> consists of a page number and the
|
|
index of an item identifier.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
The items themselves are stored in space allocated backwards from the end
|
|
of unallocated space. The exact structure varies depending on what the
|
|
table is to contain. Tables and sequences both use a structure named
|
|
<type>HeapTupleHeaderData</type>, described below.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
The final section is the <quote>special section</quote> which can
|
|
contain anything the access method wishes to store. For example,
|
|
b-tree indexes store links to the page's left and right siblings,
|
|
as well as some other data relevant to the index structure.
|
|
Ordinary tables do not use a special section at all (indicated by setting
|
|
<structfield>pd_special</> to equal the page size).
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
All table rows are structured in the same way. There is a fixed-size
|
|
header (occupying 23 bytes on most machines), followed by an optional null
|
|
bitmap, an optional object ID field, and the user data. The header is
|
|
detailed
|
|
in <xref linkend="heaptupleheaderdata-table">. The actual user data
|
|
(columns of the row) begins at the offset indicated by
|
|
<structfield>t_hoff</>, which must always be a multiple of the MAXALIGN
|
|
distance for the platform.
|
|
The null bitmap is
|
|
only present if the <firstterm>HEAP_HASNULL</firstterm> bit is set in
|
|
<structfield>t_infomask</structfield>. If it is present it begins just after
|
|
the fixed header and occupies enough bytes to have one bit per data column
|
|
(that is, <structfield>t_natts</> bits altogether). In this list of bits, a
|
|
1 bit indicates not-null, a 0 bit is a null. When the bitmap is not
|
|
present, all columns are assumed not-null.
|
|
The object ID is only present if the <firstterm>HEAP_HASOID</firstterm> bit
|
|
is set in <structfield>t_infomask</structfield>. If present, it appears just
|
|
before the <structfield>t_hoff</> boundary. Any padding needed to make
|
|
<structfield>t_hoff</> a MAXALIGN multiple will appear between the null
|
|
bitmap and the object ID. (This in turn ensures that the object ID is
|
|
suitably aligned.)
|
|
|
|
</para>
|
|
|
|
<table tocentry="1" id="heaptupleheaderdata-table">
|
|
<title>HeapTupleHeaderData Layout</title>
|
|
<titleabbrev>HeapTupleHeaderData Layout</titleabbrev>
|
|
<tgroup cols="4">
|
|
<thead>
|
|
<row>
|
|
<entry>Field</entry>
|
|
<entry>Type</entry>
|
|
<entry>Length</entry>
|
|
<entry>Description</entry>
|
|
</row>
|
|
</thead>
|
|
<tbody>
|
|
<row>
|
|
<entry>t_xmin</entry>
|
|
<entry>TransactionId</entry>
|
|
<entry>4 bytes</entry>
|
|
<entry>insert XID stamp</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_xmax</entry>
|
|
<entry>TransactionId</entry>
|
|
<entry>4 bytes</entry>
|
|
<entry>delete XID stamp</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_cid</entry>
|
|
<entry>CommandId</entry>
|
|
<entry>4 bytes</entry>
|
|
<entry>insert and/or delete CID stamp (overlays with t_xvac)</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_xvac</entry>
|
|
<entry>TransactionId</entry>
|
|
<entry>4 bytes</entry>
|
|
<entry>XID for VACUUM operation moving a row version</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_ctid</entry>
|
|
<entry>ItemPointerData</entry>
|
|
<entry>6 bytes</entry>
|
|
<entry>current TID of this or newer row version</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_infomask2</entry>
|
|
<entry>uint16</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>number of attributes, plus various flag bits</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_infomask</entry>
|
|
<entry>uint16</entry>
|
|
<entry>2 bytes</entry>
|
|
<entry>various flag bits</entry>
|
|
</row>
|
|
<row>
|
|
<entry>t_hoff</entry>
|
|
<entry>uint8</entry>
|
|
<entry>1 byte</entry>
|
|
<entry>offset to user data</entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
All the details can be found in
|
|
<filename>src/include/access/htup.h</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Interpreting the actual data can only be done with information obtained
|
|
from other tables, mostly <structname>pg_attribute</structname>. The
|
|
key values needed to identify field locations are
|
|
<structfield>attlen</structfield> and <structfield>attalign</structfield>.
|
|
There is no way to directly get a
|
|
particular attribute, except when there are only fixed width fields and no
|
|
null values. All this trickery is wrapped up in the functions
|
|
<firstterm>heap_getattr</firstterm>, <firstterm>fastgetattr</firstterm>
|
|
and <firstterm>heap_getsysattr</firstterm>.
|
|
|
|
</para>
|
|
<para>
|
|
|
|
To read the data you need to examine each attribute in turn. First check
|
|
whether the field is NULL according to the null bitmap. If it is, go to
|
|
the next. Then make sure you have the right alignment. If the field is a
|
|
fixed width field, then all the bytes are simply placed. If it's a
|
|
variable length field (attlen = -1) then it's a bit more complicated.
|
|
All variable-length data types share the common header structure
|
|
<type>struct varlena</type>, which includes the total length of the stored
|
|
value and some flag bits. Depending on the flags, the data can be either
|
|
inline or in a <acronym>TOAST</> table;
|
|
it might be compressed, too (see <xref linkend="storage-toast">).
|
|
|
|
</para>
|
|
</sect1>
|
|
|
|
</chapter>
|