1996-07-09 08:22:35 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
1999-02-14 00:22:53 +01:00
|
|
|
* relcache.c
|
1996-07-09 08:22:35 +02:00
|
|
|
* POSTGRES relation descriptor cache code
|
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
|
2000-01-26 06:58:53 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/backend/utils/cache/relcache.c
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
/*
|
|
|
|
* INTERFACE ROUTINES
|
2006-05-06 17:51:07 +02:00
|
|
|
* RelationCacheInitialize - initialize relcache (to empty)
|
2009-08-12 22:53:31 +02:00
|
|
|
* RelationCacheInitializePhase2 - initialize shared-catalog entries
|
|
|
|
* RelationCacheInitializePhase3 - finish initializing relcache
|
1996-07-09 08:22:35 +02:00
|
|
|
* RelationIdGetRelation - get a reldesc by relation id
|
|
|
|
* RelationClose - close an open relation
|
|
|
|
*
|
|
|
|
* NOTES
|
|
|
|
* The following code contains many undocumented hacks. Please be
|
|
|
|
* careful....
|
|
|
|
*/
|
2000-11-10 01:33:12 +01:00
|
|
|
#include "postgres.h"
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
#include <sys/file.h>
|
1996-11-27 09:16:44 +01:00
|
|
|
#include <fcntl.h>
|
2000-03-31 21:39:22 +02:00
|
|
|
#include <unistd.h>
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
#include "access/htup_details.h"
|
|
|
|
#include "access/multixact.h"
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
#include "access/nbtree.h"
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
#include "access/parallel.h"
|
2006-07-04 00:45:41 +02:00
|
|
|
#include "access/reloptions.h"
|
2008-05-12 02:00:54 +02:00
|
|
|
#include "access/sysattr.h"
|
2019-01-21 19:18:20 +01:00
|
|
|
#include "access/table.h"
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
#include "access/tableam.h"
|
2018-03-28 02:13:52 +02:00
|
|
|
#include "access/tupdesc_details.h"
|
2006-07-13 18:49:20 +02:00
|
|
|
#include "access/xact.h"
|
2014-11-06 12:52:08 +01:00
|
|
|
#include "access/xlog.h"
|
2022-07-28 22:03:42 +02:00
|
|
|
#include "catalog/binary_upgrade.h"
|
2006-07-31 22:09:10 +02:00
|
|
|
#include "catalog/catalog.h"
|
1998-04-27 06:08:07 +02:00
|
|
|
#include "catalog/indexing.h"
|
2002-08-06 04:36:35 +02:00
|
|
|
#include "catalog/namespace.h"
|
2020-04-08 09:59:27 +02:00
|
|
|
#include "catalog/partition.h"
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
#include "catalog/pg_am.h"
|
2002-02-19 21:11:20 +01:00
|
|
|
#include "catalog/pg_amproc.h"
|
1998-04-27 06:08:07 +02:00
|
|
|
#include "catalog/pg_attrdef.h"
|
2010-04-21 01:48:47 +02:00
|
|
|
#include "catalog/pg_auth_members.h"
|
2019-11-12 04:00:16 +01:00
|
|
|
#include "catalog/pg_authid.h"
|
2002-07-12 20:43:19 +02:00
|
|
|
#include "catalog/pg_constraint.h"
|
2009-08-12 22:53:31 +02:00
|
|
|
#include "catalog/pg_database.h"
|
2002-03-26 20:17:02 +01:00
|
|
|
#include "catalog/pg_namespace.h"
|
2002-02-19 21:11:20 +01:00
|
|
|
#include "catalog/pg_opclass.h"
|
1999-07-16 07:23:30 +02:00
|
|
|
#include "catalog/pg_proc.h"
|
2017-01-19 18:00:00 +01:00
|
|
|
#include "catalog/pg_publication.h"
|
1996-07-09 08:22:35 +02:00
|
|
|
#include "catalog/pg_rewrite.h"
|
2016-01-05 18:50:53 +01:00
|
|
|
#include "catalog/pg_shseclabel.h"
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
#include "catalog/pg_statistic_ext.h"
|
2017-01-19 18:00:00 +01:00
|
|
|
#include "catalog/pg_subscription.h"
|
2009-08-12 22:53:31 +02:00
|
|
|
#include "catalog/pg_tablespace.h"
|
2010-01-14 00:07:08 +01:00
|
|
|
#include "catalog/pg_trigger.h"
|
1996-07-09 08:22:35 +02:00
|
|
|
#include "catalog/pg_type.h"
|
2010-01-05 02:06:57 +01:00
|
|
|
#include "catalog/schemapg.h"
|
2010-02-03 02:14:17 +01:00
|
|
|
#include "catalog/storage.h"
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
#include "commands/policy.h"
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
#include "commands/publicationcmds.h"
|
2000-01-31 05:35:57 +01:00
|
|
|
#include "commands/trigger.h"
|
1998-04-27 06:08:07 +02:00
|
|
|
#include "miscadmin.h"
|
2019-01-29 21:26:44 +01:00
|
|
|
#include "nodes/makefuncs.h"
|
2016-06-10 22:03:37 +02:00
|
|
|
#include "nodes/nodeFuncs.h"
|
2019-01-29 21:48:51 +01:00
|
|
|
#include "optimizer/optimizer.h"
|
2022-04-07 06:29:46 +02:00
|
|
|
#include "pgstat.h"
|
2006-09-05 23:08:36 +02:00
|
|
|
#include "rewrite/rewriteDefine.h"
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
#include "rewrite/rowsecurity.h"
|
2008-05-12 02:00:54 +02:00
|
|
|
#include "storage/lmgr.h"
|
1998-04-27 06:08:07 +02:00
|
|
|
#include "storage/smgr.h"
|
2011-09-04 07:13:16 +02:00
|
|
|
#include "utils/array.h"
|
2000-11-10 01:33:12 +01:00
|
|
|
#include "utils/builtins.h"
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
#include "utils/catcache.h"
|
2018-03-28 02:13:52 +02:00
|
|
|
#include "utils/datum.h"
|
2000-05-28 19:56:29 +02:00
|
|
|
#include "utils/fmgroids.h"
|
2002-02-19 21:11:20 +01:00
|
|
|
#include "utils/inval.h"
|
2009-12-07 06:22:23 +01:00
|
|
|
#include "utils/lsyscache.h"
|
2005-05-06 19:24:55 +02:00
|
|
|
#include "utils/memutils.h"
|
2010-02-07 21:48:13 +01:00
|
|
|
#include "utils/relmapper.h"
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
#include "utils/resowner.h"
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
#include "utils/snapmgr.h"
|
2002-02-19 21:11:20 +01:00
|
|
|
#include "utils/syscache.h"
|
1998-04-27 06:08:07 +02:00
|
|
|
|
2010-11-29 18:29:42 +01:00
|
|
|
#define RELCACHE_INIT_FILEMAGIC 0x573266 /* version ID value */
|
2003-11-09 22:30:38 +01:00
|
|
|
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
/*
|
2021-01-06 10:15:19 +01:00
|
|
|
* Whether to bother checking if relation cache memory needs to be freed
|
|
|
|
* eagerly. See also RelationBuildDesc() and pg_config_manual.h.
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
*/
|
2021-01-06 10:15:19 +01:00
|
|
|
#if defined(RECOVER_RELATION_BUILD_MEMORY) && (RECOVER_RELATION_BUILD_MEMORY != 0)
|
|
|
|
#define MAYBE_RECOVER_RELATION_BUILD_MEMORY 1
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
#else
|
|
|
|
#define RECOVER_RELATION_BUILD_MEMORY 0
|
2021-07-13 21:01:01 +02:00
|
|
|
#ifdef DISCARD_CACHES_ENABLED
|
2021-01-06 10:15:19 +01:00
|
|
|
#define MAYBE_RECOVER_RELATION_BUILD_MEMORY 1
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
#endif
|
|
|
|
#endif
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2010-11-29 18:29:42 +01:00
|
|
|
* hardcoded tuple descriptors, contents generated by genbki.pl
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2009-08-12 22:53:31 +02:00
|
|
|
static const FormData_pg_attribute Desc_pg_class[Natts_pg_class] = {Schema_pg_class};
|
|
|
|
static const FormData_pg_attribute Desc_pg_attribute[Natts_pg_attribute] = {Schema_pg_attribute};
|
|
|
|
static const FormData_pg_attribute Desc_pg_proc[Natts_pg_proc] = {Schema_pg_proc};
|
|
|
|
static const FormData_pg_attribute Desc_pg_type[Natts_pg_type] = {Schema_pg_type};
|
|
|
|
static const FormData_pg_attribute Desc_pg_database[Natts_pg_database] = {Schema_pg_database};
|
2010-04-21 01:48:47 +02:00
|
|
|
static const FormData_pg_attribute Desc_pg_authid[Natts_pg_authid] = {Schema_pg_authid};
|
|
|
|
static const FormData_pg_attribute Desc_pg_auth_members[Natts_pg_auth_members] = {Schema_pg_auth_members};
|
2009-08-12 22:53:31 +02:00
|
|
|
static const FormData_pg_attribute Desc_pg_index[Natts_pg_index] = {Schema_pg_index};
|
2016-01-05 18:50:53 +01:00
|
|
|
static const FormData_pg_attribute Desc_pg_shseclabel[Natts_pg_shseclabel] = {Schema_pg_shseclabel};
|
2017-01-19 18:00:00 +01:00
|
|
|
static const FormData_pg_attribute Desc_pg_subscription[Natts_pg_subscription] = {Schema_pg_subscription};
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-01-31 05:35:57 +01:00
|
|
|
* Hash tables that index the relation cache
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2005-04-14 22:03:27 +02:00
|
|
|
* We used to index the cache by both name and OID, but now there
|
|
|
|
* is only an index by OID.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2005-04-14 22:03:27 +02:00
|
|
|
typedef struct relidcacheent
|
|
|
|
{
|
|
|
|
Oid reloid;
|
|
|
|
Relation reldesc;
|
|
|
|
} RelIdCacheEnt;
|
|
|
|
|
2000-01-31 05:35:57 +01:00
|
|
|
static HTAB *RelationIdCache;
|
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
/*
|
|
|
|
* This flag is false until we have prepared the critical relcache entries
|
|
|
|
* that are needed to do indexscans on the tables read by relcache building.
|
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
bool criticalRelcachesBuilt = false;
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* This flag is false until we have prepared the critical relcache entries
|
2010-04-21 01:48:47 +02:00
|
|
|
* for shared catalogs (which are the tables needed for login).
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
|
|
|
bool criticalSharedRelcachesBuilt = false;
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* This counter counts relcache inval events received since backend startup
|
|
|
|
* (but only for rels that are actually in cache). Presently, we use it only
|
|
|
|
* to detect whether data about to be written by write_relcache_init_file()
|
|
|
|
* might already be obsolete.
|
|
|
|
*/
|
|
|
|
static long relcacheInvalsReceived = 0L;
|
2000-06-30 09:04:23 +02:00
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/*
|
|
|
|
* in_progress_list is a stack of ongoing RelationBuildDesc() calls. CREATE
|
|
|
|
* INDEX CONCURRENTLY makes catalog changes under ShareUpdateExclusiveLock.
|
|
|
|
* It critically relies on each backend absorbing those changes no later than
|
|
|
|
* next transaction start. Hence, RelationBuildDesc() loops until it finishes
|
|
|
|
* without accepting a relevant invalidation. (Most invalidation consumers
|
|
|
|
* don't do this.)
|
|
|
|
*/
|
|
|
|
typedef struct inprogressent
|
|
|
|
{
|
|
|
|
Oid reloid; /* OID of relation being built */
|
|
|
|
bool invalidated; /* whether an invalidation arrived for it */
|
|
|
|
} InProgressEnt;
|
|
|
|
|
|
|
|
static InProgressEnt *in_progress_list;
|
|
|
|
static int in_progress_list_len;
|
|
|
|
static int in_progress_list_maxlen;
|
|
|
|
|
2004-11-20 21:19:52 +01:00
|
|
|
/*
|
2013-01-20 19:44:49 +01:00
|
|
|
* eoxact_list[] stores the OIDs of relations that (might) need AtEOXact
|
|
|
|
* cleanup work. This list intentionally has limited size; if it overflows,
|
|
|
|
* we fall back to scanning the whole hashtable. There is no value in a very
|
|
|
|
* large list because (1) at some point, a hash_seq_search scan is faster than
|
|
|
|
* retail lookups, and (2) the value of this is to reduce EOXact work for
|
|
|
|
* short transactions, which can't have dirtied all that many tables anyway.
|
|
|
|
* EOXactListAdd() does not bother to prevent duplicate list entries, so the
|
|
|
|
* cleanup processing must be idempotent.
|
2004-11-20 21:19:52 +01:00
|
|
|
*/
|
2013-01-20 19:44:49 +01:00
|
|
|
#define MAX_EOXACT_LIST 32
|
|
|
|
static Oid eoxact_list[MAX_EOXACT_LIST];
|
|
|
|
static int eoxact_list_len = 0;
|
|
|
|
static bool eoxact_list_overflowed = false;
|
|
|
|
|
|
|
|
#define EOXactListAdd(rel) \
|
|
|
|
do { \
|
|
|
|
if (eoxact_list_len < MAX_EOXACT_LIST) \
|
|
|
|
eoxact_list[eoxact_list_len++] = (rel)->rd_id; \
|
|
|
|
else \
|
|
|
|
eoxact_list_overflowed = true; \
|
|
|
|
} while (0)
|
2004-11-20 21:19:52 +01:00
|
|
|
|
2014-04-06 17:13:43 +02:00
|
|
|
/*
|
|
|
|
* EOXactTupleDescArray stores TupleDescs that (might) need AtEOXact
|
|
|
|
* cleanup work. The array expands as needed; there is no hashtable because
|
|
|
|
* we don't need to access individual items except at EOXact.
|
|
|
|
*/
|
|
|
|
static TupleDesc *EOXactTupleDescArray;
|
|
|
|
static int NextEOXactTupleDescNum = 0;
|
|
|
|
static int EOXactTupleDescArrayLen = 0;
|
2002-03-26 20:17:02 +01:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
* macros to manipulate the lookup hashtable
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
#define RelationCacheInsert(RELATION, replace_allowed) \
|
1998-06-15 20:40:05 +02:00
|
|
|
do { \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
RelIdCacheEnt *hentry; bool found; \
|
|
|
|
hentry = (RelIdCacheEnt *) hash_search(RelationIdCache, \
|
2023-02-06 09:05:20 +01:00
|
|
|
&((RELATION)->rd_id), \
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
HASH_ENTER, &found); \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
if (found) \
|
|
|
|
{ \
|
2014-05-19 00:17:55 +02:00
|
|
|
/* see comments in RelationBuildDesc and RelationBuildLocalRelation */ \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
Relation _old_rel = hentry->reldesc; \
|
|
|
|
Assert(replace_allowed); \
|
|
|
|
hentry->reldesc = (RELATION); \
|
|
|
|
if (RelationHasReferenceCountZero(_old_rel)) \
|
|
|
|
RelationDestroyRelation(_old_rel, false); \
|
2014-05-19 00:17:55 +02:00
|
|
|
else if (!IsBootstrapProcessingMode()) \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
elog(WARNING, "leaking still-referenced relcache entry for \"%s\"", \
|
|
|
|
RelationGetRelationName(_old_rel)); \
|
|
|
|
} \
|
|
|
|
else \
|
|
|
|
hentry->reldesc = (RELATION); \
|
1998-06-15 20:40:05 +02:00
|
|
|
} while(0)
|
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
#define RelationIdCacheLookup(ID, RELATION) \
|
1998-06-15 20:40:05 +02:00
|
|
|
do { \
|
2002-03-26 20:17:02 +01:00
|
|
|
RelIdCacheEnt *hentry; \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
hentry = (RelIdCacheEnt *) hash_search(RelationIdCache, \
|
2023-02-06 09:05:20 +01:00
|
|
|
&(ID), \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
HASH_FIND, NULL); \
|
2001-10-05 19:28:13 +02:00
|
|
|
if (hentry) \
|
1998-06-15 20:40:05 +02:00
|
|
|
RELATION = hentry->reldesc; \
|
|
|
|
else \
|
|
|
|
RELATION = NULL; \
|
|
|
|
} while(0)
|
|
|
|
|
|
|
|
#define RelationCacheDelete(RELATION) \
|
|
|
|
do { \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
RelIdCacheEnt *hentry; \
|
|
|
|
hentry = (RelIdCacheEnt *) hash_search(RelationIdCache, \
|
2023-02-06 09:05:20 +01:00
|
|
|
&((RELATION)->rd_id), \
|
2001-10-05 19:28:13 +02:00
|
|
|
HASH_REMOVE, NULL); \
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
if (hentry == NULL) \
|
|
|
|
elog(WARNING, "failed to delete relcache entry for OID %u", \
|
|
|
|
(RELATION)->rd_id); \
|
1998-06-15 20:40:05 +02:00
|
|
|
} while(0)
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Special cache for opclass-related information
|
2003-11-09 22:30:38 +01:00
|
|
|
*
|
2010-11-29 18:29:42 +01:00
|
|
|
* Note: only default support procs get cached, ie, those with
|
2006-12-23 01:43:13 +01:00
|
|
|
* lefttype = righttype = opcintype.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
typedef struct opclasscacheent
|
|
|
|
{
|
|
|
|
Oid opclassoid; /* lookup key: OID of opclass */
|
2017-08-16 06:22:32 +02:00
|
|
|
bool valid; /* set true after successful fill-in */
|
2002-02-19 21:11:20 +01:00
|
|
|
StrategyNumber numSupport; /* max # of support procs (from pg_am) */
|
2006-12-23 01:43:13 +01:00
|
|
|
Oid opcfamily; /* OID of opclass's family */
|
|
|
|
Oid opcintype; /* OID of opclass's declared input type */
|
2010-11-29 18:29:42 +01:00
|
|
|
RegProcedure *supportProcs; /* OIDs of support procedures */
|
2002-02-19 21:11:20 +01:00
|
|
|
} OpClassCacheEnt;
|
|
|
|
|
|
|
|
static HTAB *OpClassCache = NULL;
|
|
|
|
|
|
|
|
|
1997-08-21 03:36:09 +02:00
|
|
|
/* non-export function prototypes */
|
2000-01-31 05:35:57 +01:00
|
|
|
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
static void RelationCloseCleanup(Relation relation);
|
2014-04-06 17:13:43 +02:00
|
|
|
static void RelationDestroyRelation(Relation relation, bool remember_tupdesc);
|
2002-07-15 03:57:51 +02:00
|
|
|
static void RelationClearRelation(Relation relation, bool rebuild);
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2007-05-02 23:08:46 +02:00
|
|
|
static void RelationReloadIndexInfo(Relation relation);
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
static void RelationReloadNailed(Relation relation);
|
2001-01-02 05:33:24 +01:00
|
|
|
static void RelationFlushRelation(Relation relation);
|
2014-04-06 17:13:43 +02:00
|
|
|
static void RememberToFreeTupleDescAtEOX(TupleDesc td);
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
#ifdef USE_ASSERT_CHECKING
|
|
|
|
static void AssertPendingSyncConsistency(Relation relation);
|
|
|
|
#endif
|
2013-01-20 19:44:49 +01:00
|
|
|
static void AtEOXact_cleanup(Relation relation, bool isCommit);
|
|
|
|
static void AtEOSubXact_cleanup(Relation relation, bool isCommit,
|
|
|
|
SubTransactionId mySubid, SubTransactionId parentSubid);
|
2009-08-12 22:53:31 +02:00
|
|
|
static bool load_relcache_init_file(bool shared);
|
|
|
|
static void write_relcache_init_file(bool shared);
|
2006-07-04 00:45:41 +02:00
|
|
|
static void write_item(const void *data, Size len, FILE *fp);
|
2000-01-31 05:35:57 +01:00
|
|
|
|
2009-09-27 01:08:22 +02:00
|
|
|
static void formrdesc(const char *relationName, Oid relationReltype,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
bool isshared, int natts, const FormData_pg_attribute *attrs);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
static HeapTuple ScanPgRelation(Oid targetRelId, bool indexOK, bool force_non_historic);
|
2010-01-12 19:12:18 +01:00
|
|
|
static Relation AllocateRelationDesc(Form_pg_class relp);
|
2006-07-04 00:45:41 +02:00
|
|
|
static void RelationParseRelOptions(Relation relation, HeapTuple tuple);
|
2005-04-14 22:03:27 +02:00
|
|
|
static void RelationBuildTupleDesc(Relation relation);
|
2010-01-12 19:12:18 +01:00
|
|
|
static Relation RelationBuildDesc(Oid targetRelId, bool insertIt);
|
2004-06-18 08:14:31 +02:00
|
|
|
static void RelationInitPhysicalAddr(Relation relation);
|
2010-01-14 00:07:08 +01:00
|
|
|
static void load_critical_index(Oid indexoid, Oid heapoid);
|
2006-07-04 00:45:41 +02:00
|
|
|
static TupleDesc GetPgClassDescriptor(void);
|
2005-03-29 02:17:27 +02:00
|
|
|
static TupleDesc GetPgIndexDescriptor(void);
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
static void AttrDefaultFetch(Relation relation, int ndef);
|
|
|
|
static int AttrDefaultCmp(const void *a, const void *b);
|
2002-07-12 20:43:19 +02:00
|
|
|
static void CheckConstraintFetch(Relation relation);
|
Apply table and domain CHECK constraints in name order.
Previously, CHECK constraints of the same scope were checked in whatever
order they happened to be read from pg_constraint. (Usually, but not
reliably, this would be creation order for domain constraints and reverse
creation order for table constraints, because of differing implementation
details.) Nondeterministic results of this sort are problematic at least
for testing purposes, and in discussion it was agreed to be a violation of
the principle of least astonishment. Therefore, borrow the principle
already established for triggers, and apply such checks in name order
(using strcmp() sort rules). This lets users control the check order
if they have a mind to.
Domain CHECK constraints still follow the rule of checking lower nested
domains' constraints first; the name sort only applies to multiple
constraints attached to the same domain.
In passing, I failed to resist the temptation to wordsmith a bit in
create_domain.sgml.
Apply to HEAD only, since this could result in a behavioral change in
existing applications, and the potential regression test failures have
not actually been observed in our buildfarm.
2015-03-23 21:59:29 +01:00
|
|
|
static int CheckConstraintCmp(const void *a, const void *b);
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
static void InitIndexAmRoutine(Relation relation);
|
2005-03-29 02:17:27 +02:00
|
|
|
static void IndexSupportInitialize(oidvector *indclass,
|
2002-02-19 21:11:20 +01:00
|
|
|
RegProcedure *indexSupport,
|
2006-12-23 01:43:13 +01:00
|
|
|
Oid *opFamily,
|
|
|
|
Oid *opcInType,
|
2002-02-19 21:11:20 +01:00
|
|
|
StrategyNumber maxSupportNumber,
|
|
|
|
AttrNumber maxAttributeNumber);
|
|
|
|
static OpClassCacheEnt *LookupOpclassInfo(Oid operatorClassOid,
|
|
|
|
StrategyNumber numSupport);
|
2009-08-12 22:53:31 +02:00
|
|
|
static void RelationCacheInitFileRemoveInDir(const char *tblspcpath);
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
static void unlink_initfile(const char *initfilename, int elevel);
|
1997-08-21 03:36:09 +02:00
|
|
|
|
2000-04-12 19:17:23 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* ScanPgRelation
|
|
|
|
*
|
2008-04-16 20:23:04 +02:00
|
|
|
* This is used by RelationBuildDesc to find a pg_class
|
|
|
|
* tuple matching targetRelId. The caller must hold at least
|
|
|
|
* AccessShareLock on the target relid to prevent concurrent-update
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
* scenarios; it isn't guaranteed that all scans used to build the
|
|
|
|
* relcache entry will use the same snapshot. If, for example,
|
|
|
|
* an attribute were to be added after scanning pg_class and before
|
|
|
|
* scanning pg_attribute, relnatts wouldn't match.
|
2000-05-21 04:28:55 +02:00
|
|
|
*
|
|
|
|
* NB: the returned tuple has been copied into palloc'd storage
|
|
|
|
* and must eventually be freed with heap_freetuple.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static HeapTuple
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
ScanPgRelation(Oid targetRelId, bool indexOK, bool force_non_historic)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
HeapTuple pg_class_tuple;
|
|
|
|
Relation pg_class_desc;
|
2002-02-19 21:11:20 +01:00
|
|
|
SysScanDesc pg_class_scan;
|
2005-04-14 22:03:27 +02:00
|
|
|
ScanKeyData key[1];
|
2020-03-28 19:52:11 +01:00
|
|
|
Snapshot snapshot = NULL;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* If something goes wrong during backend startup, we might find ourselves
|
|
|
|
* trying to read pg_class before we've selected a database. That ain't
|
|
|
|
* gonna work, so bail out with a useful error message. If this happens,
|
|
|
|
* it probably means a relcache entry that needs to be nailed isn't.
|
|
|
|
*/
|
|
|
|
if (!OidIsValid(MyDatabaseId))
|
|
|
|
elog(FATAL, "cannot read pg_class without having selected a database");
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* form a scan key
|
|
|
|
*/
|
2005-04-14 22:03:27 +02:00
|
|
|
ScanKeyInit(&key[0],
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Anum_pg_class_oid,
|
2005-04-14 22:03:27 +02:00
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(targetRelId));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* Open pg_class and fetch a tuple. Force heap scan if we haven't yet
|
|
|
|
* built the critical relcache entries (this includes initdb and startup
|
2003-09-24 20:54:02 +02:00
|
|
|
* without a pg_internal.init file). The caller can also force a heap
|
|
|
|
* scan by setting indexOK == false.
|
1997-09-07 07:04:48 +02:00
|
|
|
*/
|
2019-01-21 19:32:19 +01:00
|
|
|
pg_class_desc = table_open(RelationRelationId, AccessShareLock);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The caller might need a tuple that's newer than the one the historic
|
|
|
|
* snapshot; currently the only case requiring to do so is looking up the
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* relfilenumber of non mapped system relations during decoding. That
|
2020-04-05 21:03:09 +02:00
|
|
|
* snapshot can't change in the midst of a relcache build, so there's no
|
2020-03-28 19:52:11 +01:00
|
|
|
* need to register the snapshot.
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
*/
|
|
|
|
if (force_non_historic)
|
|
|
|
snapshot = GetNonHistoricCatalogSnapshot(RelationRelationId);
|
|
|
|
|
2005-04-14 22:03:27 +02:00
|
|
|
pg_class_scan = systable_beginscan(pg_class_desc, ClassOidIndexId,
|
2003-09-24 20:54:02 +02:00
|
|
|
indexOK && criticalRelcachesBuilt,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
snapshot,
|
2005-04-14 22:03:27 +02:00
|
|
|
1, key);
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
pg_class_tuple = systable_getnext(pg_class_scan);
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2000-12-08 07:17:58 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* Must copy tuple before releasing buffer.
|
2000-12-08 07:17:58 +01:00
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
if (HeapTupleIsValid(pg_class_tuple))
|
|
|
|
pg_class_tuple = heap_copytuple(pg_class_tuple);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* all done */
|
|
|
|
systable_endscan(pg_class_scan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(pg_class_desc, AccessShareLock);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
return pg_class_tuple;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* AllocateRelationDesc
|
|
|
|
*
|
|
|
|
* This is used to allocate memory for a new relation descriptor
|
2010-01-12 19:12:18 +01:00
|
|
|
* and initialize the rd_rel field from the given pg_class tuple.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static Relation
|
2010-01-12 19:12:18 +01:00
|
|
|
AllocateRelationDesc(Form_pg_class relp)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2010-01-12 19:12:18 +01:00
|
|
|
Relation relation;
|
2000-06-30 09:04:23 +02:00
|
|
|
MemoryContext oldcxt;
|
1996-07-09 08:22:35 +02:00
|
|
|
Form_pg_class relationForm;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
/* Relcache entries must live in CacheMemoryContext */
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2010-01-12 19:12:18 +01:00
|
|
|
* allocate and zero space for new relation descriptor
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2010-01-12 19:12:18 +01:00
|
|
|
relation = (Relation) palloc0(sizeof(RelationData));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1999-09-04 20:42:15 +02:00
|
|
|
/* make sure relation is marked as having no open file yet */
|
2004-02-10 02:55:27 +01:00
|
|
|
relation->rd_smgr = NULL;
|
1999-09-04 20:42:15 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-06-30 09:04:23 +02:00
|
|
|
* Copy the relation tuple form
|
|
|
|
*
|
|
|
|
* We only allocate space for the fixed fields, ie, CLASS_TUPLE_SIZE. The
|
2006-07-04 00:45:41 +02:00
|
|
|
* variable-length fields (relacl, reloptions) are NOT stored in the
|
|
|
|
* relcache --- there'd be little point in it, since we don't copy the
|
|
|
|
* tuple's nulls bitmap and hence wouldn't know if the values are valid.
|
|
|
|
* Bottom line is that relacl *cannot* be retrieved from the relcache. Get
|
|
|
|
* it from the syscache if you need it. The same goes for the original
|
|
|
|
* form of reloptions (however, we do store the parsed form of reloptions
|
|
|
|
* in rd_options).
|
2000-06-30 09:04:23 +02:00
|
|
|
*/
|
|
|
|
relationForm = (Form_pg_class) palloc(CLASS_TUPLE_SIZE);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2005-05-11 03:26:02 +02:00
|
|
|
memcpy(relationForm, relp, CLASS_TUPLE_SIZE);
|
2000-06-30 09:04:23 +02:00
|
|
|
|
|
|
|
/* initialize relation tuple form */
|
1996-07-09 08:22:35 +02:00
|
|
|
relation->rd_rel = relationForm;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
/* and allocate attribute tuple form storage */
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
relation->rd_att = CreateTemplateTupleDesc(relationForm->relnatts);
|
2006-06-16 20:42:24 +02:00
|
|
|
/* which we mark as a reference-counted tupdesc */
|
|
|
|
relation->rd_att->tdrefcount = 1;
|
2000-06-30 09:04:23 +02:00
|
|
|
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
return relation;
|
|
|
|
}
|
|
|
|
|
2006-07-02 04:23:23 +02:00
|
|
|
/*
|
2006-07-04 00:45:41 +02:00
|
|
|
* RelationParseRelOptions
|
|
|
|
* Convert pg_class.reloptions into pre-parsed rd_options
|
|
|
|
*
|
|
|
|
* tuple is the real pg_class tuple (not rd_rel!) for relation
|
|
|
|
*
|
2019-01-22 02:36:55 +01:00
|
|
|
* Note: rd_rel and (if an index) rd_indam must be valid already
|
2006-07-02 04:23:23 +02:00
|
|
|
*/
|
|
|
|
static void
|
2006-07-04 00:45:41 +02:00
|
|
|
RelationParseRelOptions(Relation relation, HeapTuple tuple)
|
2006-07-02 04:23:23 +02:00
|
|
|
{
|
2006-07-04 00:45:41 +02:00
|
|
|
bytea *options;
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
amoptions_function amoptsfn;
|
2006-07-02 04:23:23 +02:00
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
relation->rd_options = NULL;
|
2006-07-02 04:23:23 +02:00
|
|
|
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
/*
|
|
|
|
* Look up any AM-specific parse function; fall out if relkind should not
|
|
|
|
* have options.
|
|
|
|
*/
|
2006-07-02 04:23:23 +02:00
|
|
|
switch (relation->rd_rel->relkind)
|
|
|
|
{
|
2006-07-04 00:45:41 +02:00
|
|
|
case RELKIND_RELATION:
|
|
|
|
case RELKIND_TOASTVALUE:
|
2011-12-22 22:15:57 +01:00
|
|
|
case RELKIND_VIEW:
|
2013-03-04 01:23:31 +01:00
|
|
|
case RELKIND_MATVIEW:
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
case RELKIND_PARTITIONED_TABLE:
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
amoptsfn = NULL;
|
|
|
|
break;
|
|
|
|
case RELKIND_INDEX:
|
|
|
|
case RELKIND_PARTITIONED_INDEX:
|
2019-01-22 02:36:55 +01:00
|
|
|
amoptsfn = relation->rd_indam->amoptions;
|
2006-07-04 00:45:41 +02:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return;
|
2006-07-02 04:23:23 +02:00
|
|
|
}
|
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
/*
|
|
|
|
* Fetch reloptions from tuple; have to use a hardwired descriptor because
|
|
|
|
* we might not have any other for pg_class yet (consider executing this
|
|
|
|
* code for pg_class itself)
|
|
|
|
*/
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
options = extractRelOptions(tuple, GetPgClassDescriptor(), amoptsfn);
|
2006-07-04 00:45:41 +02:00
|
|
|
|
2010-01-07 21:39:45 +01:00
|
|
|
/*
|
|
|
|
* Copy parsed data into CacheMemoryContext. To guard against the
|
|
|
|
* possibility of leaks in the reloptions code, we want to do the actual
|
|
|
|
* parsing in the caller's memory context and copy the results into
|
|
|
|
* CacheMemoryContext after the fact.
|
|
|
|
*/
|
2006-07-04 00:45:41 +02:00
|
|
|
if (options)
|
|
|
|
{
|
|
|
|
relation->rd_options = MemoryContextAlloc(CacheMemoryContext,
|
|
|
|
VARSIZE(options));
|
|
|
|
memcpy(relation->rd_options, options, VARSIZE(options));
|
2010-01-12 19:12:18 +01:00
|
|
|
pfree(options);
|
2006-07-02 04:23:23 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* RelationBuildTupleDesc
|
|
|
|
*
|
|
|
|
* Form the relation's tuple descriptor from information in
|
2002-07-12 20:43:19 +02:00
|
|
|
* the pg_attribute, pg_attrdef & pg_constraint system catalogs.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static void
|
2005-04-14 22:03:27 +02:00
|
|
|
RelationBuildTupleDesc(Relation relation)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
HeapTuple pg_attribute_tuple;
|
|
|
|
Relation pg_attribute_desc;
|
2002-02-19 21:11:20 +01:00
|
|
|
SysScanDesc pg_attribute_scan;
|
|
|
|
ScanKeyData skey[2];
|
1996-07-09 08:22:35 +02:00
|
|
|
int need;
|
2000-06-30 09:04:23 +02:00
|
|
|
TupleConstr *constr;
|
2018-03-28 02:13:52 +02:00
|
|
|
AttrMissing *attrmiss = NULL;
|
2000-02-18 10:30:20 +01:00
|
|
|
int ndef = 0;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Don't create pg_type entries for sequences or toast tables.
Commit f7f70d5e2 left one inconsistency behind: we're still creating
pg_type entries for the composite types of sequences and toast tables,
but not arrays over those composites. But there seems precious little
reason to have named composite types for toast tables, and not much more
to have them for sequences (especially given the thought that sequences
may someday not be standalone relations at all).
So, let's close that inconsistency by removing these composite types,
rather than adding arrays for them. This buys back a little bit of
the initial pg_type bloat added by the previous patch, and could be
a significant savings in a large database with many toast tables.
Aside from a small logic rearrangement in heap_create_with_catalog,
this patch mostly needs to clean up some places that were assuming that
pg_class.reltype always has a valid value. Those are really pre-existing
bugs, given that it's documented otherwise; notably, the plpgsql changes
fix code that gives "cache lookup failed for type 0" on indexes today.
But none of these seem interesting enough to back-patch.
Also, remove the pg_dump/pg_upgrade infrastructure for propagating
a toast table's pg_type OID into the new database, since we no longer
need that.
Discussion: https://postgr.es/m/761F1389-C6A8-4C15-80CE-950C961F5341@gmail.com
2020-07-07 21:43:22 +02:00
|
|
|
/* fill rd_att's type ID fields (compare heap.c's AddNewRelationTuple) */
|
|
|
|
relation->rd_att->tdtypeid =
|
|
|
|
relation->rd_rel->reltype ? relation->rd_rel->reltype : RECORDOID;
|
|
|
|
relation->rd_att->tdtypmod = -1; /* just to be sure */
|
2002-09-02 03:05:06 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
constr = (TupleConstr *) MemoryContextAllocZero(CacheMemoryContext,
|
|
|
|
sizeof(TupleConstr));
|
2000-02-18 10:30:20 +01:00
|
|
|
constr->has_not_null = false;
|
2019-03-30 08:13:09 +01:00
|
|
|
constr->has_generated_stored = false;
|
2000-06-30 09:04:23 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* Form a scan key that selects only user attributes (attnum > 0).
|
|
|
|
* (Eliminating system attribute rows at the index level is lots faster
|
|
|
|
* than fetching them.)
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey[0],
|
|
|
|
Anum_pg_attribute_attrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
|
|
|
ScanKeyInit(&skey[1],
|
|
|
|
Anum_pg_attribute_attnum,
|
|
|
|
BTGreaterStrategyNumber, F_INT2GT,
|
|
|
|
Int16GetDatum(0));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* Open pg_attribute and begin a scan. Force heap scan if we haven't yet
|
|
|
|
* built the critical relcache entries (this includes initdb and startup
|
|
|
|
* without a pg_internal.init file).
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2019-01-21 19:32:19 +01:00
|
|
|
pg_attribute_desc = table_open(AttributeRelationId, AccessShareLock);
|
2002-02-19 21:11:20 +01:00
|
|
|
pg_attribute_scan = systable_beginscan(pg_attribute_desc,
|
2005-04-14 22:03:27 +02:00
|
|
|
AttributeRelidNumIndexId,
|
2002-02-19 21:11:20 +01:00
|
|
|
criticalRelcachesBuilt,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL,
|
2002-02-19 21:11:20 +01:00
|
|
|
2, skey);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* add attribute data to relation->rd_att
|
|
|
|
*/
|
2018-04-07 22:00:39 +02:00
|
|
|
need = RelationGetNumberOfAttributes(relation);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
while (HeapTupleIsValid(pg_attribute_tuple = systable_getnext(pg_attribute_scan)))
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
Form_pg_attribute attp;
|
2018-03-28 02:13:52 +02:00
|
|
|
int attnum;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
attp = (Form_pg_attribute) GETSTRUCT(pg_attribute_tuple);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2018-03-28 02:13:52 +02:00
|
|
|
attnum = attp->attnum;
|
2018-04-07 22:00:39 +02:00
|
|
|
if (attnum <= 0 || attnum > RelationGetNumberOfAttributes(relation))
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
elog(ERROR, "invalid attribute number %d for relation \"%s\"",
|
2002-02-19 21:11:20 +01:00
|
|
|
attp->attnum, RelationGetRelationName(relation));
|
|
|
|
|
2018-03-28 02:13:52 +02:00
|
|
|
memcpy(TupleDescAttr(relation->rd_att, attnum - 1),
|
2005-03-07 05:42:17 +01:00
|
|
|
attp,
|
2009-01-22 21:16:10 +01:00
|
|
|
ATTRIBUTE_FIXED_PART_SIZE);
|
2000-06-30 09:04:23 +02:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
/* Update constraint/default info */
|
|
|
|
if (attp->attnotnull)
|
2002-02-19 21:11:20 +01:00
|
|
|
constr->has_not_null = true;
|
2019-03-30 08:13:09 +01:00
|
|
|
if (attp->attgenerated == ATTRIBUTE_GENERATED_STORED)
|
|
|
|
constr->has_generated_stored = true;
|
2002-02-19 21:11:20 +01:00
|
|
|
if (attp->atthasdef)
|
|
|
|
ndef++;
|
2018-03-28 02:13:52 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/* If the column has a "missing" value, put it in the attrmiss array */
|
2018-03-28 02:13:52 +02:00
|
|
|
if (attp->atthasmissing)
|
|
|
|
{
|
|
|
|
Datum missingval;
|
|
|
|
bool missingNull;
|
|
|
|
|
|
|
|
/* Do we have a missing value? */
|
|
|
|
missingval = heap_getattr(pg_attribute_tuple,
|
|
|
|
Anum_pg_attribute_attmissingval,
|
|
|
|
pg_attribute_desc->rd_att,
|
|
|
|
&missingNull);
|
|
|
|
if (!missingNull)
|
|
|
|
{
|
|
|
|
/* Yes, fetch from the array */
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
bool is_null;
|
|
|
|
int one = 1;
|
|
|
|
Datum missval;
|
|
|
|
|
|
|
|
if (attrmiss == NULL)
|
|
|
|
attrmiss = (AttrMissing *)
|
|
|
|
MemoryContextAllocZero(CacheMemoryContext,
|
|
|
|
relation->rd_rel->relnatts *
|
|
|
|
sizeof(AttrMissing));
|
|
|
|
|
|
|
|
missval = array_get_element(missingval,
|
|
|
|
1,
|
|
|
|
&one,
|
|
|
|
-1,
|
|
|
|
attp->attlen,
|
|
|
|
attp->attbyval,
|
|
|
|
attp->attalign,
|
|
|
|
&is_null);
|
|
|
|
Assert(!is_null);
|
|
|
|
if (attp->attbyval)
|
|
|
|
{
|
|
|
|
/* for copy by val just copy the datum direct */
|
2018-06-27 04:46:13 +02:00
|
|
|
attrmiss[attnum - 1].am_value = missval;
|
2018-03-28 02:13:52 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* otherwise copy in the correct context */
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
2018-06-27 04:46:13 +02:00
|
|
|
attrmiss[attnum - 1].am_value = datumCopy(missval,
|
|
|
|
attp->attbyval,
|
|
|
|
attp->attlen);
|
2018-03-28 02:13:52 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
2018-06-27 04:46:13 +02:00
|
|
|
attrmiss[attnum - 1].am_present = true;
|
2018-03-28 02:13:52 +02:00
|
|
|
}
|
|
|
|
}
|
2002-02-19 21:11:20 +01:00
|
|
|
need--;
|
|
|
|
if (need == 0)
|
|
|
|
break;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* end the scan and close the attribute relation
|
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
systable_endscan(pg_attribute_scan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(pg_attribute_desc, AccessShareLock);
|
2000-02-18 10:30:20 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
if (need != 0)
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
elog(ERROR, "pg_attribute catalog is missing %d attribute(s) for relation OID %u",
|
2002-02-19 21:11:20 +01:00
|
|
|
need, RelationGetRelid(relation));
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-11-30 19:38:47 +01:00
|
|
|
* The attcacheoff values we read from pg_attribute should all be -1
|
|
|
|
* ("unknown"). Verify this if assert checking is on. They will be
|
|
|
|
* computed when and if needed during tuple access.
|
|
|
|
*/
|
|
|
|
#ifdef USE_ASSERT_CHECKING
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
for (i = 0; i < RelationGetNumberOfAttributes(relation); i++)
|
2017-08-20 20:19:07 +02:00
|
|
|
Assert(TupleDescAttr(relation->rd_att, i)->attcacheoff == -1);
|
2000-11-30 19:38:47 +01:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-11-30 19:38:47 +01:00
|
|
|
* However, we can easily set the attcacheoff value for the first
|
|
|
|
* attribute: it must be zero. This eliminates the need for special cases
|
|
|
|
* for attnum=1 that used to exist in fastgetattr() and index_getattr().
|
|
|
|
*/
|
2018-04-07 22:00:39 +02:00
|
|
|
if (RelationGetNumberOfAttributes(relation) > 0)
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(relation->rd_att, 0)->attcacheoff = 0;
|
2000-11-30 19:38:47 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Set up constraint/default info
|
|
|
|
*/
|
2020-02-06 21:21:04 +01:00
|
|
|
if (constr->has_not_null ||
|
|
|
|
constr->has_generated_stored ||
|
|
|
|
ndef > 0 ||
|
|
|
|
attrmiss ||
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
relation->rd_rel->relchecks > 0)
|
1997-08-21 03:36:09 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
relation->rd_att->constr = constr;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
if (ndef > 0) /* DEFAULTs */
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
AttrDefaultFetch(relation, ndef);
|
2002-02-19 21:11:20 +01:00
|
|
|
else
|
|
|
|
constr->num_defval = 0;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2018-03-28 02:13:52 +02:00
|
|
|
constr->missing = attrmiss;
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
if (relation->rd_rel->relchecks > 0) /* CHECKs */
|
2002-07-12 20:43:19 +02:00
|
|
|
CheckConstraintFetch(relation);
|
2002-02-19 21:11:20 +01:00
|
|
|
else
|
|
|
|
constr->num_check = 0;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
pfree(constr);
|
|
|
|
relation->rd_att->constr = NULL;
|
1997-08-21 03:36:09 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* RelationBuildRuleLock
|
|
|
|
*
|
|
|
|
* Form the relation's rewrite rules from information in
|
|
|
|
* the pg_rewrite system catalog.
|
2000-06-30 09:04:23 +02:00
|
|
|
*
|
|
|
|
* Note: The rule parsetrees are potentially very complex node structures.
|
|
|
|
* To allow these trees to be freed when the relcache entry is flushed,
|
|
|
|
* we make a private memory context to hold the RuleLock information for
|
|
|
|
* each relcache entry that has associated rules. The context is used
|
|
|
|
* just for rule info, not for any other subsidiary data of the relcache
|
|
|
|
* entry, because that keeps the update logic in RelationClearRelation()
|
|
|
|
* manageable. The other subsidiary data structures are simple enough
|
|
|
|
* to be easy to free explicitly, anyway.
|
2022-03-22 11:28:10 +01:00
|
|
|
*
|
|
|
|
* Note: The relation's reloptions must have been extracted first.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
RelationBuildRuleLock(Relation relation)
|
|
|
|
{
|
2000-06-30 09:04:23 +02:00
|
|
|
MemoryContext rulescxt;
|
|
|
|
MemoryContext oldcxt;
|
2002-04-19 18:36:08 +02:00
|
|
|
HeapTuple rewrite_tuple;
|
|
|
|
Relation rewrite_desc;
|
|
|
|
TupleDesc rewrite_tupdesc;
|
|
|
|
SysScanDesc rewrite_scan;
|
1996-07-09 08:22:35 +02:00
|
|
|
ScanKeyData key;
|
|
|
|
RuleLock *rulelock;
|
|
|
|
int numlocks;
|
|
|
|
RewriteRule **rules;
|
|
|
|
int maxlocks;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
/*
|
Add macros to make AllocSetContextCreate() calls simpler and safer.
I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls
had typos in the context-sizing parameters. While none of these led to
especially significant problems, they did create minor inefficiencies,
and it's now clear that expecting people to copy-and-paste those calls
accurately is not a great idea. Let's reduce the risk of future errors
by introducing single macros that encapsulate the common use-cases.
Three such macros are enough to cover all but two special-purpose contexts;
those two calls can be left as-is, I think.
While this patch doesn't in itself improve matters for third-party
extensions, it doesn't break anything for them either, and they can
gradually adopt the simplified notation over time.
In passing, change TopMemoryContext to use the default allocation
parameters. Formerly it could only be extended 8K at a time. That was
probably reasonable when this code was written; but nowadays we create
many more contexts than we did then, so that it's not unusual to have a
couple hundred K in TopMemoryContext, even without considering various
dubious code that sticks other things there. There seems no good reason
not to let it use growing blocks like most other contexts.
Back-patch to 9.6, mostly because that's still close enough to HEAD that
it's easy to do so, and keeping the branches in sync can be expected to
avoid some future back-patching pain. The bugs fixed by these changes
don't seem to be significant enough to justify fixing them further back.
Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
|
|
|
* Make the private context. Assume it'll not contain much data.
|
2000-06-30 09:04:23 +02:00
|
|
|
*/
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
rulescxt = AllocSetContextCreate(CacheMemoryContext,
|
|
|
|
"relation rules",
|
|
|
|
ALLOCSET_SMALL_SIZES);
|
2000-06-30 09:04:23 +02:00
|
|
|
relation->rd_rulescxt = rulescxt;
|
2018-04-06 18:10:00 +02:00
|
|
|
MemoryContextCopyAndSetIdentifier(rulescxt,
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
RelationGetRelationName(relation));
|
2000-06-30 09:04:23 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-04-19 18:36:08 +02:00
|
|
|
* allocate an array to hold the rewrite rules (the array is extended if
|
1996-07-09 08:22:35 +02:00
|
|
|
* necessary)
|
|
|
|
*/
|
|
|
|
maxlocks = 4;
|
2000-06-30 09:04:23 +02:00
|
|
|
rules = (RewriteRule **)
|
|
|
|
MemoryContextAlloc(rulescxt, sizeof(RewriteRule *) * maxlocks);
|
1996-07-09 08:22:35 +02:00
|
|
|
numlocks = 0;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* form a scan key
|
|
|
|
*/
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&key,
|
|
|
|
Anum_pg_rewrite_ev_class,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-01-31 05:35:57 +01:00
|
|
|
* open pg_rewrite and begin a scan
|
2002-04-19 18:36:08 +02:00
|
|
|
*
|
2005-04-14 22:03:27 +02:00
|
|
|
* Note: since we scan the rules using RewriteRelRulenameIndexId, we will
|
|
|
|
* be reading the rules in name order, except possibly during
|
2006-01-05 11:07:46 +01:00
|
|
|
* emergency-recovery operations (ie, IgnoreSystemIndexes). This in turn
|
2002-04-19 18:36:08 +02:00
|
|
|
* ensures that rules will be fired in name order.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2019-01-21 19:32:19 +01:00
|
|
|
rewrite_desc = table_open(RewriteRelationId, AccessShareLock);
|
2002-04-19 18:36:08 +02:00
|
|
|
rewrite_tupdesc = RelationGetDescr(rewrite_desc);
|
|
|
|
rewrite_scan = systable_beginscan(rewrite_desc,
|
2005-04-14 22:03:27 +02:00
|
|
|
RewriteRelRulenameIndexId,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
true, NULL,
|
2002-04-19 18:36:08 +02:00
|
|
|
1, &key);
|
|
|
|
|
|
|
|
while (HeapTupleIsValid(rewrite_tuple = systable_getnext(rewrite_scan)))
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-04-19 18:36:08 +02:00
|
|
|
Form_pg_rewrite rewrite_form = (Form_pg_rewrite) GETSTRUCT(rewrite_tuple);
|
1996-07-09 08:22:35 +02:00
|
|
|
bool isnull;
|
2006-01-08 21:04:41 +01:00
|
|
|
Datum rule_datum;
|
|
|
|
char *rule_str;
|
1996-07-09 08:22:35 +02:00
|
|
|
RewriteRule *rule;
|
2022-03-22 11:28:10 +01:00
|
|
|
Oid check_as_user;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
rule = (RewriteRule *) MemoryContextAlloc(rulescxt,
|
|
|
|
sizeof(RewriteRule));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
rule->ruleId = rewrite_form->oid;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-04-18 22:01:11 +02:00
|
|
|
rule->event = rewrite_form->ev_type - '0';
|
2007-03-20 00:38:32 +01:00
|
|
|
rule->enabled = rewrite_form->ev_enabled;
|
2002-04-18 22:01:11 +02:00
|
|
|
rule->isInstead = rewrite_form->is_instead;
|
|
|
|
|
2006-01-08 21:04:41 +01:00
|
|
|
/*
|
|
|
|
* Must use heap_getattr to fetch ev_action and ev_qual. Also, the
|
|
|
|
* rule strings are often large enough to be toasted. To avoid
|
|
|
|
* leaking memory in the caller's context, do the detoasting here so
|
|
|
|
* we can free the detoasted version.
|
|
|
|
*/
|
|
|
|
rule_datum = heap_getattr(rewrite_tuple,
|
2000-01-31 05:35:57 +01:00
|
|
|
Anum_pg_rewrite_ev_action,
|
2002-04-19 18:36:08 +02:00
|
|
|
rewrite_tupdesc,
|
1996-07-09 08:22:35 +02:00
|
|
|
&isnull);
|
2000-07-31 00:14:09 +02:00
|
|
|
Assert(!isnull);
|
2008-03-25 23:42:46 +01:00
|
|
|
rule_str = TextDatumGetCString(rule_datum);
|
2001-01-06 22:53:18 +01:00
|
|
|
oldcxt = MemoryContextSwitchTo(rulescxt);
|
2006-01-08 21:04:41 +01:00
|
|
|
rule->actions = (List *) stringToNode(rule_str);
|
2000-06-30 09:04:23 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
2006-01-08 21:04:41 +01:00
|
|
|
pfree(rule_str);
|
2000-01-31 05:35:57 +01:00
|
|
|
|
2006-01-08 21:04:41 +01:00
|
|
|
rule_datum = heap_getattr(rewrite_tuple,
|
|
|
|
Anum_pg_rewrite_ev_qual,
|
|
|
|
rewrite_tupdesc,
|
|
|
|
&isnull);
|
2000-07-31 00:14:09 +02:00
|
|
|
Assert(!isnull);
|
2008-03-25 23:42:46 +01:00
|
|
|
rule_str = TextDatumGetCString(rule_datum);
|
2001-01-06 22:53:18 +01:00
|
|
|
oldcxt = MemoryContextSwitchTo(rulescxt);
|
2006-01-08 21:04:41 +01:00
|
|
|
rule->qual = (Node *) stringToNode(rule_str);
|
2000-06-30 09:04:23 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
2006-01-08 21:04:41 +01:00
|
|
|
pfree(rule_str);
|
2000-01-31 05:35:57 +01:00
|
|
|
|
2006-09-05 23:08:36 +02:00
|
|
|
/*
|
2022-03-22 11:28:10 +01:00
|
|
|
* If this is a SELECT rule defining a view, and the view has
|
|
|
|
* "security_invoker" set, we must perform all permissions checks on
|
|
|
|
* relations referred to by the rule as the invoking user.
|
|
|
|
*
|
|
|
|
* In all other cases (including non-SELECT rules on security invoker
|
|
|
|
* views), perform the permissions checks as the relation owner.
|
|
|
|
*/
|
|
|
|
if (rule->event == CMD_SELECT &&
|
|
|
|
relation->rd_rel->relkind == RELKIND_VIEW &&
|
|
|
|
RelationHasSecurityInvoker(relation))
|
|
|
|
check_as_user = InvalidOid;
|
|
|
|
else
|
|
|
|
check_as_user = relation->rd_rel->relowner;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Scan through the rule's actions and set the checkAsUser field on
|
2022-12-06 16:09:24 +01:00
|
|
|
* all RTEPermissionInfos. We have to look at the qual as well, in
|
|
|
|
* case it contains sublinks.
|
2006-09-05 23:08:36 +02:00
|
|
|
*
|
|
|
|
* The reason for doing this when the rule is loaded, rather than when
|
|
|
|
* it is stored, is that otherwise ALTER TABLE OWNER would have to
|
|
|
|
* grovel through stored rules to update checkAsUser fields. Scanning
|
|
|
|
* the rule tree during load is relatively cheap (compared to
|
|
|
|
* constructing it in the first place), so we do it here.
|
|
|
|
*/
|
2022-03-22 11:28:10 +01:00
|
|
|
setRuleCheckAsUser((Node *) rule->actions, check_as_user);
|
|
|
|
setRuleCheckAsUser(rule->qual, check_as_user);
|
2006-09-05 23:08:36 +02:00
|
|
|
|
2000-01-31 05:35:57 +01:00
|
|
|
if (numlocks >= maxlocks)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
maxlocks *= 2;
|
2000-06-30 09:04:23 +02:00
|
|
|
rules = (RewriteRule **)
|
|
|
|
repalloc(rules, sizeof(RewriteRule *) * maxlocks);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
2000-01-31 05:35:57 +01:00
|
|
|
rules[numlocks++] = rule;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* end the scan and close the attribute relation
|
|
|
|
*/
|
2002-04-19 18:36:08 +02:00
|
|
|
systable_endscan(rewrite_scan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(rewrite_desc, AccessShareLock);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2008-11-10 01:49:37 +01:00
|
|
|
/*
|
|
|
|
* there might not be any rules (if relhasrules is out-of-date)
|
|
|
|
*/
|
|
|
|
if (numlocks == 0)
|
|
|
|
{
|
|
|
|
relation->rd_rules = NULL;
|
|
|
|
relation->rd_rulescxt = NULL;
|
|
|
|
MemoryContextDelete(rulescxt);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* form a RuleLock and insert into relation
|
|
|
|
*/
|
2000-06-30 09:04:23 +02:00
|
|
|
rulelock = (RuleLock *) MemoryContextAlloc(rulescxt, sizeof(RuleLock));
|
1996-07-09 08:22:35 +02:00
|
|
|
rulelock->numLocks = numlocks;
|
|
|
|
rulelock->rules = rules;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
relation->rd_rules = rulelock;
|
2000-01-31 05:35:57 +01:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-01-31 05:35:57 +01:00
|
|
|
* equalRuleLocks
|
|
|
|
*
|
|
|
|
* Determine whether two RuleLocks are equivalent
|
|
|
|
*
|
|
|
|
* Probably this should be in the rules code someplace...
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
equalRuleLocks(RuleLock *rlock1, RuleLock *rlock2)
|
|
|
|
{
|
2002-04-19 18:36:08 +02:00
|
|
|
int i;
|
2000-01-31 05:35:57 +01:00
|
|
|
|
2002-04-19 18:36:08 +02:00
|
|
|
/*
|
|
|
|
* As of 7.3 we assume the rule ordering is repeatable, because
|
|
|
|
* RelationBuildRuleLock should read 'em in a consistent order. So just
|
|
|
|
* compare corresponding slots.
|
|
|
|
*/
|
2000-01-31 05:35:57 +01:00
|
|
|
if (rlock1 != NULL)
|
|
|
|
{
|
|
|
|
if (rlock2 == NULL)
|
|
|
|
return false;
|
|
|
|
if (rlock1->numLocks != rlock2->numLocks)
|
|
|
|
return false;
|
|
|
|
for (i = 0; i < rlock1->numLocks; i++)
|
|
|
|
{
|
|
|
|
RewriteRule *rule1 = rlock1->rules[i];
|
2002-04-19 18:36:08 +02:00
|
|
|
RewriteRule *rule2 = rlock2->rules[i];
|
|
|
|
|
|
|
|
if (rule1->ruleId != rule2->ruleId)
|
2000-01-31 05:35:57 +01:00
|
|
|
return false;
|
|
|
|
if (rule1->event != rule2->event)
|
|
|
|
return false;
|
2008-12-30 04:59:19 +01:00
|
|
|
if (rule1->enabled != rule2->enabled)
|
|
|
|
return false;
|
2000-01-31 05:35:57 +01:00
|
|
|
if (rule1->isInstead != rule2->isInstead)
|
|
|
|
return false;
|
|
|
|
if (!equal(rule1->qual, rule2->qual))
|
|
|
|
return false;
|
|
|
|
if (!equal(rule1->actions, rule2->actions))
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else if (rlock2 != NULL)
|
|
|
|
return false;
|
|
|
|
return true;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
/*
|
|
|
|
* equalPolicy
|
|
|
|
*
|
|
|
|
* Determine whether two policies are equivalent
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
equalPolicy(RowSecurityPolicy *policy1, RowSecurityPolicy *policy2)
|
|
|
|
{
|
|
|
|
int i;
|
2014-09-26 08:43:46 +02:00
|
|
|
Oid *r1,
|
|
|
|
*r2;
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
|
|
|
|
if (policy1 != NULL)
|
|
|
|
{
|
|
|
|
if (policy2 == NULL)
|
|
|
|
return false;
|
|
|
|
|
2015-01-24 22:16:22 +01:00
|
|
|
if (policy1->polcmd != policy2->polcmd)
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
return false;
|
2014-09-26 18:46:26 +02:00
|
|
|
if (policy1->hassublinks != policy2->hassublinks)
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
return false;
|
|
|
|
if (strcmp(policy1->policy_name, policy2->policy_name) != 0)
|
|
|
|
return false;
|
|
|
|
if (ARR_DIMS(policy1->roles)[0] != ARR_DIMS(policy2->roles)[0])
|
|
|
|
return false;
|
|
|
|
|
|
|
|
r1 = (Oid *) ARR_DATA_PTR(policy1->roles);
|
|
|
|
r2 = (Oid *) ARR_DATA_PTR(policy2->roles);
|
|
|
|
|
|
|
|
for (i = 0; i < ARR_DIMS(policy1->roles)[0]; i++)
|
|
|
|
{
|
|
|
|
if (r1[i] != r2[i])
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2015-04-17 22:37:11 +02:00
|
|
|
if (!equal(policy1->qual, policy2->qual))
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
return false;
|
|
|
|
if (!equal(policy1->with_check_qual, policy2->with_check_qual))
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
else if (policy2 != NULL)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* equalRSDesc
|
|
|
|
*
|
|
|
|
* Determine whether two RowSecurityDesc's are equivalent
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
|
|
|
|
{
|
|
|
|
ListCell *lc,
|
|
|
|
*rc;
|
|
|
|
|
|
|
|
if (rsdesc1 == NULL && rsdesc2 == NULL)
|
|
|
|
return true;
|
|
|
|
|
|
|
|
if ((rsdesc1 != NULL && rsdesc2 == NULL) ||
|
|
|
|
(rsdesc1 == NULL && rsdesc2 != NULL))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
if (list_length(rsdesc1->policies) != list_length(rsdesc2->policies))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
/* RelationBuildRowSecurity should build policies in order */
|
|
|
|
forboth(lc, rsdesc1->policies, rc, rsdesc2->policies)
|
|
|
|
{
|
|
|
|
RowSecurityPolicy *l = (RowSecurityPolicy *) lfirst(lc);
|
|
|
|
RowSecurityPolicy *r = (RowSecurityPolicy *) lfirst(rc);
|
|
|
|
|
|
|
|
if (!equalPolicy(l, r))
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2014-09-26 18:46:26 +02:00
|
|
|
return true;
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2008-04-16 20:23:04 +02:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* RelationBuildDesc
|
|
|
|
*
|
2010-01-12 19:12:18 +01:00
|
|
|
* Build a relation descriptor. The caller must hold at least
|
2008-04-16 20:23:04 +02:00
|
|
|
* AccessShareLock on the target relid.
|
2006-07-31 22:09:10 +02:00
|
|
|
*
|
2010-01-12 19:12:18 +01:00
|
|
|
* The new descriptor is inserted into the hash table if insertIt is true.
|
|
|
|
*
|
2006-07-31 22:09:10 +02:00
|
|
|
* Returns NULL if no pg_class row could be found for the given relid
|
|
|
|
* (suggesting we are trying to access a just-deleted relation).
|
|
|
|
* Any other error is reported via elog.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static Relation
|
2010-01-12 19:12:18 +01:00
|
|
|
RelationBuildDesc(Oid targetRelId, bool insertIt)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2021-10-24 03:36:38 +02:00
|
|
|
int in_progress_offset;
|
1996-07-09 08:22:35 +02:00
|
|
|
Relation relation;
|
|
|
|
Oid relid;
|
1999-12-30 06:05:13 +01:00
|
|
|
HeapTuple pg_class_tuple;
|
1996-07-09 08:22:35 +02:00
|
|
|
Form_pg_class relp;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
/*
|
|
|
|
* This function and its subroutines can allocate a good deal of transient
|
|
|
|
* data in CurrentMemoryContext. Traditionally we've just leaked that
|
|
|
|
* data, reasoning that the caller's context is at worst of transaction
|
|
|
|
* scope, and relcache loads shouldn't happen so often that it's essential
|
|
|
|
* to recover transient data before end of statement/transaction. However
|
2021-07-13 21:01:01 +02:00
|
|
|
* that's definitely not true when debug_discard_caches is active, and
|
|
|
|
* perhaps it's not true in other cases.
|
2021-01-06 10:15:19 +01:00
|
|
|
*
|
2021-07-13 21:01:01 +02:00
|
|
|
* When debug_discard_caches is active or when forced to by
|
2021-01-06 10:15:19 +01:00
|
|
|
* RECOVER_RELATION_BUILD_MEMORY=1, arrange to allocate the junk in a
|
|
|
|
* temporary context that we'll free before returning. Make it a child of
|
|
|
|
* caller's context so that it will get cleaned up appropriately if we
|
|
|
|
* error out partway through.
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
*/
|
2021-01-06 10:15:19 +01:00
|
|
|
#ifdef MAYBE_RECOVER_RELATION_BUILD_MEMORY
|
|
|
|
MemoryContext tmpcxt = NULL;
|
|
|
|
MemoryContext oldcxt = NULL;
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
|
2021-07-13 21:01:01 +02:00
|
|
|
if (RECOVER_RELATION_BUILD_MEMORY || debug_discard_caches > 0)
|
2021-01-06 10:15:19 +01:00
|
|
|
{
|
|
|
|
tmpcxt = AllocSetContextCreate(CurrentMemoryContext,
|
|
|
|
"RelationBuildDesc workspace",
|
|
|
|
ALLOCSET_DEFAULT_SIZES);
|
|
|
|
oldcxt = MemoryContextSwitchTo(tmpcxt);
|
|
|
|
}
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
#endif
|
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/* Register to catch invalidation messages */
|
|
|
|
if (in_progress_list_len >= in_progress_list_maxlen)
|
|
|
|
{
|
|
|
|
int allocsize;
|
|
|
|
|
|
|
|
allocsize = in_progress_list_maxlen * 2;
|
|
|
|
in_progress_list = repalloc(in_progress_list,
|
|
|
|
allocsize * sizeof(*in_progress_list));
|
|
|
|
in_progress_list_maxlen = allocsize;
|
|
|
|
}
|
|
|
|
in_progress_offset = in_progress_list_len++;
|
|
|
|
in_progress_list[in_progress_offset].reloid = targetRelId;
|
|
|
|
retry:
|
|
|
|
in_progress_list[in_progress_offset].invalidated = false;
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* find the tuple in pg_class corresponding to the given relation id
|
|
|
|
*/
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
pg_class_tuple = ScanPgRelation(targetRelId, true, false);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* if no such tuple exists, return NULL
|
|
|
|
*/
|
|
|
|
if (!HeapTupleIsValid(pg_class_tuple))
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
{
|
2021-01-06 10:15:19 +01:00
|
|
|
#ifdef MAYBE_RECOVER_RELATION_BUILD_MEMORY
|
|
|
|
if (tmpcxt)
|
|
|
|
{
|
|
|
|
/* Return to caller's context, and blow away the temporary context */
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
MemoryContextDelete(tmpcxt);
|
|
|
|
}
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
#endif
|
2021-10-24 03:36:38 +02:00
|
|
|
Assert(in_progress_offset + 1 == in_progress_list_len);
|
|
|
|
in_progress_list_len--;
|
1996-07-09 08:22:35 +02:00
|
|
|
return NULL;
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* get information from the pg_class_tuple
|
|
|
|
*/
|
|
|
|
relp = (Form_pg_class) GETSTRUCT(pg_class_tuple);
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
relid = relp->oid;
|
2010-02-07 21:48:13 +01:00
|
|
|
Assert(relid == targetRelId);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* allocate storage for the relation descriptor, and copy pg_class_tuple
|
2000-06-30 09:04:23 +02:00
|
|
|
* to relation->rd_rel.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2010-01-12 19:12:18 +01:00
|
|
|
relation = AllocateRelationDesc(relp);
|
2000-06-30 09:04:23 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* initialize the relation's relation id (relation->rd_id)
|
|
|
|
*/
|
1998-08-19 04:04:17 +02:00
|
|
|
RelationGetRelid(relation) = relid;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* Normal relations are not nailed into the cache. Since we don't flush
|
|
|
|
* new relations, it won't be new. It could be temp though.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2004-07-17 05:32:14 +02:00
|
|
|
relation->rd_refcnt = 0;
|
2004-08-28 22:31:44 +02:00
|
|
|
relation->rd_isnailed = false;
|
2004-09-16 18:58:44 +02:00
|
|
|
relation->rd_createSubid = InvalidSubTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
relation->rd_droppedSubid = InvalidSubTransactionId;
|
2010-12-13 18:34:26 +01:00
|
|
|
switch (relation->rd_rel->relpersistence)
|
2010-08-13 22:10:54 +02:00
|
|
|
{
|
2010-12-29 12:48:53 +01:00
|
|
|
case RELPERSISTENCE_UNLOGGED:
|
2010-12-13 18:34:26 +01:00
|
|
|
case RELPERSISTENCE_PERMANENT:
|
|
|
|
relation->rd_backend = InvalidBackendId;
|
2012-12-18 02:15:32 +01:00
|
|
|
relation->rd_islocaltemp = false;
|
2010-12-13 18:34:26 +01:00
|
|
|
break;
|
|
|
|
case RELPERSISTENCE_TEMP:
|
2014-08-26 03:28:19 +02:00
|
|
|
if (isTempOrTempToastNamespace(relation->rd_rel->relnamespace))
|
2012-12-18 02:15:32 +01:00
|
|
|
{
|
Improve the situation for parallel query versus temp relations.
Transmit the leader's temp-namespace state to workers. This is important
because without it, the workers do not really have the same search path
as the leader. For example, there is no good reason (and no extant code
either) to prevent a worker from executing a temp function that the
leader created previously; but as things stood it would fail to find the
temp function, and then either fail or execute the wrong function entirely.
We still prohibit a worker from creating a temp namespace on its own.
In effect, a worker can only see the session's temp namespace if the leader
had created it before starting the worker, which seems like the right
semantics.
Also, transmit the leader's BackendId to workers, and arrange for workers
to use that when determining the physical file path of a temp relation
belonging to their session. While the original intent was to prevent such
accesses entirely, there were a number of holes in that, notably in places
like dbsize.c which assume they can safely access temp rels of other
sessions anyway. We might as well get this right, as a small down payment
on someday allowing workers to access the leader's temp tables. (With
this change, directly using "MyBackendId" as a relation or buffer backend
ID is deprecated; you should use BackendIdForTempRelations() instead.
I left a couple of such uses alone though, as they're not going to be
reachable in parallel workers until we do something about localbuf.c.)
Move the thou-shalt-not-access-thy-leader's-temp-tables prohibition down
into localbuf.c, which is where it actually matters, instead of having it
in relation_open(). This amounts to recognizing that access to temp
tables' catalog entries is perfectly safe in a worker, it's only the data
in local buffers that is problematic.
Having done all that, we can get rid of the test in has_parallel_hazard()
that says that use of a temp table's rowtype is unsafe in parallel workers.
That test was unduly expensive, and if we really did need such a
prohibition, that was not even close to being a bulletproof guard for it.
(For example, any user-defined function executed in a parallel worker
might have attempted such access.)
2016-06-10 02:16:11 +02:00
|
|
|
relation->rd_backend = BackendIdForTempRelations();
|
2012-12-18 02:15:32 +01:00
|
|
|
relation->rd_islocaltemp = true;
|
|
|
|
}
|
2010-12-13 18:34:26 +01:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
2012-12-18 02:15:32 +01:00
|
|
|
* If it's a temp table, but not one of ours, we have to use
|
|
|
|
* the slow, grotty method to figure out the owning backend.
|
|
|
|
*
|
|
|
|
* Note: it's possible that rd_backend gets set to MyBackendId
|
|
|
|
* here, in case we are looking at a pg_class entry left over
|
|
|
|
* from a crashed backend that coincidentally had the same
|
|
|
|
* BackendId we're using. We should *not* consider such a
|
|
|
|
* table to be "ours"; this is why we need the separate
|
|
|
|
* rd_islocaltemp flag. The pg_class entry will get flushed
|
|
|
|
* if/when we clean out the corresponding temp table namespace
|
|
|
|
* in preparation for using it.
|
2010-12-13 18:34:26 +01:00
|
|
|
*/
|
|
|
|
relation->rd_backend =
|
|
|
|
GetTempNamespaceBackendId(relation->rd_rel->relnamespace);
|
|
|
|
Assert(relation->rd_backend != InvalidBackendId);
|
2012-12-18 02:15:32 +01:00
|
|
|
relation->rd_islocaltemp = false;
|
2010-12-13 18:34:26 +01:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
elog(ERROR, "invalid relpersistence: %c",
|
|
|
|
relation->rd_rel->relpersistence);
|
|
|
|
break;
|
2010-08-13 22:10:54 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* initialize the tuple descriptor (relation->rd_att).
|
|
|
|
*/
|
2005-04-14 22:03:27 +02:00
|
|
|
RelationBuildTupleDesc(relation);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2016-06-18 21:22:34 +02:00
|
|
|
/* foreign key data is not loaded till asked for */
|
|
|
|
relation->rd_fkeylist = NIL;
|
|
|
|
relation->rd_fkeyvalid = false;
|
|
|
|
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
/* partitioning data is not loaded till asked for */
|
|
|
|
relation->rd_partkey = NULL;
|
|
|
|
relation->rd_partkeycxt = NULL;
|
|
|
|
relation->rd_partdesc = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
relation->rd_partdesc_nodetached = NULL;
|
|
|
|
relation->rd_partdesc_nodetached_xmin = InvalidTransactionId;
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
relation->rd_pdcxt = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
relation->rd_pddcxt = NULL;
|
2019-04-13 19:22:26 +02:00
|
|
|
relation->rd_partcheck = NIL;
|
|
|
|
relation->rd_partcheckvalid = false;
|
|
|
|
relation->rd_partcheckcxt = NULL;
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
* initialize access method information
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2021-12-03 13:38:26 +01:00
|
|
|
if (relation->rd_rel->relkind == RELKIND_INDEX ||
|
|
|
|
relation->rd_rel->relkind == RELKIND_PARTITIONED_INDEX)
|
|
|
|
RelationInitIndexAccessInfo(relation);
|
|
|
|
else if (RELKIND_HAS_TABLE_AM(relation->rd_rel->relkind) ||
|
|
|
|
relation->rd_rel->relkind == RELKIND_SEQUENCE)
|
|
|
|
RelationInitTableAccessMethod(relation);
|
|
|
|
else
|
|
|
|
Assert(relation->rd_rel->relam == InvalidOid);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
/* extract reloptions if any */
|
|
|
|
RelationParseRelOptions(relation, pg_class_tuple);
|
|
|
|
|
2022-03-22 11:28:10 +01:00
|
|
|
/*
|
|
|
|
* Fetch rules and triggers that affect this relation.
|
|
|
|
*
|
|
|
|
* Note that RelationBuildRuleLock() relies on this being done after
|
|
|
|
* extracting the relation's reloptions.
|
|
|
|
*/
|
|
|
|
if (relation->rd_rel->relhasrules)
|
|
|
|
RelationBuildRuleLock(relation);
|
|
|
|
else
|
|
|
|
{
|
|
|
|
relation->rd_rules = NULL;
|
|
|
|
relation->rd_rulescxt = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (relation->rd_rel->relhastriggers)
|
|
|
|
RelationBuildTriggers(relation);
|
|
|
|
else
|
|
|
|
relation->trigdesc = NULL;
|
|
|
|
|
|
|
|
if (relation->rd_rel->relrowsecurity)
|
|
|
|
RelationBuildRowSecurity(relation);
|
|
|
|
else
|
|
|
|
relation->rd_rsdesc = NULL;
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* initialize the relation lock manager information
|
|
|
|
*/
|
|
|
|
RelationInitLockInfo(relation); /* see lmgr.c */
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
/*
|
|
|
|
* initialize physical addressing information for the relation
|
|
|
|
*/
|
|
|
|
RelationInitPhysicalAddr(relation);
|
2000-10-16 16:52:28 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* make sure relation is marked as having no open file yet */
|
2004-02-10 02:55:27 +01:00
|
|
|
relation->rd_smgr = NULL;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2006-07-02 04:23:23 +02:00
|
|
|
/*
|
|
|
|
* now we can free the memory allocated for pg_class_tuple
|
|
|
|
*/
|
|
|
|
heap_freetuple(pg_class_tuple);
|
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/*
|
|
|
|
* If an invalidation arrived mid-build, start over. Between here and the
|
|
|
|
* end of this function, don't add code that does or reasonably could read
|
|
|
|
* system catalogs. That range must be free from invalidation processing
|
|
|
|
* for the !insertIt case. For the insertIt case, RelationCacheInsert()
|
|
|
|
* will enroll this relation in ordinary relcache invalidation processing,
|
|
|
|
*/
|
|
|
|
if (in_progress_list[in_progress_offset].invalidated)
|
|
|
|
{
|
|
|
|
RelationDestroyRelation(relation, false);
|
|
|
|
goto retry;
|
|
|
|
}
|
|
|
|
Assert(in_progress_offset + 1 == in_progress_list_len);
|
|
|
|
in_progress_list_len--;
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2010-01-12 19:12:18 +01:00
|
|
|
* Insert newly created relation into relcache hash table, if requested.
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
*
|
|
|
|
* There is one scenario in which we might find a hashtable entry already
|
|
|
|
* present, even though our caller failed to find it: if the relation is a
|
|
|
|
* system catalog or index that's used during relcache load, we might have
|
|
|
|
* recursively created the same relcache entry during the preceding steps.
|
|
|
|
* So allow RelationCacheInsert to delete any already-present relcache
|
|
|
|
* entry for the same OID. The already-present entry should have refcount
|
|
|
|
* zero (else somebody forgot to close it); in the event that it doesn't,
|
|
|
|
* we'll elog a WARNING and leak the already-present entry.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2010-01-12 19:12:18 +01:00
|
|
|
if (insertIt)
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
RelationCacheInsert(relation, true);
|
1999-12-30 06:05:13 +01:00
|
|
|
|
2004-08-28 22:31:44 +02:00
|
|
|
/* It's fully valid */
|
|
|
|
relation->rd_isvalid = true;
|
|
|
|
|
2021-01-06 10:15:19 +01:00
|
|
|
#ifdef MAYBE_RECOVER_RELATION_BUILD_MEMORY
|
|
|
|
if (tmpcxt)
|
|
|
|
{
|
|
|
|
/* Return to caller's context, and blow away the temporary context */
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
MemoryContextDelete(tmpcxt);
|
|
|
|
}
|
Further reduce memory footprint of CLOBBER_CACHE_ALWAYS testing.
Some buildfarm members using CLOBBER_CACHE_ALWAYS have been having OOM
problems of late. Commit 2455ab488 addressed this problem by recovering
space transiently used within RelationBuildPartitionDesc, but it turns
out that leaves quite a lot on the table, because other subroutines of
RelationBuildDesc also leak memory like mad. Let's move the temp-context
management into RelationBuildDesc so that leakage from the other
subroutines is also recovered.
I examined this issue by arranging for postgres.c to dump the size of
MessageContext just before resetting it in each command cycle, and
then running the update.sql regression test (which is one of the two
that are seeing buildfarm OOMs) with and without CLOBBER_CACHE_ALWAYS.
Before 2455ab488, the peak space usage with CCA was as much as 250MB.
That patch got it down to ~80MB, but with this patch it's about 0.5MB,
and indeed the space usage now seems nearly indistinguishable from a
non-CCA build.
RelationBuildDesc's traditional behavior of not worrying about leaking
transient data is of many years' standing, so I'm pretty hesitant to
change that without more evidence that it'd be useful in a normal build.
(So far as I can see, non-CCA memory consumption is about the same with
or without this change, whuch if anything suggests that it isn't useful.)
Hence, configure the patch so that we recover space only when
CLOBBER_CACHE_ALWAYS or CLOBBER_CACHE_RECURSIVELY is defined. However,
that choice can be overridden at compile time, in case somebody would
like to do some performance testing and try to develop evidence for
changing that decision.
It's possible that we ought to back-patch this change, but in the
absence of back-branch OOM problems in the buildfarm, I'm not in
a hurry to do that.
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-03-15 18:46:26 +01:00
|
|
|
#endif
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
return relation;
|
|
|
|
}
|
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* Initialize the physical addressing info (RelFileLocator) for a relcache entry
|
2010-02-07 21:48:13 +01:00
|
|
|
*
|
|
|
|
* Note: at the physical level, relations in the pg_global tablespace must
|
|
|
|
* be treated as shared, even if relisshared isn't set. Hence we do not
|
|
|
|
* look at relisshared here.
|
2004-06-18 08:14:31 +02:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
RelationInitPhysicalAddr(Relation relation)
|
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelFileNumber oldnumber = relation->rd_locator.relNumber;
|
2020-09-10 03:50:24 +02:00
|
|
|
|
2019-01-04 18:51:17 +01:00
|
|
|
/* these relations kinds never have storage */
|
|
|
|
if (!RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
|
|
|
|
return;
|
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
if (relation->rd_rel->reltablespace)
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_locator.spcOid = relation->rd_rel->reltablespace;
|
2004-06-18 08:14:31 +02:00
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_locator.spcOid = MyDatabaseTableSpace;
|
|
|
|
if (relation->rd_locator.spcOid == GLOBALTABLESPACE_OID)
|
|
|
|
relation->rd_locator.dbOid = InvalidOid;
|
2004-06-18 08:14:31 +02:00
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_locator.dbOid = MyDatabaseId;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
if (relation->rd_rel->relfilenode)
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Even if we are using a decoding snapshot that doesn't represent the
|
|
|
|
* current state of the catalog we need to make sure the filenode
|
|
|
|
* points to the current file since the older file will be gone (or
|
|
|
|
* truncated). The new file will still contain older rows so lookups
|
|
|
|
* in them will work correctly. This wouldn't work correctly if
|
2017-02-06 10:33:58 +01:00
|
|
|
* rewrites were allowed to change the schema in an incompatible way,
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
* but those are prevented both on catalog tables and on user tables
|
|
|
|
* declared as additional catalog tables.
|
|
|
|
*/
|
|
|
|
if (HistoricSnapshotActive()
|
|
|
|
&& RelationIsAccessibleInLogicalDecoding(relation)
|
|
|
|
&& IsTransactionState())
|
|
|
|
{
|
|
|
|
HeapTuple phys_tuple;
|
|
|
|
Form_pg_class physrel;
|
|
|
|
|
|
|
|
phys_tuple = ScanPgRelation(RelationGetRelid(relation),
|
|
|
|
RelationGetRelid(relation) != ClassOidIndexId,
|
|
|
|
true);
|
|
|
|
if (!HeapTupleIsValid(phys_tuple))
|
|
|
|
elog(ERROR, "could not find pg_class entry for %u",
|
|
|
|
RelationGetRelid(relation));
|
|
|
|
physrel = (Form_pg_class) GETSTRUCT(phys_tuple);
|
|
|
|
|
|
|
|
relation->rd_rel->reltablespace = physrel->reltablespace;
|
|
|
|
relation->rd_rel->relfilenode = physrel->relfilenode;
|
|
|
|
heap_freetuple(phys_tuple);
|
|
|
|
}
|
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_locator.relNumber = relation->rd_rel->relfilenode;
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
}
|
2010-02-07 21:48:13 +01:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Consult the relation mapper */
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_locator.relNumber =
|
|
|
|
RelationMapOidToFilenumber(relation->rd_id,
|
|
|
|
relation->rd_rel->relisshared);
|
|
|
|
if (!RelFileNumberIsValid(relation->rd_locator.relNumber))
|
2010-02-07 21:48:13 +01:00
|
|
|
elog(ERROR, "could not find relation mapping for relation \"%s\", OID %u",
|
|
|
|
RelationGetRelationName(relation), relation->rd_id);
|
|
|
|
}
|
2020-09-10 03:50:24 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* For RelationNeedsWAL() to answer correctly on parallel workers, restore
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* rd_firstRelfilelocatorSubid. No subtransactions start or end while in
|
2020-09-10 03:50:24 +02:00
|
|
|
* parallel mode, so the specific SubTransactionId does not matter.
|
|
|
|
*/
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
if (IsParallelWorker() && oldnumber != relation->rd_locator.relNumber)
|
2020-09-10 03:50:24 +02:00
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
if (RelFileLocatorSkippingWAL(relation->rd_locator))
|
|
|
|
relation->rd_firstRelfilelocatorSubid = TopSubTransactionId;
|
2020-09-10 03:50:24 +02:00
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
2020-09-10 03:50:24 +02:00
|
|
|
}
|
2004-06-18 08:14:31 +02:00
|
|
|
}
|
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
/*
|
|
|
|
* Fill in the IndexAmRoutine for an index relation.
|
|
|
|
*
|
|
|
|
* relation's rd_amhandler and rd_indexcxt must be valid already.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
InitIndexAmRoutine(Relation relation)
|
|
|
|
{
|
|
|
|
IndexAmRoutine *cached,
|
|
|
|
*tmp;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Call the amhandler in current, short-lived memory context, just in case
|
|
|
|
* it leaks anything (it probably won't, but let's be paranoid).
|
|
|
|
*/
|
|
|
|
tmp = GetIndexAmRoutine(relation->rd_amhandler);
|
|
|
|
|
|
|
|
/* OK, now transfer the data into relation's rd_indexcxt. */
|
|
|
|
cached = (IndexAmRoutine *) MemoryContextAlloc(relation->rd_indexcxt,
|
|
|
|
sizeof(IndexAmRoutine));
|
|
|
|
memcpy(cached, tmp, sizeof(IndexAmRoutine));
|
2019-01-22 02:36:55 +01:00
|
|
|
relation->rd_indam = cached;
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
|
|
|
|
pfree(tmp);
|
|
|
|
}
|
|
|
|
|
2001-10-07 01:21:45 +02:00
|
|
|
/*
|
|
|
|
* Initialize index-access-method support data for an index relation
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationInitIndexAccessInfo(Relation relation)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
HeapTuple tuple;
|
|
|
|
Form_pg_am aform;
|
2011-02-08 22:04:18 +01:00
|
|
|
Datum indcollDatum;
|
2005-03-29 02:17:27 +02:00
|
|
|
Datum indclassDatum;
|
2007-01-09 03:14:16 +01:00
|
|
|
Datum indoptionDatum;
|
2005-03-29 02:17:27 +02:00
|
|
|
bool isnull;
|
2011-02-08 22:04:18 +01:00
|
|
|
oidvector *indcoll;
|
2006-12-23 01:43:13 +01:00
|
|
|
oidvector *indclass;
|
2007-01-09 03:14:16 +01:00
|
|
|
int2vector *indoption;
|
2001-10-07 01:21:45 +02:00
|
|
|
MemoryContext indexcxt;
|
2003-05-28 18:04:02 +02:00
|
|
|
MemoryContext oldcontext;
|
2018-04-07 22:00:39 +02:00
|
|
|
int indnatts;
|
|
|
|
int indnkeyatts;
|
2001-06-01 04:41:36 +02:00
|
|
|
uint16 amsupport;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
2003-05-28 18:04:02 +02:00
|
|
|
* Make a copy of the pg_index entry for the index. Since pg_index
|
|
|
|
* contains variable-length and possibly-null fields, we have to do this
|
|
|
|
* honestly rather than just treating it as a Form_pg_index struct.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
2010-02-14 19:42:19 +01:00
|
|
|
tuple = SearchSysCache1(INDEXRELID,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
2002-02-19 21:11:20 +01:00
|
|
|
if (!HeapTupleIsValid(tuple))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "cache lookup failed for index %u",
|
2002-02-19 21:11:20 +01:00
|
|
|
RelationGetRelid(relation));
|
2003-05-28 18:04:02 +02:00
|
|
|
oldcontext = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
relation->rd_indextuple = heap_copytuple(tuple);
|
|
|
|
relation->rd_index = (Form_pg_index) GETSTRUCT(relation->rd_indextuple);
|
|
|
|
MemoryContextSwitchTo(oldcontext);
|
2002-02-19 21:11:20 +01:00
|
|
|
ReleaseSysCache(tuple);
|
|
|
|
|
|
|
|
/*
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
* Look up the index's access method, save the OID of its handler function
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
2021-12-03 13:38:26 +01:00
|
|
|
Assert(relation->rd_rel->relam != InvalidOid);
|
2010-02-14 19:42:19 +01:00
|
|
|
tuple = SearchSysCache1(AMOID, ObjectIdGetDatum(relation->rd_rel->relam));
|
2002-02-19 21:11:20 +01:00
|
|
|
if (!HeapTupleIsValid(tuple))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "cache lookup failed for access method %u",
|
2002-02-19 21:11:20 +01:00
|
|
|
relation->rd_rel->relam);
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
aform = (Form_pg_am) GETSTRUCT(tuple);
|
|
|
|
relation->rd_amhandler = aform->amhandler;
|
2002-02-19 21:11:20 +01:00
|
|
|
ReleaseSysCache(tuple);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
indnatts = RelationGetNumberOfAttributes(relation);
|
|
|
|
if (indnatts != IndexRelationGetNumberOfAttributes(relation))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "relnatts disagrees with indnatts for index %u",
|
2003-05-28 18:04:02 +02:00
|
|
|
RelationGetRelid(relation));
|
2018-04-07 22:00:39 +02:00
|
|
|
indnkeyatts = IndexRelationGetNumberOfKeyAttributes(relation);
|
2001-06-01 04:41:36 +02:00
|
|
|
|
2001-10-07 01:21:45 +02:00
|
|
|
/*
|
|
|
|
* Make the private context to hold index access info. The reason we need
|
|
|
|
* a context, and not just a couple of pallocs, is so that we won't leak
|
|
|
|
* any subsidiary info attached to fmgr lookup records.
|
|
|
|
*/
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
indexcxt = AllocSetContextCreate(CacheMemoryContext,
|
|
|
|
"index info",
|
|
|
|
ALLOCSET_SMALL_SIZES);
|
2001-10-07 01:21:45 +02:00
|
|
|
relation->rd_indexcxt = indexcxt;
|
2018-04-06 18:10:00 +02:00
|
|
|
MemoryContextCopyAndSetIdentifier(indexcxt,
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
RelationGetRelationName(relation));
|
2001-10-07 01:21:45 +02:00
|
|
|
|
|
|
|
/*
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
* Now we can fetch the index AM's API struct
|
2001-10-07 01:21:45 +02:00
|
|
|
*/
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
InitIndexAmRoutine(relation);
|
2005-05-28 01:31:21 +02:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
/*
|
2018-04-07 22:00:39 +02:00
|
|
|
* Allocate arrays to hold data. Opclasses are not used for included
|
|
|
|
* columns, so allocate them for indnkeyatts only.
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
*/
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_opfamily = (Oid *)
|
2018-04-07 22:00:39 +02:00
|
|
|
MemoryContextAllocZero(indexcxt, indnkeyatts * sizeof(Oid));
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_opcintype = (Oid *)
|
2018-04-07 22:00:39 +02:00
|
|
|
MemoryContextAllocZero(indexcxt, indnkeyatts * sizeof(Oid));
|
2006-12-23 01:43:13 +01:00
|
|
|
|
2019-01-22 02:36:55 +01:00
|
|
|
amsupport = relation->rd_indam->amsupport;
|
2001-06-01 04:41:36 +02:00
|
|
|
if (amsupport > 0)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2020-03-30 22:40:22 +02:00
|
|
|
int nsupport = indnatts * amsupport;
|
2001-10-07 01:21:45 +02:00
|
|
|
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_support = (RegProcedure *)
|
2003-11-09 22:30:38 +01:00
|
|
|
MemoryContextAllocZero(indexcxt, nsupport * sizeof(RegProcedure));
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_supportinfo = (FmgrInfo *)
|
2003-11-09 22:30:38 +01:00
|
|
|
MemoryContextAllocZero(indexcxt, nsupport * sizeof(FmgrInfo));
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
else
|
2001-10-07 01:21:45 +02:00
|
|
|
{
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_support = NULL;
|
|
|
|
relation->rd_supportinfo = NULL;
|
2001-10-07 01:21:45 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2011-02-08 22:04:18 +01:00
|
|
|
relation->rd_indcollation = (Oid *)
|
2018-04-12 15:37:22 +02:00
|
|
|
MemoryContextAllocZero(indexcxt, indnkeyatts * sizeof(Oid));
|
2011-02-08 22:04:18 +01:00
|
|
|
|
2007-01-09 03:14:16 +01:00
|
|
|
relation->rd_indoption = (int16 *)
|
2018-04-12 15:37:22 +02:00
|
|
|
MemoryContextAllocZero(indexcxt, indnkeyatts * sizeof(int16));
|
2007-01-09 03:14:16 +01:00
|
|
|
|
2011-02-08 22:04:18 +01:00
|
|
|
/*
|
|
|
|
* indcollation cannot be referenced directly through the C struct,
|
|
|
|
* because it comes after the variable-width indkey field. Must extract
|
|
|
|
* the datum the hard way...
|
|
|
|
*/
|
|
|
|
indcollDatum = fastgetattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indcollation,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
|
|
|
Assert(!isnull);
|
|
|
|
indcoll = (oidvector *) DatumGetPointer(indcollDatum);
|
2018-04-12 15:37:22 +02:00
|
|
|
memcpy(relation->rd_indcollation, indcoll->values, indnkeyatts * sizeof(Oid));
|
2011-02-08 22:04:18 +01:00
|
|
|
|
2006-12-23 01:43:13 +01:00
|
|
|
/*
|
|
|
|
* indclass cannot be referenced directly through the C struct, because it
|
|
|
|
* comes after the variable-width indkey field. Must extract the datum
|
|
|
|
* the hard way...
|
|
|
|
*/
|
|
|
|
indclassDatum = fastgetattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indclass,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
|
|
|
Assert(!isnull);
|
|
|
|
indclass = (oidvector *) DatumGetPointer(indclassDatum);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2001-10-07 01:21:45 +02:00
|
|
|
/*
|
2010-11-29 18:29:42 +01:00
|
|
|
* Fill the support procedure OID array, as well as the info about
|
|
|
|
* opfamilies and opclass input types. (aminfo and supportinfo are left
|
|
|
|
* as zeroes, and are filled on-the-fly when used)
|
2001-10-07 01:21:45 +02:00
|
|
|
*/
|
2010-11-29 18:29:42 +01:00
|
|
|
IndexSupportInitialize(indclass, relation->rd_support,
|
2006-12-23 01:43:13 +01:00
|
|
|
relation->rd_opfamily, relation->rd_opcintype,
|
2018-04-07 22:00:39 +02:00
|
|
|
amsupport, indnkeyatts);
|
2003-05-28 18:04:02 +02:00
|
|
|
|
2007-01-09 03:14:16 +01:00
|
|
|
/*
|
|
|
|
* Similarly extract indoption and copy it to the cache entry
|
|
|
|
*/
|
|
|
|
indoptionDatum = fastgetattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indoption,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
|
|
|
Assert(!isnull);
|
|
|
|
indoption = (int2vector *) DatumGetPointer(indoptionDatum);
|
2018-04-12 15:37:22 +02:00
|
|
|
memcpy(relation->rd_indoption, indoption->values, indnkeyatts * sizeof(int16));
|
2007-01-09 03:14:16 +01:00
|
|
|
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
(void) RelationGetIndexAttOptions(relation, false);
|
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
/*
|
2009-12-07 06:22:23 +01:00
|
|
|
* expressions, predicate, exclusion caches will be filled later
|
2003-05-28 18:04:02 +02:00
|
|
|
*/
|
|
|
|
relation->rd_indexprs = NIL;
|
|
|
|
relation->rd_indpred = NIL;
|
2009-12-07 06:22:23 +01:00
|
|
|
relation->rd_exclops = NULL;
|
|
|
|
relation->rd_exclprocs = NULL;
|
|
|
|
relation->rd_exclstrats = NULL;
|
2006-04-26 00:46:05 +02:00
|
|
|
relation->rd_amcache = NULL;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* IndexSupportInitialize
|
2003-11-12 22:15:59 +01:00
|
|
|
* Initializes an index's cached opclass information,
|
2005-03-29 02:17:27 +02:00
|
|
|
* given the index's pg_index.indclass entry.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2010-11-29 18:29:42 +01:00
|
|
|
* Data is returned into *indexSupport, *opFamily, and *opcInType,
|
|
|
|
* which are arrays allocated by the caller.
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2010-11-29 18:29:42 +01:00
|
|
|
* The caller also passes maxSupportNumber and maxAttributeNumber, since these
|
|
|
|
* indicate the size of the arrays it has allocated --- but in practice these
|
|
|
|
* numbers must always match those obtainable from the system catalog entries
|
|
|
|
* for the index and access method.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
2005-03-29 02:17:27 +02:00
|
|
|
IndexSupportInitialize(oidvector *indclass,
|
2002-02-19 21:11:20 +01:00
|
|
|
RegProcedure *indexSupport,
|
2006-12-23 01:43:13 +01:00
|
|
|
Oid *opFamily,
|
|
|
|
Oid *opcInType,
|
2002-02-19 21:11:20 +01:00
|
|
|
StrategyNumber maxSupportNumber,
|
|
|
|
AttrNumber maxAttributeNumber)
|
|
|
|
{
|
|
|
|
int attIndex;
|
|
|
|
|
|
|
|
for (attIndex = 0; attIndex < maxAttributeNumber; attIndex++)
|
|
|
|
{
|
|
|
|
OpClassCacheEnt *opcentry;
|
|
|
|
|
2005-03-29 02:17:27 +02:00
|
|
|
if (!OidIsValid(indclass->values[attIndex]))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "bogus pg_index tuple");
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/* look up the info for this opclass, using a cache */
|
2005-03-29 02:17:27 +02:00
|
|
|
opcentry = LookupOpclassInfo(indclass->values[attIndex],
|
2002-02-19 21:11:20 +01:00
|
|
|
maxSupportNumber);
|
|
|
|
|
2003-11-09 22:30:38 +01:00
|
|
|
/* copy cached data into relcache entry */
|
2006-12-23 01:43:13 +01:00
|
|
|
opFamily[attIndex] = opcentry->opcfamily;
|
|
|
|
opcInType[attIndex] = opcentry->opcintype;
|
2002-02-19 21:11:20 +01:00
|
|
|
if (maxSupportNumber > 0)
|
2020-03-30 22:40:22 +02:00
|
|
|
memcpy(&indexSupport[attIndex * maxSupportNumber],
|
2003-11-09 22:30:38 +01:00
|
|
|
opcentry->supportProcs,
|
2020-03-30 22:40:22 +02:00
|
|
|
maxSupportNumber * sizeof(RegProcedure));
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* LookupOpclassInfo
|
|
|
|
*
|
|
|
|
* This routine maintains a per-opclass cache of the information needed
|
|
|
|
* by IndexSupportInitialize(). This is more efficient than relying on
|
|
|
|
* the catalog cache, because we can load all the info about a particular
|
2010-11-29 18:29:42 +01:00
|
|
|
* opclass in a single indexscan of pg_amproc.
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2010-11-29 18:29:42 +01:00
|
|
|
* The information from pg_am about expected range of support function
|
2002-02-19 21:11:20 +01:00
|
|
|
* numbers is passed in, rather than being looked up, mainly because the
|
|
|
|
* caller will have it already.
|
|
|
|
*
|
2007-11-28 21:44:26 +01:00
|
|
|
* Note there is no provision for flushing the cache. This is OK at the
|
|
|
|
* moment because there is no way to ALTER any interesting properties of an
|
|
|
|
* existing opclass --- all you can do is drop it, which will result in
|
|
|
|
* a useless but harmless dead entry in the cache. To support altering
|
|
|
|
* opclass membership (not the same as opfamily membership!), we'd need to
|
|
|
|
* be able to flush this cache as well as the contents of relcache entries
|
|
|
|
* for indexes.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
static OpClassCacheEnt *
|
|
|
|
LookupOpclassInfo(Oid operatorClassOid,
|
|
|
|
StrategyNumber numSupport)
|
|
|
|
{
|
|
|
|
OpClassCacheEnt *opcentry;
|
|
|
|
bool found;
|
2003-11-12 22:15:59 +01:00
|
|
|
Relation rel;
|
|
|
|
SysScanDesc scan;
|
2006-12-23 01:43:13 +01:00
|
|
|
ScanKeyData skey[3];
|
2002-02-19 21:11:20 +01:00
|
|
|
HeapTuple htup;
|
|
|
|
bool indexOK;
|
|
|
|
|
|
|
|
if (OpClassCache == NULL)
|
|
|
|
{
|
|
|
|
/* First time through: initialize the opclass cache */
|
|
|
|
HASHCTL ctl;
|
|
|
|
|
2021-07-05 22:51:57 +02:00
|
|
|
/* Also make sure CacheMemoryContext exists */
|
|
|
|
if (!CacheMemoryContext)
|
|
|
|
CreateCacheMemoryContext();
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
ctl.keysize = sizeof(Oid);
|
|
|
|
ctl.entrysize = sizeof(OpClassCacheEnt);
|
|
|
|
OpClassCache = hash_create("Operator class cache", 64,
|
Improve hash_create's API for selecting simple-binary-key hash functions.
Previously, if you wanted anything besides C-string hash keys, you had to
specify a custom hashing function to hash_create(). Nearly all such
callers were specifying tag_hash or oid_hash; which is tedious, and rather
error-prone, since a caller could easily miss the opportunity to optimize
by using hash_uint32 when appropriate. Replace this with a design whereby
callers using simple binary-data keys just specify HASH_BLOBS and don't
need to mess with specific support functions. hash_create() itself will
take care of optimizing when the key size is four bytes.
This nets out saving a few hundred bytes of code space, and offers
a measurable performance improvement in tidbitmap.c (which was not
exploiting the opportunity to use hash_uint32 for its 4-byte keys).
There might be some wins elsewhere too, I didn't analyze closely.
In future we could look into offering a similar optimized hashing function
for 8-byte keys. Under this design that could be done in a centralized
and machine-independent fashion, whereas getting it right for keys of
platform-dependent sizes would've been notationally painful before.
For the moment, the old way still works fine, so as not to break source
code compatibility for loadable modules. Eventually we might want to
remove tag_hash and friends from the exported API altogether, since there's
no real need for them to be explicitly referenced from outside dynahash.c.
Teodor Sigaev and Tom Lane
2014-12-18 19:36:29 +01:00
|
|
|
&ctl, HASH_ELEM | HASH_BLOBS);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
opcentry = (OpClassCacheEnt *) hash_search(OpClassCache,
|
2023-02-06 09:05:20 +01:00
|
|
|
&operatorClassOid,
|
2002-02-19 21:11:20 +01:00
|
|
|
HASH_ENTER, &found);
|
|
|
|
|
2007-11-28 21:44:26 +01:00
|
|
|
if (!found)
|
|
|
|
{
|
2021-07-05 22:51:57 +02:00
|
|
|
/* Initialize new entry */
|
2007-11-28 21:44:26 +01:00
|
|
|
opcentry->valid = false; /* until known OK */
|
|
|
|
opcentry->numSupport = numSupport;
|
2021-07-05 22:51:57 +02:00
|
|
|
opcentry->supportProcs = NULL; /* filled below */
|
2007-11-28 21:44:26 +01:00
|
|
|
}
|
|
|
|
else
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
|
|
|
Assert(numSupport == opcentry->numSupport);
|
|
|
|
}
|
|
|
|
|
2007-11-28 21:44:26 +01:00
|
|
|
/*
|
2021-07-05 22:51:57 +02:00
|
|
|
* When aggressively testing cache-flush hazards, we disable the operator
|
|
|
|
* class cache and force reloading of the info on each call. This models
|
|
|
|
* no real-world behavior, since the cache entries are never invalidated
|
|
|
|
* otherwise. However it can be helpful for detecting bugs in the cache
|
|
|
|
* loading logic itself, such as reliance on a non-nailed index. Given
|
|
|
|
* the limited use-case and the fact that this adds a great deal of
|
2021-07-13 21:01:01 +02:00
|
|
|
* expense, we enable it only for high values of debug_discard_caches.
|
2007-11-28 21:44:26 +01:00
|
|
|
*/
|
2021-07-13 21:01:01 +02:00
|
|
|
#ifdef DISCARD_CACHES_ENABLED
|
|
|
|
if (debug_discard_caches > 2)
|
2021-01-06 10:15:19 +01:00
|
|
|
opcentry->valid = false;
|
2007-11-28 21:44:26 +01:00
|
|
|
#endif
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2007-11-28 21:44:26 +01:00
|
|
|
if (opcentry->valid)
|
|
|
|
return opcentry;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
2021-07-05 22:51:57 +02:00
|
|
|
* Need to fill in new entry. First allocate space, unless we already did
|
|
|
|
* so in some previous attempt.
|
|
|
|
*/
|
|
|
|
if (opcentry->supportProcs == NULL && numSupport > 0)
|
|
|
|
opcentry->supportProcs = (RegProcedure *)
|
|
|
|
MemoryContextAllocZero(CacheMemoryContext,
|
|
|
|
numSupport * sizeof(RegProcedure));
|
|
|
|
|
|
|
|
/*
|
2003-11-12 22:15:59 +01:00
|
|
|
* To avoid infinite recursion during startup, force heap scans if we're
|
2002-02-19 21:11:20 +01:00
|
|
|
* looking up info for the opclasses used by the indexes we would like to
|
|
|
|
* reference here.
|
|
|
|
*/
|
|
|
|
indexOK = criticalRelcachesBuilt ||
|
|
|
|
(operatorClassOid != OID_BTREE_OPS_OID &&
|
|
|
|
operatorClassOid != INT2_BTREE_OPS_OID);
|
|
|
|
|
2006-12-23 01:43:13 +01:00
|
|
|
/*
|
|
|
|
* We have to fetch the pg_opclass row to determine its opfamily and
|
2010-11-29 18:29:42 +01:00
|
|
|
* opcintype, which are needed to look up related operators and functions.
|
2006-12-23 01:43:13 +01:00
|
|
|
* It'd be convenient to use the syscache here, but that probably doesn't
|
|
|
|
* work while bootstrapping.
|
|
|
|
*/
|
|
|
|
ScanKeyInit(&skey[0],
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Anum_pg_opclass_oid,
|
2006-12-23 01:43:13 +01:00
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(operatorClassOid));
|
2019-01-21 19:32:19 +01:00
|
|
|
rel = table_open(OperatorClassRelationId, AccessShareLock);
|
2006-12-23 01:43:13 +01:00
|
|
|
scan = systable_beginscan(rel, OpclassOidIndexId, indexOK,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 1, skey);
|
2006-12-23 01:43:13 +01:00
|
|
|
|
|
|
|
if (HeapTupleIsValid(htup = systable_getnext(scan)))
|
|
|
|
{
|
|
|
|
Form_pg_opclass opclassform = (Form_pg_opclass) GETSTRUCT(htup);
|
|
|
|
|
|
|
|
opcentry->opcfamily = opclassform->opcfamily;
|
|
|
|
opcentry->opcintype = opclassform->opcintype;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
elog(ERROR, "could not find tuple for opclass %u", operatorClassOid);
|
|
|
|
|
|
|
|
systable_endscan(scan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(rel, AccessShareLock);
|
2006-12-23 01:43:13 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
2003-11-12 22:15:59 +01:00
|
|
|
* Scan pg_amproc to obtain support procs for the opclass. We only fetch
|
2006-12-23 01:43:13 +01:00
|
|
|
* the default ones (those with lefttype = righttype = opcintype).
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
if (numSupport > 0)
|
|
|
|
{
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey[0],
|
2006-12-23 01:43:13 +01:00
|
|
|
Anum_pg_amproc_amprocfamily,
|
2003-11-12 22:15:59 +01:00
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
2006-12-23 01:43:13 +01:00
|
|
|
ObjectIdGetDatum(opcentry->opcfamily));
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey[1],
|
2006-12-23 01:43:13 +01:00
|
|
|
Anum_pg_amproc_amproclefttype,
|
2003-11-12 22:15:59 +01:00
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
2006-12-23 01:43:13 +01:00
|
|
|
ObjectIdGetDatum(opcentry->opcintype));
|
|
|
|
ScanKeyInit(&skey[2],
|
|
|
|
Anum_pg_amproc_amprocrighttype,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(opcentry->opcintype));
|
2019-01-21 19:32:19 +01:00
|
|
|
rel = table_open(AccessMethodProcedureRelationId, AccessShareLock);
|
2005-04-14 22:03:27 +02:00
|
|
|
scan = systable_beginscan(rel, AccessMethodProcedureIndexId, indexOK,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 3, skey);
|
2003-11-12 22:15:59 +01:00
|
|
|
|
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(scan)))
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
|
|
|
Form_pg_amproc amprocform = (Form_pg_amproc) GETSTRUCT(htup);
|
|
|
|
|
2020-03-30 22:40:22 +02:00
|
|
|
if (amprocform->amprocnum <= 0 ||
|
2002-02-19 21:11:20 +01:00
|
|
|
(StrategyNumber) amprocform->amprocnum > numSupport)
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "invalid amproc number %d for opclass %u",
|
2002-02-19 21:11:20 +01:00
|
|
|
amprocform->amprocnum, operatorClassOid);
|
|
|
|
|
2020-03-30 22:40:22 +02:00
|
|
|
opcentry->supportProcs[amprocform->amprocnum - 1] =
|
|
|
|
amprocform->amproc;
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
2003-11-12 22:15:59 +01:00
|
|
|
systable_endscan(scan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(rel, AccessShareLock);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
opcentry->valid = true;
|
|
|
|
return opcentry;
|
|
|
|
}
|
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
/*
|
|
|
|
* Fill in the TableAmRoutine for a relation
|
|
|
|
*
|
|
|
|
* relation's rd_amhandler must be valid already.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
InitTableAmRoutine(Relation relation)
|
|
|
|
{
|
|
|
|
relation->rd_tableam = GetTableAmRoutine(relation->rd_amhandler);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-03-11 03:46:14 +01:00
|
|
|
* Initialize table access method support for a table like relation
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationInitTableAccessMethod(Relation relation)
|
|
|
|
{
|
|
|
|
HeapTuple tuple;
|
|
|
|
Form_pg_am aform;
|
|
|
|
|
|
|
|
if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Sequences are currently accessed like heap tables, but it doesn't
|
|
|
|
* seem prudent to show that in the catalog. So just overwrite it
|
|
|
|
* here.
|
|
|
|
*/
|
2021-12-03 13:38:26 +01:00
|
|
|
Assert(relation->rd_rel->relam == InvalidOid);
|
2020-10-28 17:18:45 +01:00
|
|
|
relation->rd_amhandler = F_HEAP_TABLEAM_HANDLER;
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
}
|
|
|
|
else if (IsCatalogRelation(relation))
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Avoid doing a syscache lookup for catalog tables.
|
|
|
|
*/
|
|
|
|
Assert(relation->rd_rel->relam == HEAP_TABLE_AM_OID);
|
2020-10-28 17:18:45 +01:00
|
|
|
relation->rd_amhandler = F_HEAP_TABLEAM_HANDLER;
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Look up the table access method, save the OID of its handler
|
|
|
|
* function.
|
|
|
|
*/
|
|
|
|
Assert(relation->rd_rel->relam != InvalidOid);
|
|
|
|
tuple = SearchSysCache1(AMOID,
|
|
|
|
ObjectIdGetDatum(relation->rd_rel->relam));
|
|
|
|
if (!HeapTupleIsValid(tuple))
|
|
|
|
elog(ERROR, "cache lookup failed for access method %u",
|
|
|
|
relation->rd_rel->relam);
|
|
|
|
aform = (Form_pg_am) GETSTRUCT(tuple);
|
|
|
|
relation->rd_amhandler = aform->amhandler;
|
|
|
|
ReleaseSysCache(tuple);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Now we can fetch the table AM's API struct
|
|
|
|
*/
|
|
|
|
InitTableAmRoutine(relation);
|
|
|
|
}
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* formrdesc
|
|
|
|
*
|
2009-08-12 22:53:31 +02:00
|
|
|
* This is a special cut-down version of RelationBuildDesc(),
|
|
|
|
* used while initializing the relcache.
|
2002-02-19 21:11:20 +01:00
|
|
|
* The relation descriptor is built just from the supplied parameters,
|
2000-11-10 01:33:12 +01:00
|
|
|
* without actually looking at any system table entries. We cheat
|
|
|
|
* quite a lot since we only need to work for a few basic system
|
2002-04-27 23:24:34 +02:00
|
|
|
* catalogs.
|
|
|
|
*
|
2018-03-01 18:03:29 +01:00
|
|
|
* The catalogs this is used for can't have constraints (except attnotnull),
|
2002-11-15 18:18:49 +01:00
|
|
|
* default values, rules, or triggers, since we don't cope with any of that.
|
2009-08-12 22:53:31 +02:00
|
|
|
* (Well, actually, this only matters for properties that need to be valid
|
|
|
|
* during bootstrap or before RelationCacheInitializePhase3 runs, and none of
|
|
|
|
* these properties matter then...)
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2000-06-30 09:04:23 +02:00
|
|
|
* NOTE: we assume we are already switched into CacheMemoryContext.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
static void
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc(const char *relationName, Oid relationReltype,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
bool isshared,
|
2009-09-27 01:08:22 +02:00
|
|
|
int natts, const FormData_pg_attribute *attrs)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
Relation relation;
|
2000-06-30 09:04:23 +02:00
|
|
|
int i;
|
2002-11-15 18:18:49 +01:00
|
|
|
bool has_not_null;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2004-12-12 06:07:50 +01:00
|
|
|
* allocate new relation desc, clear all fields of reldesc
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2002-11-13 01:39:48 +01:00
|
|
|
relation = (Relation) palloc0(sizeof(RelationData));
|
2001-06-28 01:31:40 +02:00
|
|
|
|
|
|
|
/* make sure relation is marked as having no open file yet */
|
2004-02-10 02:55:27 +01:00
|
|
|
relation->rd_smgr = NULL;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2004-07-17 05:32:14 +02:00
|
|
|
* initialize reference count: 1 because it is nailed in cache
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2004-07-17 05:32:14 +02:00
|
|
|
relation->rd_refcnt = 1;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2002-08-06 04:36:35 +02:00
|
|
|
* all entries built with this routine are nailed-in-cache; none are for
|
|
|
|
* new or temp relations.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2004-08-28 22:31:44 +02:00
|
|
|
relation->rd_isnailed = true;
|
2004-09-16 18:58:44 +02:00
|
|
|
relation->rd_createSubid = InvalidSubTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
relation->rd_droppedSubid = InvalidSubTransactionId;
|
2010-08-13 22:10:54 +02:00
|
|
|
relation->rd_backend = InvalidBackendId;
|
2012-12-18 02:15:32 +01:00
|
|
|
relation->rd_islocaltemp = false;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-08-06 06:40:08 +02:00
|
|
|
* initialize relation tuple form
|
|
|
|
*
|
|
|
|
* The data we insert here is pretty incomplete/bogus, but it'll serve to
|
2009-08-12 22:53:31 +02:00
|
|
|
* get us launched. RelationCacheInitializePhase3() will read the real
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
* data from pg_class and replace what we've done here. Note in
|
|
|
|
* particular that relowner is left as zero; this cues
|
|
|
|
* RelationCacheInitializePhase3 that the real data isn't there yet.
|
2000-06-30 09:04:23 +02:00
|
|
|
*/
|
2002-11-13 01:39:48 +01:00
|
|
|
relation->rd_rel = (Form_pg_class) palloc0(CLASS_TUPLE_SIZE);
|
2000-08-06 06:40:08 +02:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
namestrcpy(&relation->rd_rel->relname, relationName);
|
|
|
|
relation->rd_rel->relnamespace = PG_CATALOG_NAMESPACE;
|
2009-09-27 01:08:22 +02:00
|
|
|
relation->rd_rel->reltype = relationReltype;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2001-06-12 07:55:50 +02:00
|
|
|
* It's important to distinguish between shared and non-shared relations,
|
2009-08-12 22:53:31 +02:00
|
|
|
* even at bootstrap time, to make sure we know where they are stored.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2009-08-12 22:53:31 +02:00
|
|
|
relation->rd_rel->relisshared = isshared;
|
|
|
|
if (isshared)
|
|
|
|
relation->rd_rel->reltablespace = GLOBALTABLESPACE_OID;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2010-12-13 18:34:26 +01:00
|
|
|
/* formrdesc is used only for permanent relations */
|
|
|
|
relation->rd_rel->relpersistence = RELPERSISTENCE_PERMANENT;
|
2009-03-31 19:59:56 +02:00
|
|
|
|
2013-05-06 19:26:51 +02:00
|
|
|
/* ... and they're always populated, too */
|
|
|
|
relation->rd_rel->relispopulated = true;
|
|
|
|
|
2013-11-08 18:30:43 +01:00
|
|
|
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
relation->rd_rel->relpages = 0;
|
Redefine pg_class.reltuples to be -1 before the first VACUUM or ANALYZE.
Historically, we've considered the state with relpages and reltuples
both zero as indicating that we do not know the table's tuple density.
This is problematic because it's impossible to distinguish "never yet
vacuumed" from "vacuumed and seen to be empty". In particular, a user
cannot use VACUUM or ANALYZE to override the planner's normal heuristic
that an empty table should not be believed to be empty because it is
probably about to get populated. That heuristic is a good safety
measure, so I don't care to abandon it, but there should be a way to
override it if the table is indeed intended to stay empty.
Hence, represent the initial state of ignorance by setting reltuples
to -1 (relpages is still set to zero), and apply the minimum-ten-pages
heuristic only when reltuples is still -1. If the table is empty,
VACUUM or ANALYZE (but not CREATE INDEX) will override that to
reltuples = relpages = 0, and then we'll plan on that basis.
This requires a bunch of fiddly little changes, but we can get rid of
some ugly kluges that were formerly needed to maintain the old definition.
One notable point is that FDWs' GetForeignRelSize methods will see
baserel->tuples = -1 when no ANALYZE has been done on the foreign table.
That seems like a net improvement, since those methods were formerly
also in the dark about what baserel->tuples = 0 really meant. Still,
it is an API change.
I bumped catversion because code predating this change would get confused
by seeing reltuples = -1.
Discussion: https://postgr.es/m/F02298E0-6EF4-49A1-BCB6-C484794D9ACC@thebuild.com
2020-08-30 18:21:51 +02:00
|
|
|
relation->rd_rel->reltuples = -1;
|
2011-10-14 23:23:01 +02:00
|
|
|
relation->rd_rel->relallvisible = 0;
|
1996-07-09 08:22:35 +02:00
|
|
|
relation->rd_rel->relkind = RELKIND_RELATION;
|
2000-06-30 09:04:23 +02:00
|
|
|
relation->rd_rel->relnatts = (int16) natts;
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
relation->rd_rel->relam = HEAP_TABLE_AM_OID;
|
2000-08-06 06:40:08 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-08-06 06:40:08 +02:00
|
|
|
* initialize attribute tuple form
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
|
|
|
* Unlike the case with the relation tuple, this data had better be right
|
2010-04-21 01:48:47 +02:00
|
|
|
* because it will never be replaced. The data comes from
|
|
|
|
* src/include/catalog/ headers via genbki.pl.
|
2000-08-06 06:40:08 +02:00
|
|
|
*/
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
relation->rd_att = CreateTemplateTupleDesc(natts);
|
2006-06-16 20:42:24 +02:00
|
|
|
relation->rd_att->tdrefcount = 1; /* mark as refcounted */
|
|
|
|
|
2009-09-27 01:08:22 +02:00
|
|
|
relation->rd_att->tdtypeid = relationReltype;
|
Don't create pg_type entries for sequences or toast tables.
Commit f7f70d5e2 left one inconsistency behind: we're still creating
pg_type entries for the composite types of sequences and toast tables,
but not arrays over those composites. But there seems precious little
reason to have named composite types for toast tables, and not much more
to have them for sequences (especially given the thought that sequences
may someday not be standalone relations at all).
So, let's close that inconsistency by removing these composite types,
rather than adding arrays for them. This buys back a little bit of
the initial pg_type bloat added by the previous patch, and could be
a significant savings in a large database with many toast tables.
Aside from a small logic rearrangement in heap_create_with_catalog,
this patch mostly needs to clean up some places that were assuming that
pg_class.reltype always has a valid value. Those are really pre-existing
bugs, given that it's documented otherwise; notably, the plpgsql changes
fix code that gives "cache lookup failed for type 0" on indexes today.
But none of these seem interesting enough to back-patch.
Also, remove the pg_dump/pg_upgrade infrastructure for propagating
a toast table's pg_type OID into the new database, since we no longer
need that.
Discussion: https://postgr.es/m/761F1389-C6A8-4C15-80CE-950C961F5341@gmail.com
2020-07-07 21:43:22 +02:00
|
|
|
relation->rd_att->tdtypmod = -1; /* just to be sure */
|
2009-09-27 01:08:22 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* initialize tuple desc info
|
|
|
|
*/
|
2002-11-15 18:18:49 +01:00
|
|
|
has_not_null = false;
|
1996-07-09 08:22:35 +02:00
|
|
|
for (i = 0; i < natts; i++)
|
|
|
|
{
|
2017-08-20 20:19:07 +02:00
|
|
|
memcpy(TupleDescAttr(relation->rd_att, i),
|
2009-08-12 22:53:31 +02:00
|
|
|
&attrs[i],
|
2009-01-22 21:16:10 +01:00
|
|
|
ATTRIBUTE_FIXED_PART_SIZE);
|
2009-08-12 22:53:31 +02:00
|
|
|
has_not_null |= attrs[i].attnotnull;
|
2002-02-19 21:11:20 +01:00
|
|
|
/* make sure attcacheoff is valid */
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(relation->rd_att, i)->attcacheoff = -1;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* initialize first attribute's attcacheoff, cf RelationBuildTupleDesc */
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(relation->rd_att, 0)->attcacheoff = 0;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2002-11-15 18:18:49 +01:00
|
|
|
/* mark not-null status */
|
|
|
|
if (has_not_null)
|
|
|
|
{
|
|
|
|
TupleConstr *constr = (TupleConstr *) palloc0(sizeof(TupleConstr));
|
|
|
|
|
|
|
|
constr->has_not_null = true;
|
|
|
|
relation->rd_att->constr = constr;
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2001-06-01 04:41:36 +02:00
|
|
|
* initialize relation id from info in att array (my, this is ugly)
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2017-08-20 20:19:07 +02:00
|
|
|
RelationGetRelid(relation) = TupleDescAttr(relation->rd_att, 0)->attrelid;
|
2010-02-07 21:48:13 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* All relations made with formrdesc are mapped. This is necessarily so
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* because there is no other way to know what filenumber they currently
|
2010-02-07 21:48:13 +01:00
|
|
|
* have. In bootstrap mode, add them to the initial relation mapper data,
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* specifying that the initial filenumber is the same as the OID.
|
2010-02-07 21:48:13 +01:00
|
|
|
*/
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_rel->relfilenode = InvalidRelFileNumber;
|
2010-02-07 21:48:13 +01:00
|
|
|
if (IsBootstrapProcessingMode())
|
|
|
|
RelationMapUpdateMap(RelationGetRelid(relation),
|
|
|
|
RelationGetRelid(relation),
|
|
|
|
isshared, true);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2004-06-18 08:14:31 +02:00
|
|
|
* initialize the relation lock manager information
|
1999-09-18 21:08:25 +02:00
|
|
|
*/
|
|
|
|
RelationInitLockInfo(relation); /* see lmgr.c */
|
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
/*
|
|
|
|
* initialize physical addressing information for the relation
|
|
|
|
*/
|
|
|
|
RelationInitPhysicalAddr(relation);
|
2000-10-16 16:52:28 +02:00
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
/*
|
|
|
|
* initialize the table am handler
|
|
|
|
*/
|
|
|
|
relation->rd_rel->relam = HEAP_TABLE_AM_OID;
|
|
|
|
relation->rd_tableam = GetHeapamTableAmRoutine();
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-11-10 01:33:12 +01:00
|
|
|
* initialize the rel-has-index flag, using hardwired knowledge
|
2000-10-23 06:10:24 +02:00
|
|
|
*/
|
2004-12-12 06:07:50 +01:00
|
|
|
if (IsBootstrapProcessingMode())
|
|
|
|
{
|
|
|
|
/* In bootstrap mode, we have no indexes */
|
|
|
|
relation->rd_rel->relhasindex = false;
|
|
|
|
}
|
|
|
|
else
|
2000-11-10 01:33:12 +01:00
|
|
|
{
|
2002-04-27 23:24:34 +02:00
|
|
|
/* Otherwise, all the rels formrdesc is used for have indexes */
|
|
|
|
relation->rd_rel->relhasindex = true;
|
2000-11-10 01:33:12 +01:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-11-10 01:33:12 +01:00
|
|
|
* add new reldesc to relcache
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
RelationCacheInsert(relation, false);
|
2004-08-28 22:31:44 +02:00
|
|
|
|
|
|
|
/* It's fully valid */
|
|
|
|
relation->rd_isvalid = true;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* ----------------------------------------------------------------
|
|
|
|
* Relation Descriptor Lookup Interface
|
|
|
|
* ----------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2006-07-31 22:09:10 +02:00
|
|
|
* RelationIdGetRelation
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2006-07-31 22:09:10 +02:00
|
|
|
* Lookup a reldesc by OID; make one if not already in cache.
|
2000-06-30 09:04:23 +02:00
|
|
|
*
|
2006-07-31 22:09:10 +02:00
|
|
|
* Returns NULL if no pg_class row could be found for the given relid
|
|
|
|
* (suggesting we are trying to access a just-deleted relation).
|
|
|
|
* Any other error is reported via elog.
|
1999-09-18 21:08:25 +02:00
|
|
|
*
|
2006-07-31 22:09:10 +02:00
|
|
|
* NB: caller should already have at least AccessShareLock on the
|
|
|
|
* relation ID, else there are nasty race conditions.
|
|
|
|
*
|
|
|
|
* NB: relation ref count is incremented, or set to 1 if new entry.
|
1999-09-18 21:08:25 +02:00
|
|
|
* Caller should eventually decrement count. (Usually,
|
|
|
|
* that happens by calling RelationClose().)
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
Relation
|
2006-07-31 22:09:10 +02:00
|
|
|
RelationIdGetRelation(Oid relationId)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
Relation rd;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2014-02-06 17:28:13 +01:00
|
|
|
/* Make sure we're in an xact, even if this ends up being a cache hit */
|
|
|
|
Assert(IsTransactionState());
|
|
|
|
|
2006-07-31 22:09:10 +02:00
|
|
|
/*
|
|
|
|
* first try to find reldesc in the cache
|
|
|
|
*/
|
1996-07-09 08:22:35 +02:00
|
|
|
RelationIdCacheLookup(relationId, rd);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
if (RelationIsValid(rd))
|
2003-09-24 20:54:02 +02:00
|
|
|
{
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/* return NULL for dropped relations */
|
|
|
|
if (rd->rd_droppedSubid != InvalidSubTransactionId)
|
|
|
|
{
|
|
|
|
Assert(!rd->rd_isvalid);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
RelationIncrementReferenceCount(rd);
|
2010-01-12 19:12:18 +01:00
|
|
|
/* revalidate cache entry if necessary */
|
2004-08-28 22:31:44 +02:00
|
|
|
if (!rd->rd_isvalid)
|
2010-01-12 19:12:18 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Indexes only have a limited number of possible schema changes,
|
|
|
|
* and we don't want to use the full-blown procedure because it's
|
|
|
|
* a headache for indexes that reload itself depends on.
|
|
|
|
*/
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
if (rd->rd_rel->relkind == RELKIND_INDEX ||
|
|
|
|
rd->rd_rel->relkind == RELKIND_PARTITIONED_INDEX)
|
2010-01-12 19:12:18 +01:00
|
|
|
RelationReloadIndexInfo(rd);
|
|
|
|
else
|
|
|
|
RelationClearRelation(rd, true);
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Normally entries need to be valid here, but before the relcache
|
|
|
|
* has been initialized, not enough infrastructure exists to
|
|
|
|
* perform pg_class lookups. The structure of such entries doesn't
|
|
|
|
* change, but we still want to update the rd_rel entry. So
|
|
|
|
* rd_isvalid = false is left in place for a later lookup.
|
|
|
|
*/
|
|
|
|
Assert(rd->rd_isvalid ||
|
|
|
|
(rd->rd_isnailed && !criticalRelcachesBuilt));
|
2010-01-12 19:12:18 +01:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
return rd;
|
2006-07-31 22:09:10 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1996-07-09 08:22:35 +02:00
|
|
|
* no reldesc in the cache, so have RelationBuildDesc() build one and add
|
|
|
|
* it.
|
|
|
|
*/
|
2010-01-12 19:12:18 +01:00
|
|
|
rd = RelationBuildDesc(relationId, true);
|
2004-07-17 05:32:14 +02:00
|
|
|
if (RelationIsValid(rd))
|
|
|
|
RelationIncrementReferenceCount(rd);
|
1996-07-09 08:22:35 +02:00
|
|
|
return rd;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* ----------------------------------------------------------------
|
|
|
|
* cache invalidation support routines
|
|
|
|
* ----------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
/* ResourceOwner callbacks to track relcache references */
|
|
|
|
static void ResOwnerReleaseRelation(Datum res);
|
|
|
|
static char *ResOwnerPrintRelCache(Datum res);
|
|
|
|
|
|
|
|
static const ResourceOwnerDesc relref_resowner_desc =
|
|
|
|
{
|
|
|
|
.name = "relcache reference",
|
|
|
|
.release_phase = RESOURCE_RELEASE_BEFORE_LOCKS,
|
|
|
|
.release_priority = RELEASE_PRIO_RELCACHE_REFS,
|
|
|
|
.ReleaseResource = ResOwnerReleaseRelation,
|
|
|
|
.DebugPrint = ResOwnerPrintRelCache
|
|
|
|
};
|
|
|
|
|
|
|
|
/* Convenience wrappers over ResourceOwnerRemember/Forget */
|
|
|
|
static inline void
|
|
|
|
ResourceOwnerRememberRelationRef(ResourceOwner owner, Relation rel)
|
|
|
|
{
|
|
|
|
ResourceOwnerRemember(owner, PointerGetDatum(rel), &relref_resowner_desc);
|
|
|
|
}
|
|
|
|
static inline void
|
|
|
|
ResourceOwnerForgetRelationRef(ResourceOwner owner, Relation rel)
|
|
|
|
{
|
|
|
|
ResourceOwnerForget(owner, PointerGetDatum(rel), &relref_resowner_desc);
|
|
|
|
}
|
|
|
|
|
2004-07-17 05:32:14 +02:00
|
|
|
/*
|
|
|
|
* RelationIncrementReferenceCount
|
|
|
|
* Increments relation reference count.
|
|
|
|
*
|
|
|
|
* Note: bootstrap mode has its own weird ideas about relation refcount
|
|
|
|
* behavior; we ought to fix it someday, but for now, just disable
|
|
|
|
* reference count ownership tracking in bootstrap mode.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationIncrementReferenceCount(Relation rel)
|
|
|
|
{
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
ResourceOwnerEnlarge(CurrentResourceOwner);
|
2004-07-17 05:32:14 +02:00
|
|
|
rel->rd_refcnt += 1;
|
|
|
|
if (!IsBootstrapProcessingMode())
|
|
|
|
ResourceOwnerRememberRelationRef(CurrentResourceOwner, rel);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RelationDecrementReferenceCount
|
|
|
|
* Decrements relation reference count.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationDecrementReferenceCount(Relation rel)
|
|
|
|
{
|
|
|
|
Assert(rel->rd_refcnt > 0);
|
|
|
|
rel->rd_refcnt -= 1;
|
|
|
|
if (!IsBootstrapProcessingMode())
|
|
|
|
ResourceOwnerForgetRelationRef(CurrentResourceOwner, rel);
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1999-10-04 01:55:40 +02:00
|
|
|
* RelationClose - close an open relation
|
|
|
|
*
|
2001-01-08 19:34:44 +01:00
|
|
|
* Actually, we just decrement the refcount.
|
|
|
|
*
|
|
|
|
* NOTE: if compiled with -DRELCACHE_FORCE_RELEASE then relcache entries
|
|
|
|
* will be freed as soon as their refcount goes to zero. In combination
|
|
|
|
* with aset.c's CLOBBER_FREED_MEMORY option, this provides a good test
|
|
|
|
* to catch references to already-released relcache entries. It slows
|
|
|
|
* things down quite a bit, however.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationClose(Relation relation)
|
|
|
|
{
|
|
|
|
/* Note: no locking manipulations needed */
|
|
|
|
RelationDecrementReferenceCount(relation);
|
2001-01-08 19:34:44 +01:00
|
|
|
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
RelationCloseCleanup(relation);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
RelationCloseCleanup(Relation relation)
|
|
|
|
{
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
/*
|
|
|
|
* If the relation is no longer open in this session, we can clean up any
|
|
|
|
* stale partition descriptors it has. This is unlikely, so check to see
|
|
|
|
* if there are child contexts before expending a call to mcxt.c.
|
|
|
|
*/
|
2021-04-28 21:44:35 +02:00
|
|
|
if (RelationHasReferenceCountZero(relation))
|
|
|
|
{
|
|
|
|
if (relation->rd_pdcxt != NULL &&
|
|
|
|
relation->rd_pdcxt->firstchild != NULL)
|
|
|
|
MemoryContextDeleteChildren(relation->rd_pdcxt);
|
|
|
|
|
|
|
|
if (relation->rd_pddcxt != NULL &&
|
|
|
|
relation->rd_pddcxt->firstchild != NULL)
|
|
|
|
MemoryContextDeleteChildren(relation->rd_pddcxt);
|
|
|
|
}
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
|
2001-01-08 19:34:44 +01:00
|
|
|
#ifdef RELCACHE_FORCE_RELEASE
|
2002-08-06 04:36:35 +02:00
|
|
|
if (RelationHasReferenceCountZero(relation) &&
|
2007-03-29 02:15:39 +02:00
|
|
|
relation->rd_createSubid == InvalidSubTransactionId &&
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid == InvalidSubTransactionId)
|
2001-01-08 19:34:44 +01:00
|
|
|
RelationClearRelation(relation, false);
|
|
|
|
#endif
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2007-05-02 23:08:46 +02:00
|
|
|
* RelationReloadIndexInfo - reload minimal information for an open index
|
2003-09-24 20:54:02 +02:00
|
|
|
*
|
2007-05-02 23:08:46 +02:00
|
|
|
* This function is used only for indexes. A relcache inval on an index
|
|
|
|
* can mean that its pg_class or pg_index row changed. There are only
|
|
|
|
* very limited changes that are allowed to an existing index's schema,
|
|
|
|
* so we can update the relcache entry without a complete rebuild; which
|
|
|
|
* is fortunate because we can't rebuild an index entry that is "nailed"
|
|
|
|
* and/or in active use. We support full replacement of the pg_class row,
|
|
|
|
* as well as updates of a few simple fields of the pg_index row.
|
2003-09-24 20:54:02 +02:00
|
|
|
*
|
2007-05-02 23:08:46 +02:00
|
|
|
* We can't necessarily reread the catalog rows right away; we might be
|
2003-09-24 20:54:02 +02:00
|
|
|
* in a failed transaction when we receive the SI notification. If so,
|
|
|
|
* RelationClearRelation just marks the entry as invalid by setting
|
2004-08-28 22:31:44 +02:00
|
|
|
* rd_isvalid to false. This routine is called to fix the entry when it
|
2003-09-24 20:54:02 +02:00
|
|
|
* is next needed.
|
2008-04-16 20:23:04 +02:00
|
|
|
*
|
|
|
|
* We assume that at the time we are called, we have at least AccessShareLock
|
|
|
|
* on the target index. (Note: in the calls from RelationClearRelation,
|
|
|
|
* this is legitimate because we know the rel has positive refcount.)
|
2011-03-22 18:00:24 +01:00
|
|
|
*
|
|
|
|
* If the target index is an index on pg_class or pg_index, we'd better have
|
|
|
|
* previously gotten at least AccessShareLock on its underlying catalog,
|
|
|
|
* else we are at risk of deadlock against someone trying to exclusive-lock
|
|
|
|
* the heap and index in that order. This is ensured in current usage by
|
|
|
|
* only applying this to indexes being opened or having positive refcount.
|
2000-12-08 07:17:58 +01:00
|
|
|
*/
|
|
|
|
static void
|
2007-05-02 23:08:46 +02:00
|
|
|
RelationReloadIndexInfo(Relation relation)
|
2000-12-08 07:17:58 +01:00
|
|
|
{
|
2003-09-24 20:54:02 +02:00
|
|
|
bool indexOK;
|
2000-12-08 07:17:58 +01:00
|
|
|
HeapTuple pg_class_tuple;
|
|
|
|
Form_pg_class relp;
|
|
|
|
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/* Should be called only for invalidated, live indexes */
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
Assert((relation->rd_rel->relkind == RELKIND_INDEX ||
|
|
|
|
relation->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) &&
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
!relation->rd_isvalid &&
|
|
|
|
relation->rd_droppedSubid == InvalidSubTransactionId);
|
2014-02-07 01:37:58 +01:00
|
|
|
|
|
|
|
/* Ensure it's closed at smgr level */
|
|
|
|
RelationCloseSmgr(relation);
|
2005-10-15 04:49:52 +02:00
|
|
|
|
2010-02-09 22:43:30 +01:00
|
|
|
/* Must free any AM cached data upon relcache flush */
|
2009-08-12 22:53:31 +02:00
|
|
|
if (relation->rd_amcache)
|
|
|
|
pfree(relation->rd_amcache);
|
|
|
|
relation->rd_amcache = NULL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If it's a shared index, we might be called before backend startup has
|
|
|
|
* finished selecting a database, in which case we have no way to read
|
|
|
|
* pg_class yet. However, a shared index can never have any significant
|
|
|
|
* schema updates, so it's okay to ignore the invalidation signal. Just
|
|
|
|
* mark it valid and return without doing anything more.
|
|
|
|
*/
|
|
|
|
if (relation->rd_rel->relisshared && !criticalRelcachesBuilt)
|
|
|
|
{
|
|
|
|
relation->rd_isvalid = true;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2003-09-24 20:54:02 +02:00
|
|
|
/*
|
2005-04-14 22:03:27 +02:00
|
|
|
* Read the pg_class row
|
|
|
|
*
|
2003-09-24 20:54:02 +02:00
|
|
|
* Don't try to use an indexscan of pg_class_oid_index to reload the info
|
|
|
|
* for pg_class_oid_index ...
|
|
|
|
*/
|
2005-04-14 22:03:27 +02:00
|
|
|
indexOK = (RelationGetRelid(relation) != ClassOidIndexId);
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
pg_class_tuple = ScanPgRelation(RelationGetRelid(relation), indexOK, false);
|
2000-12-08 07:17:58 +01:00
|
|
|
if (!HeapTupleIsValid(pg_class_tuple))
|
2006-01-19 21:28:43 +01:00
|
|
|
elog(ERROR, "could not find pg_class tuple for index %u",
|
2005-04-14 22:03:27 +02:00
|
|
|
RelationGetRelid(relation));
|
2000-12-08 07:17:58 +01:00
|
|
|
relp = (Form_pg_class) GETSTRUCT(pg_class_tuple);
|
2006-01-19 21:28:43 +01:00
|
|
|
memcpy(relation->rd_rel, relp, CLASS_TUPLE_SIZE);
|
2006-07-04 00:45:41 +02:00
|
|
|
/* Reload reloptions in case they changed */
|
2006-07-02 04:23:23 +02:00
|
|
|
if (relation->rd_options)
|
|
|
|
pfree(relation->rd_options);
|
2006-07-04 00:45:41 +02:00
|
|
|
RelationParseRelOptions(relation, pg_class_tuple);
|
|
|
|
/* done with pg_class tuple */
|
2000-12-08 07:17:58 +01:00
|
|
|
heap_freetuple(pg_class_tuple);
|
2006-01-19 21:28:43 +01:00
|
|
|
/* We must recalculate physical address in case it changed */
|
|
|
|
RelationInitPhysicalAddr(relation);
|
2009-06-11 16:49:15 +02:00
|
|
|
|
2007-05-02 23:08:46 +02:00
|
|
|
/*
|
|
|
|
* For a non-system index, there are fields of the pg_index row that are
|
|
|
|
* allowed to change, so re-read that row and update the relcache entry.
|
|
|
|
* Most of the info derived from pg_index (such as support function lookup
|
|
|
|
* info) cannot change, and indeed the whole point of this routine is to
|
|
|
|
* update the relcache entry without clobbering that data; so wholesale
|
|
|
|
* replacement is not appropriate.
|
|
|
|
*/
|
|
|
|
if (!IsSystemRelation(relation))
|
|
|
|
{
|
|
|
|
HeapTuple tuple;
|
|
|
|
Form_pg_index index;
|
|
|
|
|
2010-02-14 19:42:19 +01:00
|
|
|
tuple = SearchSysCache1(INDEXRELID,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
2007-05-02 23:08:46 +02:00
|
|
|
if (!HeapTupleIsValid(tuple))
|
|
|
|
elog(ERROR, "cache lookup failed for index %u",
|
|
|
|
RelationGetRelid(relation));
|
|
|
|
index = (Form_pg_index) GETSTRUCT(tuple);
|
|
|
|
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
/*
|
|
|
|
* Basically, let's just copy all the bool fields. There are one or
|
|
|
|
* two of these that can't actually change in the current code, but
|
|
|
|
* it's not worth it to track exactly which ones they are. None of
|
|
|
|
* the array fields are allowed to change, though.
|
|
|
|
*/
|
|
|
|
relation->rd_index->indisunique = index->indisunique;
|
2022-02-03 11:29:54 +01:00
|
|
|
relation->rd_index->indnullsnotdistinct = index->indnullsnotdistinct;
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
relation->rd_index->indisprimary = index->indisprimary;
|
|
|
|
relation->rd_index->indisexclusion = index->indisexclusion;
|
|
|
|
relation->rd_index->indimmediate = index->indimmediate;
|
|
|
|
relation->rd_index->indisclustered = index->indisclustered;
|
2007-05-02 23:08:46 +02:00
|
|
|
relation->rd_index->indisvalid = index->indisvalid;
|
2007-09-20 19:56:33 +02:00
|
|
|
relation->rd_index->indcheckxmin = index->indcheckxmin;
|
|
|
|
relation->rd_index->indisready = index->indisready;
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
relation->rd_index->indislive = index->indislive;
|
2023-07-14 04:15:34 +02:00
|
|
|
relation->rd_index->indisreplident = index->indisreplident;
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
|
|
|
|
/* Copy xmin too, as that is needed to make sense of indcheckxmin */
|
2007-09-20 19:56:33 +02:00
|
|
|
HeapTupleHeaderSetXmin(relation->rd_indextuple->t_data,
|
|
|
|
HeapTupleHeaderGetXmin(tuple->t_data));
|
2007-05-02 23:08:46 +02:00
|
|
|
|
|
|
|
ReleaseSysCache(tuple);
|
|
|
|
}
|
|
|
|
|
2003-09-24 20:54:02 +02:00
|
|
|
/* Okay, now it's valid again */
|
2004-08-28 22:31:44 +02:00
|
|
|
relation->rd_isvalid = true;
|
2000-12-08 07:17:58 +01:00
|
|
|
}
|
2001-01-08 19:34:44 +01:00
|
|
|
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
/*
|
|
|
|
* RelationReloadNailed - reload minimal information for nailed relations.
|
|
|
|
*
|
|
|
|
* The structure of a nailed relation can never change (which is good, because
|
|
|
|
* we rely on knowing their structure to be able to read catalog content). But
|
|
|
|
* some parts, e.g. pg_class.relfrozenxid, are still important to have
|
|
|
|
* accurate content for. Therefore those need to be reloaded after the arrival
|
|
|
|
* of invalidations.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
RelationReloadNailed(Relation relation)
|
|
|
|
{
|
|
|
|
Assert(relation->rd_isnailed);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Redo RelationInitPhysicalAddr in case it is a mapped relation whose
|
|
|
|
* mapping changed.
|
|
|
|
*/
|
|
|
|
RelationInitPhysicalAddr(relation);
|
|
|
|
|
|
|
|
/* flag as needing to be revalidated */
|
|
|
|
relation->rd_isvalid = false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Can only reread catalog contents if in a transaction. If the relation
|
|
|
|
* is currently open (not counting the nailed refcount), do so
|
|
|
|
* immediately. Otherwise we've already marked the entry as possibly
|
|
|
|
* invalid, and it'll be fixed when next opened.
|
|
|
|
*/
|
|
|
|
if (!IsTransactionState() || relation->rd_refcnt <= 1)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (relation->rd_rel->relkind == RELKIND_INDEX)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If it's a nailed-but-not-mapped index, then we need to re-read the
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* pg_class row to see if its relfilenumber changed.
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
*/
|
|
|
|
RelationReloadIndexInfo(relation);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Reload a non-index entry. We can't easily do so if relcaches
|
|
|
|
* aren't yet built, but that's fine because at that stage the
|
|
|
|
* attributes that need to be current (like relfrozenxid) aren't yet
|
|
|
|
* accessed. To ensure the entry will later be revalidated, we leave
|
|
|
|
* it in invalid state, but allow use (cf. RelationIdGetRelation()).
|
|
|
|
*/
|
|
|
|
if (criticalRelcachesBuilt)
|
|
|
|
{
|
|
|
|
HeapTuple pg_class_tuple;
|
|
|
|
Form_pg_class relp;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* NB: Mark the entry as valid before starting to scan, to avoid
|
|
|
|
* self-recursion when re-building pg_class.
|
|
|
|
*/
|
|
|
|
relation->rd_isvalid = true;
|
|
|
|
|
|
|
|
pg_class_tuple = ScanPgRelation(RelationGetRelid(relation),
|
|
|
|
true, false);
|
|
|
|
relp = (Form_pg_class) GETSTRUCT(pg_class_tuple);
|
|
|
|
memcpy(relation->rd_rel, relp, CLASS_TUPLE_SIZE);
|
|
|
|
heap_freetuple(pg_class_tuple);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Again mark as valid, to protect against concurrently arriving
|
|
|
|
* invalidations.
|
|
|
|
*/
|
|
|
|
relation->rd_isvalid = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-01-12 19:12:18 +01:00
|
|
|
/*
|
|
|
|
* RelationDestroyRelation
|
|
|
|
*
|
|
|
|
* Physically delete a relation cache entry and all subsidiary data.
|
|
|
|
* Caller must already have unhooked the entry from the hash table.
|
|
|
|
*/
|
|
|
|
static void
|
2014-04-06 17:13:43 +02:00
|
|
|
RelationDestroyRelation(Relation relation, bool remember_tupdesc)
|
2010-01-12 19:12:18 +01:00
|
|
|
{
|
|
|
|
Assert(RelationHasReferenceCountZero(relation));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure smgr and lower levels close the relation's files, if they
|
|
|
|
* weren't closed already. (This was probably done by caller, but let's
|
|
|
|
* just be real sure.)
|
|
|
|
*/
|
|
|
|
RelationCloseSmgr(relation);
|
|
|
|
|
2022-04-07 06:29:46 +02:00
|
|
|
/* break mutual link with stats entry */
|
|
|
|
pgstat_unlink_relation(relation);
|
|
|
|
|
2010-01-12 19:12:18 +01:00
|
|
|
/*
|
|
|
|
* Free all the subsidiary data structures of the relcache entry, then the
|
|
|
|
* entry itself.
|
|
|
|
*/
|
|
|
|
if (relation->rd_rel)
|
|
|
|
pfree(relation->rd_rel);
|
|
|
|
/* can't use DecrTupleDescRefCount here */
|
|
|
|
Assert(relation->rd_att->tdrefcount > 0);
|
|
|
|
if (--relation->rd_att->tdrefcount == 0)
|
2014-04-06 17:13:43 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If we Rebuilt a relcache entry during a transaction then its
|
|
|
|
* possible we did that because the TupDesc changed as the result of
|
|
|
|
* an ALTER TABLE that ran at less than AccessExclusiveLock. It's
|
|
|
|
* possible someone copied that TupDesc, in which case the copy would
|
|
|
|
* point to free'd memory. So if we rebuild an entry we keep the
|
|
|
|
* TupDesc around until end of transaction, to be safe.
|
|
|
|
*/
|
|
|
|
if (remember_tupdesc)
|
|
|
|
RememberToFreeTupleDescAtEOX(relation->rd_att);
|
|
|
|
else
|
|
|
|
FreeTupleDesc(relation->rd_att);
|
|
|
|
}
|
2016-06-18 21:22:34 +02:00
|
|
|
FreeTriggerDesc(relation->trigdesc);
|
|
|
|
list_free_deep(relation->rd_fkeylist);
|
2010-01-12 19:12:18 +01:00
|
|
|
list_free(relation->rd_indexlist);
|
2021-03-19 01:37:09 +01:00
|
|
|
list_free(relation->rd_statlist);
|
2014-05-14 20:55:48 +02:00
|
|
|
bms_free(relation->rd_keyattr);
|
2017-01-19 18:00:00 +01:00
|
|
|
bms_free(relation->rd_pkattr);
|
2014-05-14 20:55:48 +02:00
|
|
|
bms_free(relation->rd_idattr);
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
bms_free(relation->rd_hotblockingattr);
|
|
|
|
bms_free(relation->rd_summarizedattr);
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
if (relation->rd_pubdesc)
|
|
|
|
pfree(relation->rd_pubdesc);
|
2010-01-12 19:12:18 +01:00
|
|
|
if (relation->rd_options)
|
|
|
|
pfree(relation->rd_options);
|
|
|
|
if (relation->rd_indextuple)
|
|
|
|
pfree(relation->rd_indextuple);
|
2019-07-30 20:43:27 +02:00
|
|
|
if (relation->rd_amcache)
|
|
|
|
pfree(relation->rd_amcache);
|
|
|
|
if (relation->rd_fdwroutine)
|
|
|
|
pfree(relation->rd_fdwroutine);
|
2010-01-12 19:12:18 +01:00
|
|
|
if (relation->rd_indexcxt)
|
|
|
|
MemoryContextDelete(relation->rd_indexcxt);
|
|
|
|
if (relation->rd_rulescxt)
|
|
|
|
MemoryContextDelete(relation->rd_rulescxt);
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
if (relation->rd_rsdesc)
|
|
|
|
MemoryContextDelete(relation->rd_rsdesc->rscxt);
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
if (relation->rd_partkeycxt)
|
|
|
|
MemoryContextDelete(relation->rd_partkeycxt);
|
|
|
|
if (relation->rd_pdcxt)
|
|
|
|
MemoryContextDelete(relation->rd_pdcxt);
|
2021-04-28 21:44:35 +02:00
|
|
|
if (relation->rd_pddcxt)
|
|
|
|
MemoryContextDelete(relation->rd_pddcxt);
|
2019-04-13 19:22:26 +02:00
|
|
|
if (relation->rd_partcheckcxt)
|
|
|
|
MemoryContextDelete(relation->rd_partcheckcxt);
|
2010-01-12 19:12:18 +01:00
|
|
|
pfree(relation);
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1999-10-04 01:55:40 +02:00
|
|
|
* RelationClearRelation
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
1999-10-04 01:55:40 +02:00
|
|
|
* Physically blow away a relation cache entry, or reset it and rebuild
|
|
|
|
* it from scratch (that is, from catalog entries). The latter path is
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
* used when we are notified of a change to an open relation (one with
|
|
|
|
* refcount > 0).
|
2008-04-16 20:23:04 +02:00
|
|
|
*
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
* NB: when rebuilding, we'd better hold some lock on the relation,
|
|
|
|
* else the catalog data we need to read could be changing under us.
|
|
|
|
* Also, a rel to be rebuilt had better have refcnt > 0. This is because
|
2018-03-29 21:18:53 +02:00
|
|
|
* a sinval reset could happen while we're accessing the catalogs, and
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
* the rel would get blown away underneath us by RelationCacheInvalidate
|
|
|
|
* if it has zero refcnt.
|
|
|
|
*
|
|
|
|
* The "rebuild" parameter is redundant in current usage because it has
|
|
|
|
* to match the relation's refcnt status, but we keep it as a crosscheck
|
|
|
|
* that we're doing what the caller expects.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
1997-08-19 23:40:56 +02:00
|
|
|
static void
|
2002-07-15 03:57:51 +02:00
|
|
|
RelationClearRelation(Relation relation, bool rebuild)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
/*
|
|
|
|
* As per notes above, a rel to be rebuilt MUST have refcnt > 0; while of
|
Fix subtransaction cleanup after an outer-subtransaction portal fails.
Formerly, we treated only portals created in the current subtransaction as
having failed during subtransaction abort. However, if the error occurred
while running a portal created in an outer subtransaction (ie, a cursor
declared before the last savepoint), that has to be considered broken too.
To allow reliable detection of which ones those are, add a bookkeeping
field to struct Portal that tracks the innermost subtransaction in which
each portal has actually been executed. (Without this, we'd end up
failing portals containing functions that had called the subtransaction,
thereby breaking plpgsql exception blocks completely.)
In addition, when we fail an outer-subtransaction Portal, transfer its
resources into the subtransaction's resource owner, so that they're
released early in cleanup of the subxact. This fixes a problem reported by
Jim Nasby in which a function executed in an outer-subtransaction cursor
could cause an Assert failure or crash by referencing a relation created
within the inner subtransaction.
The proximate cause of the Assert failure is that AtEOSubXact_RelationCache
assumed it could blow away a relcache entry without first checking that the
entry had zero refcount. That was a bad idea on its own terms, so add such
a check there, and to the similar coding in AtEOXact_RelationCache. This
provides an independent safety measure in case there are still ways to
provoke the situation despite the Portal-level changes.
This has been broken since subtransactions were invented, so back-patch
to all supported branches.
Tom Lane and Michael Paquier
2015-09-04 19:36:49 +02:00
|
|
|
* course it would be an equally bad idea to blow away one with nonzero
|
|
|
|
* refcnt, since that would leave someone somewhere with a dangling
|
|
|
|
* pointer. All callers are expected to have verified that this holds.
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
*/
|
|
|
|
Assert(rebuild ?
|
|
|
|
!RelationHasReferenceCountZero(relation) :
|
|
|
|
RelationHasReferenceCountZero(relation));
|
|
|
|
|
1999-09-04 20:42:15 +02:00
|
|
|
/*
|
|
|
|
* Make sure smgr and lower levels close the relation's files, if they
|
2000-11-08 23:10:03 +01:00
|
|
|
* weren't closed already. If the relation is not getting deleted, the
|
|
|
|
* next smgr access should reopen the files automatically. This ensures
|
|
|
|
* that the low-level file access state is updated after, say, a vacuum
|
|
|
|
* truncation.
|
1999-09-04 20:42:15 +02:00
|
|
|
*/
|
2005-01-10 21:02:24 +01:00
|
|
|
RelationCloseSmgr(relation);
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2019-07-30 20:43:27 +02:00
|
|
|
/* Free AM cached data, if any */
|
|
|
|
if (relation->rd_amcache)
|
|
|
|
pfree(relation->rd_amcache);
|
|
|
|
relation->rd_amcache = NULL;
|
|
|
|
|
1999-10-04 01:55:40 +02:00
|
|
|
/*
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
* Treat nailed-in system relations separately, they always need to be
|
|
|
|
* accessible, so we can't blow them away.
|
1999-10-04 01:55:40 +02:00
|
|
|
*/
|
|
|
|
if (relation->rd_isnailed)
|
2000-12-08 07:17:58 +01:00
|
|
|
{
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
RelationReloadNailed(relation);
|
1996-07-09 08:22:35 +02:00
|
|
|
return;
|
2000-12-08 07:17:58 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/* Mark it invalid until we've finished rebuild */
|
|
|
|
relation->rd_isvalid = false;
|
|
|
|
|
|
|
|
/* See RelationForgetRelation(). */
|
|
|
|
if (relation->rd_droppedSubid != InvalidSubTransactionId)
|
|
|
|
return;
|
|
|
|
|
2006-01-19 21:28:43 +01:00
|
|
|
/*
|
|
|
|
* Even non-system indexes should not be blown away if they are open and
|
|
|
|
* have valid index support information. This avoids problems with active
|
|
|
|
* use of the index support information. As with nailed indexes, we
|
|
|
|
* re-read the pg_class row to handle possible physical relocation of the
|
2007-05-02 23:08:46 +02:00
|
|
|
* index, and we check for pg_index updates too.
|
2006-01-19 21:28:43 +01:00
|
|
|
*/
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
if ((relation->rd_rel->relkind == RELKIND_INDEX ||
|
|
|
|
relation->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) &&
|
2006-01-19 21:28:43 +01:00
|
|
|
relation->rd_refcnt > 0 &&
|
|
|
|
relation->rd_indexcxt != NULL)
|
|
|
|
{
|
2014-02-07 01:37:58 +01:00
|
|
|
if (IsTransactionState())
|
|
|
|
RelationReloadIndexInfo(relation);
|
2006-01-19 21:28:43 +01:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
1999-10-04 01:55:40 +02:00
|
|
|
/*
|
|
|
|
* If we're really done with the relcache entry, blow it away. But if
|
1999-09-04 20:42:15 +02:00
|
|
|
* someone is still using it, reconstruct the whole deal without moving
|
|
|
|
* the physical RelationData record (so that the someone's pointer is
|
2000-01-31 05:35:57 +01:00
|
|
|
* still valid).
|
1999-09-04 20:42:15 +02:00
|
|
|
*/
|
2002-07-15 03:57:51 +02:00
|
|
|
if (!rebuild)
|
1999-09-04 20:42:15 +02:00
|
|
|
{
|
2010-01-12 19:12:18 +01:00
|
|
|
/* Remove it from the hash table */
|
|
|
|
RelationCacheDelete(relation);
|
|
|
|
|
|
|
|
/* And release storage */
|
2014-04-06 17:13:43 +02:00
|
|
|
RelationDestroyRelation(relation, false);
|
1999-09-04 20:42:15 +02:00
|
|
|
}
|
2014-02-07 01:37:58 +01:00
|
|
|
else if (!IsTransactionState())
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If we're not inside a valid transaction, we can't do any catalog
|
|
|
|
* access so it's not possible to rebuild yet. Just exit, leaving
|
|
|
|
* rd_isvalid = false so that the rebuild will occur when the entry is
|
|
|
|
* next opened.
|
|
|
|
*
|
|
|
|
* Note: it's possible that we come here during subtransaction abort,
|
|
|
|
* and the reason for wanting to rebuild is that the rel is open in
|
|
|
|
* the outer transaction. In that case it might seem unsafe to not
|
|
|
|
* rebuild immediately, since whatever code has the rel already open
|
|
|
|
* will keep on using the relcache entry as-is. However, in such a
|
|
|
|
* case the outer transaction should be holding a lock that's
|
|
|
|
* sufficient to prevent any significant change in the rel's schema,
|
|
|
|
* so the existing entry contents should be good enough for its
|
|
|
|
* purposes; at worst we might be behind on statistics updates or the
|
|
|
|
* like. (See also CheckTableNotInUse() and its callers.) These same
|
|
|
|
* remarks also apply to the cases above where we exit without having
|
|
|
|
* done RelationReloadIndexInfo() yet.
|
|
|
|
*/
|
|
|
|
return;
|
|
|
|
}
|
1999-09-04 20:42:15 +02:00
|
|
|
else
|
|
|
|
{
|
2000-01-31 05:35:57 +01:00
|
|
|
/*
|
2010-01-12 19:12:18 +01:00
|
|
|
* Our strategy for rebuilding an open relcache entry is to build a
|
|
|
|
* new entry from scratch, swap its contents with the old entry, and
|
|
|
|
* finally delete the new entry (along with any infrastructure swapped
|
|
|
|
* over from the old entry). This is to avoid trouble in case an
|
|
|
|
* error causes us to lose control partway through. The old entry
|
|
|
|
* will still be marked !rd_isvalid, so we'll try to rebuild it again
|
|
|
|
* on next access. Meanwhile it's not any less valid than it was
|
|
|
|
* before, so any code that might expect to continue accessing it
|
|
|
|
* isn't hurt by the rebuild failure. (Consider for example a
|
2011-06-29 08:26:14 +02:00
|
|
|
* subtransaction that ALTERs a table and then gets canceled partway
|
2010-01-12 19:12:18 +01:00
|
|
|
* through the cache entry rebuild. The outer transaction should
|
|
|
|
* still see the not-modified cache entry as valid.) The worst
|
|
|
|
* consequence of an error is leaking the necessarily-unreferenced new
|
|
|
|
* entry, and this shouldn't happen often enough for that to be a big
|
|
|
|
* problem.
|
|
|
|
*
|
2010-02-04 01:09:14 +01:00
|
|
|
* When rebuilding an open relcache entry, we must preserve ref count,
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* rd_*Subid, and rd_toastoid state. Also attempt to preserve the
|
|
|
|
* pg_class entry (rd_rel), tupledesc, rewrite-rule, partition key,
|
|
|
|
* and partition descriptor substructures in place, because various
|
|
|
|
* places assume that these structures won't move while they are
|
|
|
|
* working with an open relcache entry. (Note: the refcount
|
|
|
|
* mechanism for tupledescs might someday allow us to remove this hack
|
|
|
|
* for the tupledesc.)
|
2004-07-17 05:32:14 +02:00
|
|
|
*
|
|
|
|
* Note that this process does not touch CurrentResourceOwner; which
|
|
|
|
* is good because whatever ref counts the entry may have do not
|
|
|
|
* necessarily belong to that resource owner.
|
2000-01-31 05:35:57 +01:00
|
|
|
*/
|
2010-01-12 19:12:18 +01:00
|
|
|
Relation newrel;
|
2005-04-14 22:03:27 +02:00
|
|
|
Oid save_relid = RelationGetRelid(relation);
|
2010-01-12 19:12:18 +01:00
|
|
|
bool keep_tupdesc;
|
|
|
|
bool keep_rules;
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
bool keep_policies;
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
bool keep_partkey;
|
2010-01-12 19:12:18 +01:00
|
|
|
|
|
|
|
/* Build temporary entry, but don't link it into hashtable */
|
|
|
|
newrel = RelationBuildDesc(save_relid, false);
|
2021-10-24 03:36:38 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Between here and the end of the swap, don't add code that does or
|
|
|
|
* reasonably could read system catalogs. That range must be free
|
|
|
|
* from invalidation processing. See RelationBuildDesc() manipulation
|
|
|
|
* of in_progress_list.
|
|
|
|
*/
|
|
|
|
|
2010-01-12 19:12:18 +01:00
|
|
|
if (newrel == NULL)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2015-01-07 00:10:19 +01:00
|
|
|
/*
|
|
|
|
* We can validly get here, if we're using a historic snapshot in
|
|
|
|
* which a relation, accessed from outside logical decoding, is
|
|
|
|
* still invisible. In that case it's fine to just mark the
|
|
|
|
* relation as invalid and return - it'll fully get reloaded by
|
|
|
|
* the cache reset at the end of logical decoding (or at the next
|
|
|
|
* access). During normal processing we don't want to ignore this
|
|
|
|
* case as it shouldn't happen there, as explained below.
|
|
|
|
*/
|
|
|
|
if (HistoricSnapshotActive())
|
|
|
|
return;
|
|
|
|
|
2015-01-07 00:10:18 +01:00
|
|
|
/*
|
|
|
|
* This shouldn't happen as dropping a relation is intended to be
|
2018-01-11 14:31:11 +01:00
|
|
|
* impossible if still referenced (cf. CheckTableNotInUse()). But
|
2015-01-07 00:10:18 +01:00
|
|
|
* if we get here anyway, we can't just delete the relcache entry,
|
|
|
|
* as it possibly could get accessed later (as e.g. the error
|
|
|
|
* might get trapped and handled via a subtransaction rollback).
|
|
|
|
*/
|
2005-04-14 22:03:27 +02:00
|
|
|
elog(ERROR, "relation %u deleted while still in use", save_relid);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
2007-01-25 03:17:26 +01:00
|
|
|
|
Prevent pgstats from getting confused when relkind of a relation changes
When the relkind of a relache entry changes, because a table is converted into
a view, pgstats can get confused in 15+, leading to crashes or assertion
failures.
For HEAD, Tom fixed this in b23cd185fd5, by removing support for converting a
table to a view, removing the source of the inconsistency. This commit just
adds an assertion that a relcache entry's relkind does not change, just in
case we end up with another case of that in the future. As there's no cases of
changing relkind anymore, we can't add a test that that's handled correctly.
For 15, fix the problem by not maintaining the association with the old pgstat
entry when the relkind changes during a relcache invalidation processing. In
that case the pgstat entry needs to be unlinked first, to avoid
PgStat_TableStatus->relation getting out of sync. Also add a test reproducing
the issues.
No known problem exists in 11-14, so just add the test there.
Reported-by: vignesh C <vignesh21@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CALDaNm2yXz+zOtv7y5zBd5WKT8O0Ld3YxikuU3dcyCvxF7gypA@mail.gmail.com
Discussion: https://postgr.es/m/CALDaNm3oZA-8Wbps2Jd1g5_Gjrr-x3YWrJPek-mF5Asrrvz2Dg@mail.gmail.com
Backpatch: 15-
2022-12-03 03:09:55 +01:00
|
|
|
/*
|
|
|
|
* If we were to, again, have cases of the relkind of a relcache entry
|
|
|
|
* changing, we would need to ensure that pgstats does not get
|
|
|
|
* confused.
|
|
|
|
*/
|
|
|
|
Assert(relation->rd_rel->relkind == newrel->rd_rel->relkind);
|
|
|
|
|
2010-01-12 19:12:18 +01:00
|
|
|
keep_tupdesc = equalTupleDescs(relation->rd_att, newrel->rd_att);
|
|
|
|
keep_rules = equalRuleLocks(relation->rd_rules, newrel->rd_rules);
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
keep_policies = equalRSDesc(relation->rd_rsdesc, newrel->rd_rsdesc);
|
Fix up run-time partition pruning's use of relcache's partition data.
The previous coding saved pointers into the partitioned table's relcache
entry, but then closed the relcache entry, causing those pointers to
nominally become dangling. Actual trouble would be seen in the field
only if a relcache flush occurred mid-query, but that's hardly out of
the question.
While we could fix this by copying all the data in question at query
start, it seems better to just hold the relcache entry open for the
whole query.
While at it, improve the handling of support-function lookups: do that
once per query not once per pruning test. There's still something to be
desired here, in that we fail to exploit the possibility of caching data
across queries in the fn_extra fields of the relcache's FmgrInfo structs,
which could happen if we just used those structs in-place rather than
copying them. However, combining that with the possibility of per-query
lookups of cross-type comparison functions seems to require changes in the
APIs of a lot of the pruning support functions, so it's too invasive to
consider as part of this patch. A win would ensue only for complex
partition key data types (e.g. arrays), so it may not be worth the
trouble.
David Rowley and Tom Lane
Discussion: https://postgr.es/m/17850.1528755844@sss.pgh.pa.us
2018-06-13 18:03:19 +02:00
|
|
|
/* partkey is immutable once set up, so we can always keep it */
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
keep_partkey = (relation->rd_partkey != NULL);
|
2010-01-12 19:12:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Perform swapping of the relcache entry contents. Within this
|
|
|
|
* process the old entry is momentarily invalid, so there *must* be no
|
|
|
|
* possibility of CHECK_FOR_INTERRUPTS within this sequence. Do it in
|
|
|
|
* all-in-line code for safety.
|
|
|
|
*
|
|
|
|
* Since the vast majority of fields should be swapped, our method is
|
|
|
|
* to swap the whole structures and then re-swap those few fields we
|
|
|
|
* didn't want swapped.
|
|
|
|
*/
|
|
|
|
#define SWAPFIELD(fldtype, fldname) \
|
|
|
|
do { \
|
|
|
|
fldtype _tmp = newrel->fldname; \
|
|
|
|
newrel->fldname = relation->fldname; \
|
|
|
|
relation->fldname = _tmp; \
|
|
|
|
} while (0)
|
|
|
|
|
|
|
|
/* swap all Relation struct fields */
|
2000-01-31 05:35:57 +01:00
|
|
|
{
|
2010-01-12 19:12:18 +01:00
|
|
|
RelationData tmpstruct;
|
|
|
|
|
|
|
|
memcpy(&tmpstruct, newrel, sizeof(RelationData));
|
|
|
|
memcpy(newrel, relation, sizeof(RelationData));
|
|
|
|
memcpy(relation, &tmpstruct, sizeof(RelationData));
|
2000-01-31 05:35:57 +01:00
|
|
|
}
|
2010-01-12 19:12:18 +01:00
|
|
|
|
|
|
|
/* rd_smgr must not be swapped, due to back-links from smgr level */
|
|
|
|
SWAPFIELD(SMgrRelation, rd_smgr);
|
|
|
|
/* rd_refcnt must be preserved */
|
|
|
|
SWAPFIELD(int, rd_refcnt);
|
|
|
|
/* isnailed shouldn't change */
|
|
|
|
Assert(newrel->rd_isnailed == relation->rd_isnailed);
|
|
|
|
/* creation sub-XIDs must be preserved */
|
|
|
|
SWAPFIELD(SubTransactionId, rd_createSubid);
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
SWAPFIELD(SubTransactionId, rd_newRelfilelocatorSubid);
|
|
|
|
SWAPFIELD(SubTransactionId, rd_firstRelfilelocatorSubid);
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
SWAPFIELD(SubTransactionId, rd_droppedSubid);
|
2010-01-12 19:12:18 +01:00
|
|
|
/* un-swap rd_rel pointers, swap contents instead */
|
|
|
|
SWAPFIELD(Form_pg_class, rd_rel);
|
|
|
|
/* ... but actually, we don't have to update newrel->rd_rel */
|
|
|
|
memcpy(relation->rd_rel, newrel->rd_rel, CLASS_TUPLE_SIZE);
|
Fix up run-time partition pruning's use of relcache's partition data.
The previous coding saved pointers into the partitioned table's relcache
entry, but then closed the relcache entry, causing those pointers to
nominally become dangling. Actual trouble would be seen in the field
only if a relcache flush occurred mid-query, but that's hardly out of
the question.
While we could fix this by copying all the data in question at query
start, it seems better to just hold the relcache entry open for the
whole query.
While at it, improve the handling of support-function lookups: do that
once per query not once per pruning test. There's still something to be
desired here, in that we fail to exploit the possibility of caching data
across queries in the fn_extra fields of the relcache's FmgrInfo structs,
which could happen if we just used those structs in-place rather than
copying them. However, combining that with the possibility of per-query
lookups of cross-type comparison functions seems to require changes in the
APIs of a lot of the pruning support functions, so it's too invasive to
consider as part of this patch. A win would ensue only for complex
partition key data types (e.g. arrays), so it may not be worth the
trouble.
David Rowley and Tom Lane
Discussion: https://postgr.es/m/17850.1528755844@sss.pgh.pa.us
2018-06-13 18:03:19 +02:00
|
|
|
/* preserve old tupledesc, rules, policies if no logical change */
|
2010-01-12 19:12:18 +01:00
|
|
|
if (keep_tupdesc)
|
|
|
|
SWAPFIELD(TupleDesc, rd_att);
|
|
|
|
if (keep_rules)
|
2000-01-31 05:35:57 +01:00
|
|
|
{
|
2010-01-12 19:12:18 +01:00
|
|
|
SWAPFIELD(RuleLock *, rd_rules);
|
|
|
|
SWAPFIELD(MemoryContext, rd_rulescxt);
|
2000-01-31 05:35:57 +01:00
|
|
|
}
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
if (keep_policies)
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
SWAPFIELD(RowSecurityDesc *, rd_rsdesc);
|
2010-02-04 01:09:14 +01:00
|
|
|
/* toast OID override must be preserved */
|
|
|
|
SWAPFIELD(Oid, rd_toastoid);
|
2022-04-07 06:29:46 +02:00
|
|
|
/* pgstat_info / enabled must be preserved */
|
2010-01-12 19:12:18 +01:00
|
|
|
SWAPFIELD(struct PgStat_TableStatus *, pgstat_info);
|
2022-04-07 06:29:46 +02:00
|
|
|
SWAPFIELD(bool, pgstat_enabled);
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
/* preserve old partition key if we have one */
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
if (keep_partkey)
|
|
|
|
{
|
|
|
|
SWAPFIELD(PartitionKey, rd_partkey);
|
|
|
|
SWAPFIELD(MemoryContext, rd_partkeycxt);
|
|
|
|
}
|
2021-04-28 21:44:35 +02:00
|
|
|
if (newrel->rd_pdcxt != NULL || newrel->rd_pddcxt != NULL)
|
Allow ATTACH PARTITION with only ShareUpdateExclusiveLock.
We still require AccessExclusiveLock on the partition itself, because
otherwise an insert that violates the newly-imposed partition
constraint could be in progress at the same time that we're changing
that constraint; only the lock level on the parent relation is
weakened.
To make this safe, we have to cope with (at least) three separate
problems. First, relevant DDL might commit while we're in the process
of building a PartitionDesc. If so, find_inheritance_children() might
see a new partition while the RELOID system cache still has the old
partition bound cached, and even before invalidation messages have
been queued. To fix that, if we see that the pg_class tuple seems to
be missing or to have a null relpartbound, refetch the value directly
from the table. We can't get the wrong value, because DETACH PARTITION
still requires AccessExclusiveLock throughout; if we ever want to
change that, this will need more thought. In testing, I found it quite
difficult to hit even the null-relpartbound case; the race condition
is extremely tight, but the theoretical risk is there.
Second, successive calls to RelationGetPartitionDesc might not return
the same answer. The query planner will get confused if lookup up the
PartitionDesc for a particular relation does not return a consistent
answer for the entire duration of query planning. Likewise, query
execution will get confused if the same relation seems to have a
different PartitionDesc at different times. Invent a new
PartitionDirectory concept and use it to ensure consistency. This
ensures that a single invocation of either the planner or the executor
sees the same view of the PartitionDesc from beginning to end, but it
does not guarantee that the planner and the executor see the same
view. Since this allows pointers to old PartitionDesc entries to
survive even after a relcache rebuild, also postpone removing the old
PartitionDesc entry until we're certain no one is using it.
For the most part, it seems to be OK for the planner and executor to
have different views of the PartitionDesc, because the executor will
just ignore any concurrently added partitions which were unknown at
plan time; those partitions won't be part of the inheritance
expansion, but invalidation messages will trigger replanning at some
point. Normally, this happens by the time the very next command is
executed, but if the next command acquires no locks and executes a
prepared query, it can manage not to notice until a new transaction is
started. We might want to tighten that up, but it's material for a
separate patch. There would still be a small window where a query
that started just after an ATTACH PARTITION command committed might
fail to notice its results -- but only if the command starts before
the commit has been acknowledged to the user. All in all, the warts
here around serializability seem small enough to be worth accepting
for the considerable advantage of being able to add partitions without
a full table lock.
Although in general the consequences of new partitions showing up
between planning and execution are limited to the query not noticing
the new partitions, run-time partition pruning will get confused in
that case, so that's the third problem that this patch fixes.
Run-time partition pruning assumes that indexes into the PartitionDesc
are stable between planning and execution. So, add code so that if
new partitions are added between plan time and execution time, the
indexes stored in the subplan_map[] and subpart_map[] arrays within
the plan's PartitionedRelPruneInfo get adjusted accordingly. There
does not seem to be a simple way to generalize this scheme to cope
with partitions that are removed, mostly because they could then get
added back again with different bounds, but it works OK for added
partitions.
This code does not try to ensure that every backend participating in
a parallel query sees the same view of the PartitionDesc. That
currently doesn't matter, because we never pass PartitionDesc
indexes between backends. Each backend will ignore the concurrently
added partitions which it notices, and it doesn't matter if different
backends are ignoring different sets of concurrently added partitions.
If in the future that matters, for example because we allow writes in
parallel query and want all participants to do tuple routing to the same
set of partitions, the PartitionDirectory concept could be improved to
share PartitionDescs across backends. There is a draft patch to
serialize and restore PartitionDescs on the thread where this patch
was discussed, which may be a useful place to start.
Patch by me. Thanks to Alvaro Herrera, David Rowley, Simon Riggs,
Amit Langote, and Michael Paquier for discussion, and to Alvaro
Herrera for some review.
Discussion: http://postgr.es/m/CA+Tgmobt2upbSocvvDej3yzokd7AkiT+PvgFH+a9-5VV1oJNSQ@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoZE0r9-cyA-aY6f8WFEROaDLLL7Vf81kZ8MtFCkxpeQSw@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoY13KQZF-=HNTrt9UYWYx3_oYOQpu9ioNT49jGgiDpUEA@mail.gmail.com
2019-03-07 17:13:12 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We are rebuilding a partitioned relation with a non-zero
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
* reference count, so we must keep the old partition descriptor
|
|
|
|
* around, in case there's a PartitionDirectory with a pointer to
|
|
|
|
* it. This means we can't free the old rd_pdcxt yet. (This is
|
|
|
|
* necessary because RelationGetPartitionDesc hands out direct
|
|
|
|
* pointers to the relcache's data structure, unlike our usual
|
|
|
|
* practice which is to hand out copies. We'd have the same
|
|
|
|
* problem with rd_partkey, except that we always preserve that
|
|
|
|
* once created.)
|
|
|
|
*
|
|
|
|
* To ensure that it's not leaked completely, re-attach it to the
|
|
|
|
* new reldesc, or make it a child of the new reldesc's rd_pdcxt
|
|
|
|
* in the unlikely event that there is one already. (Compare hack
|
|
|
|
* in RelationBuildPartitionDesc.) RelationClose will clean up
|
|
|
|
* any such contexts once the reference count reaches zero.
|
|
|
|
*
|
|
|
|
* In the case where the reference count is zero, this code is not
|
|
|
|
* reached, which should be OK because in that case there should
|
|
|
|
* be no PartitionDirectory with a pointer to the old entry.
|
Allow ATTACH PARTITION with only ShareUpdateExclusiveLock.
We still require AccessExclusiveLock on the partition itself, because
otherwise an insert that violates the newly-imposed partition
constraint could be in progress at the same time that we're changing
that constraint; only the lock level on the parent relation is
weakened.
To make this safe, we have to cope with (at least) three separate
problems. First, relevant DDL might commit while we're in the process
of building a PartitionDesc. If so, find_inheritance_children() might
see a new partition while the RELOID system cache still has the old
partition bound cached, and even before invalidation messages have
been queued. To fix that, if we see that the pg_class tuple seems to
be missing or to have a null relpartbound, refetch the value directly
from the table. We can't get the wrong value, because DETACH PARTITION
still requires AccessExclusiveLock throughout; if we ever want to
change that, this will need more thought. In testing, I found it quite
difficult to hit even the null-relpartbound case; the race condition
is extremely tight, but the theoretical risk is there.
Second, successive calls to RelationGetPartitionDesc might not return
the same answer. The query planner will get confused if lookup up the
PartitionDesc for a particular relation does not return a consistent
answer for the entire duration of query planning. Likewise, query
execution will get confused if the same relation seems to have a
different PartitionDesc at different times. Invent a new
PartitionDirectory concept and use it to ensure consistency. This
ensures that a single invocation of either the planner or the executor
sees the same view of the PartitionDesc from beginning to end, but it
does not guarantee that the planner and the executor see the same
view. Since this allows pointers to old PartitionDesc entries to
survive even after a relcache rebuild, also postpone removing the old
PartitionDesc entry until we're certain no one is using it.
For the most part, it seems to be OK for the planner and executor to
have different views of the PartitionDesc, because the executor will
just ignore any concurrently added partitions which were unknown at
plan time; those partitions won't be part of the inheritance
expansion, but invalidation messages will trigger replanning at some
point. Normally, this happens by the time the very next command is
executed, but if the next command acquires no locks and executes a
prepared query, it can manage not to notice until a new transaction is
started. We might want to tighten that up, but it's material for a
separate patch. There would still be a small window where a query
that started just after an ATTACH PARTITION command committed might
fail to notice its results -- but only if the command starts before
the commit has been acknowledged to the user. All in all, the warts
here around serializability seem small enough to be worth accepting
for the considerable advantage of being able to add partitions without
a full table lock.
Although in general the consequences of new partitions showing up
between planning and execution are limited to the query not noticing
the new partitions, run-time partition pruning will get confused in
that case, so that's the third problem that this patch fixes.
Run-time partition pruning assumes that indexes into the PartitionDesc
are stable between planning and execution. So, add code so that if
new partitions are added between plan time and execution time, the
indexes stored in the subplan_map[] and subpart_map[] arrays within
the plan's PartitionedRelPruneInfo get adjusted accordingly. There
does not seem to be a simple way to generalize this scheme to cope
with partitions that are removed, mostly because they could then get
added back again with different bounds, but it works OK for added
partitions.
This code does not try to ensure that every backend participating in
a parallel query sees the same view of the PartitionDesc. That
currently doesn't matter, because we never pass PartitionDesc
indexes between backends. Each backend will ignore the concurrently
added partitions which it notices, and it doesn't matter if different
backends are ignoring different sets of concurrently added partitions.
If in the future that matters, for example because we allow writes in
parallel query and want all participants to do tuple routing to the same
set of partitions, the PartitionDirectory concept could be improved to
share PartitionDescs across backends. There is a draft patch to
serialize and restore PartitionDescs on the thread where this patch
was discussed, which may be a useful place to start.
Patch by me. Thanks to Alvaro Herrera, David Rowley, Simon Riggs,
Amit Langote, and Michael Paquier for discussion, and to Alvaro
Herrera for some review.
Discussion: http://postgr.es/m/CA+Tgmobt2upbSocvvDej3yzokd7AkiT+PvgFH+a9-5VV1oJNSQ@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoZE0r9-cyA-aY6f8WFEROaDLLL7Vf81kZ8MtFCkxpeQSw@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoY13KQZF-=HNTrt9UYWYx3_oYOQpu9ioNT49jGgiDpUEA@mail.gmail.com
2019-03-07 17:13:12 +01:00
|
|
|
*
|
|
|
|
* Note that newrel and relation have already been swapped, so the
|
|
|
|
* "old" partition descriptor is actually the one hanging off of
|
|
|
|
* newrel.
|
|
|
|
*/
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
relation->rd_partdesc = NULL; /* ensure rd_partdesc is invalid */
|
2021-04-28 21:44:35 +02:00
|
|
|
relation->rd_partdesc_nodetached = NULL;
|
|
|
|
relation->rd_partdesc_nodetached_xmin = InvalidTransactionId;
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
if (relation->rd_pdcxt != NULL) /* probably never happens */
|
|
|
|
MemoryContextSetParent(newrel->rd_pdcxt, relation->rd_pdcxt);
|
|
|
|
else
|
|
|
|
relation->rd_pdcxt = newrel->rd_pdcxt;
|
2021-04-28 21:44:35 +02:00
|
|
|
if (relation->rd_pddcxt != NULL)
|
|
|
|
MemoryContextSetParent(newrel->rd_pddcxt, relation->rd_pddcxt);
|
|
|
|
else
|
|
|
|
relation->rd_pddcxt = newrel->rd_pddcxt;
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
/* drop newrel's pointers so we don't destroy it below */
|
Allow ATTACH PARTITION with only ShareUpdateExclusiveLock.
We still require AccessExclusiveLock on the partition itself, because
otherwise an insert that violates the newly-imposed partition
constraint could be in progress at the same time that we're changing
that constraint; only the lock level on the parent relation is
weakened.
To make this safe, we have to cope with (at least) three separate
problems. First, relevant DDL might commit while we're in the process
of building a PartitionDesc. If so, find_inheritance_children() might
see a new partition while the RELOID system cache still has the old
partition bound cached, and even before invalidation messages have
been queued. To fix that, if we see that the pg_class tuple seems to
be missing or to have a null relpartbound, refetch the value directly
from the table. We can't get the wrong value, because DETACH PARTITION
still requires AccessExclusiveLock throughout; if we ever want to
change that, this will need more thought. In testing, I found it quite
difficult to hit even the null-relpartbound case; the race condition
is extremely tight, but the theoretical risk is there.
Second, successive calls to RelationGetPartitionDesc might not return
the same answer. The query planner will get confused if lookup up the
PartitionDesc for a particular relation does not return a consistent
answer for the entire duration of query planning. Likewise, query
execution will get confused if the same relation seems to have a
different PartitionDesc at different times. Invent a new
PartitionDirectory concept and use it to ensure consistency. This
ensures that a single invocation of either the planner or the executor
sees the same view of the PartitionDesc from beginning to end, but it
does not guarantee that the planner and the executor see the same
view. Since this allows pointers to old PartitionDesc entries to
survive even after a relcache rebuild, also postpone removing the old
PartitionDesc entry until we're certain no one is using it.
For the most part, it seems to be OK for the planner and executor to
have different views of the PartitionDesc, because the executor will
just ignore any concurrently added partitions which were unknown at
plan time; those partitions won't be part of the inheritance
expansion, but invalidation messages will trigger replanning at some
point. Normally, this happens by the time the very next command is
executed, but if the next command acquires no locks and executes a
prepared query, it can manage not to notice until a new transaction is
started. We might want to tighten that up, but it's material for a
separate patch. There would still be a small window where a query
that started just after an ATTACH PARTITION command committed might
fail to notice its results -- but only if the command starts before
the commit has been acknowledged to the user. All in all, the warts
here around serializability seem small enough to be worth accepting
for the considerable advantage of being able to add partitions without
a full table lock.
Although in general the consequences of new partitions showing up
between planning and execution are limited to the query not noticing
the new partitions, run-time partition pruning will get confused in
that case, so that's the third problem that this patch fixes.
Run-time partition pruning assumes that indexes into the PartitionDesc
are stable between planning and execution. So, add code so that if
new partitions are added between plan time and execution time, the
indexes stored in the subplan_map[] and subpart_map[] arrays within
the plan's PartitionedRelPruneInfo get adjusted accordingly. There
does not seem to be a simple way to generalize this scheme to cope
with partitions that are removed, mostly because they could then get
added back again with different bounds, but it works OK for added
partitions.
This code does not try to ensure that every backend participating in
a parallel query sees the same view of the PartitionDesc. That
currently doesn't matter, because we never pass PartitionDesc
indexes between backends. Each backend will ignore the concurrently
added partitions which it notices, and it doesn't matter if different
backends are ignoring different sets of concurrently added partitions.
If in the future that matters, for example because we allow writes in
parallel query and want all participants to do tuple routing to the same
set of partitions, the PartitionDirectory concept could be improved to
share PartitionDescs across backends. There is a draft patch to
serialize and restore PartitionDescs on the thread where this patch
was discussed, which may be a useful place to start.
Patch by me. Thanks to Alvaro Herrera, David Rowley, Simon Riggs,
Amit Langote, and Michael Paquier for discussion, and to Alvaro
Herrera for some review.
Discussion: http://postgr.es/m/CA+Tgmobt2upbSocvvDej3yzokd7AkiT+PvgFH+a9-5VV1oJNSQ@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoZE0r9-cyA-aY6f8WFEROaDLLL7Vf81kZ8MtFCkxpeQSw@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoY13KQZF-=HNTrt9UYWYx3_oYOQpu9ioNT49jGgiDpUEA@mail.gmail.com
2019-03-07 17:13:12 +01:00
|
|
|
newrel->rd_partdesc = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
newrel->rd_partdesc_nodetached = NULL;
|
|
|
|
newrel->rd_partdesc_nodetached_xmin = InvalidTransactionId;
|
Allow ATTACH PARTITION with only ShareUpdateExclusiveLock.
We still require AccessExclusiveLock on the partition itself, because
otherwise an insert that violates the newly-imposed partition
constraint could be in progress at the same time that we're changing
that constraint; only the lock level on the parent relation is
weakened.
To make this safe, we have to cope with (at least) three separate
problems. First, relevant DDL might commit while we're in the process
of building a PartitionDesc. If so, find_inheritance_children() might
see a new partition while the RELOID system cache still has the old
partition bound cached, and even before invalidation messages have
been queued. To fix that, if we see that the pg_class tuple seems to
be missing or to have a null relpartbound, refetch the value directly
from the table. We can't get the wrong value, because DETACH PARTITION
still requires AccessExclusiveLock throughout; if we ever want to
change that, this will need more thought. In testing, I found it quite
difficult to hit even the null-relpartbound case; the race condition
is extremely tight, but the theoretical risk is there.
Second, successive calls to RelationGetPartitionDesc might not return
the same answer. The query planner will get confused if lookup up the
PartitionDesc for a particular relation does not return a consistent
answer for the entire duration of query planning. Likewise, query
execution will get confused if the same relation seems to have a
different PartitionDesc at different times. Invent a new
PartitionDirectory concept and use it to ensure consistency. This
ensures that a single invocation of either the planner or the executor
sees the same view of the PartitionDesc from beginning to end, but it
does not guarantee that the planner and the executor see the same
view. Since this allows pointers to old PartitionDesc entries to
survive even after a relcache rebuild, also postpone removing the old
PartitionDesc entry until we're certain no one is using it.
For the most part, it seems to be OK for the planner and executor to
have different views of the PartitionDesc, because the executor will
just ignore any concurrently added partitions which were unknown at
plan time; those partitions won't be part of the inheritance
expansion, but invalidation messages will trigger replanning at some
point. Normally, this happens by the time the very next command is
executed, but if the next command acquires no locks and executes a
prepared query, it can manage not to notice until a new transaction is
started. We might want to tighten that up, but it's material for a
separate patch. There would still be a small window where a query
that started just after an ATTACH PARTITION command committed might
fail to notice its results -- but only if the command starts before
the commit has been acknowledged to the user. All in all, the warts
here around serializability seem small enough to be worth accepting
for the considerable advantage of being able to add partitions without
a full table lock.
Although in general the consequences of new partitions showing up
between planning and execution are limited to the query not noticing
the new partitions, run-time partition pruning will get confused in
that case, so that's the third problem that this patch fixes.
Run-time partition pruning assumes that indexes into the PartitionDesc
are stable between planning and execution. So, add code so that if
new partitions are added between plan time and execution time, the
indexes stored in the subplan_map[] and subpart_map[] arrays within
the plan's PartitionedRelPruneInfo get adjusted accordingly. There
does not seem to be a simple way to generalize this scheme to cope
with partitions that are removed, mostly because they could then get
added back again with different bounds, but it works OK for added
partitions.
This code does not try to ensure that every backend participating in
a parallel query sees the same view of the PartitionDesc. That
currently doesn't matter, because we never pass PartitionDesc
indexes between backends. Each backend will ignore the concurrently
added partitions which it notices, and it doesn't matter if different
backends are ignoring different sets of concurrently added partitions.
If in the future that matters, for example because we allow writes in
parallel query and want all participants to do tuple routing to the same
set of partitions, the PartitionDirectory concept could be improved to
share PartitionDescs across backends. There is a draft patch to
serialize and restore PartitionDescs on the thread where this patch
was discussed, which may be a useful place to start.
Patch by me. Thanks to Alvaro Herrera, David Rowley, Simon Riggs,
Amit Langote, and Michael Paquier for discussion, and to Alvaro
Herrera for some review.
Discussion: http://postgr.es/m/CA+Tgmobt2upbSocvvDej3yzokd7AkiT+PvgFH+a9-5VV1oJNSQ@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoZE0r9-cyA-aY6f8WFEROaDLLL7Vf81kZ8MtFCkxpeQSw@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoY13KQZF-=HNTrt9UYWYx3_oYOQpu9ioNT49jGgiDpUEA@mail.gmail.com
2019-03-07 17:13:12 +01:00
|
|
|
newrel->rd_pdcxt = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
newrel->rd_pddcxt = NULL;
|
Allow ATTACH PARTITION with only ShareUpdateExclusiveLock.
We still require AccessExclusiveLock on the partition itself, because
otherwise an insert that violates the newly-imposed partition
constraint could be in progress at the same time that we're changing
that constraint; only the lock level on the parent relation is
weakened.
To make this safe, we have to cope with (at least) three separate
problems. First, relevant DDL might commit while we're in the process
of building a PartitionDesc. If so, find_inheritance_children() might
see a new partition while the RELOID system cache still has the old
partition bound cached, and even before invalidation messages have
been queued. To fix that, if we see that the pg_class tuple seems to
be missing or to have a null relpartbound, refetch the value directly
from the table. We can't get the wrong value, because DETACH PARTITION
still requires AccessExclusiveLock throughout; if we ever want to
change that, this will need more thought. In testing, I found it quite
difficult to hit even the null-relpartbound case; the race condition
is extremely tight, but the theoretical risk is there.
Second, successive calls to RelationGetPartitionDesc might not return
the same answer. The query planner will get confused if lookup up the
PartitionDesc for a particular relation does not return a consistent
answer for the entire duration of query planning. Likewise, query
execution will get confused if the same relation seems to have a
different PartitionDesc at different times. Invent a new
PartitionDirectory concept and use it to ensure consistency. This
ensures that a single invocation of either the planner or the executor
sees the same view of the PartitionDesc from beginning to end, but it
does not guarantee that the planner and the executor see the same
view. Since this allows pointers to old PartitionDesc entries to
survive even after a relcache rebuild, also postpone removing the old
PartitionDesc entry until we're certain no one is using it.
For the most part, it seems to be OK for the planner and executor to
have different views of the PartitionDesc, because the executor will
just ignore any concurrently added partitions which were unknown at
plan time; those partitions won't be part of the inheritance
expansion, but invalidation messages will trigger replanning at some
point. Normally, this happens by the time the very next command is
executed, but if the next command acquires no locks and executes a
prepared query, it can manage not to notice until a new transaction is
started. We might want to tighten that up, but it's material for a
separate patch. There would still be a small window where a query
that started just after an ATTACH PARTITION command committed might
fail to notice its results -- but only if the command starts before
the commit has been acknowledged to the user. All in all, the warts
here around serializability seem small enough to be worth accepting
for the considerable advantage of being able to add partitions without
a full table lock.
Although in general the consequences of new partitions showing up
between planning and execution are limited to the query not noticing
the new partitions, run-time partition pruning will get confused in
that case, so that's the third problem that this patch fixes.
Run-time partition pruning assumes that indexes into the PartitionDesc
are stable between planning and execution. So, add code so that if
new partitions are added between plan time and execution time, the
indexes stored in the subplan_map[] and subpart_map[] arrays within
the plan's PartitionedRelPruneInfo get adjusted accordingly. There
does not seem to be a simple way to generalize this scheme to cope
with partitions that are removed, mostly because they could then get
added back again with different bounds, but it works OK for added
partitions.
This code does not try to ensure that every backend participating in
a parallel query sees the same view of the PartitionDesc. That
currently doesn't matter, because we never pass PartitionDesc
indexes between backends. Each backend will ignore the concurrently
added partitions which it notices, and it doesn't matter if different
backends are ignoring different sets of concurrently added partitions.
If in the future that matters, for example because we allow writes in
parallel query and want all participants to do tuple routing to the same
set of partitions, the PartitionDirectory concept could be improved to
share PartitionDescs across backends. There is a draft patch to
serialize and restore PartitionDescs on the thread where this patch
was discussed, which may be a useful place to start.
Patch by me. Thanks to Alvaro Herrera, David Rowley, Simon Riggs,
Amit Langote, and Michael Paquier for discussion, and to Alvaro
Herrera for some review.
Discussion: http://postgr.es/m/CA+Tgmobt2upbSocvvDej3yzokd7AkiT+PvgFH+a9-5VV1oJNSQ@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoZE0r9-cyA-aY6f8WFEROaDLLL7Vf81kZ8MtFCkxpeQSw@mail.gmail.com
Discussion: http://postgr.es/m/CA+TgmoY13KQZF-=HNTrt9UYWYx3_oYOQpu9ioNT49jGgiDpUEA@mail.gmail.com
2019-03-07 17:13:12 +01:00
|
|
|
}
|
2010-01-12 19:12:18 +01:00
|
|
|
|
|
|
|
#undef SWAPFIELD
|
|
|
|
|
|
|
|
/* And now we can throw away the temporary entry */
|
2014-04-06 17:13:43 +02:00
|
|
|
RelationDestroyRelation(newrel, !keep_tupdesc);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
1999-10-04 01:55:40 +02:00
|
|
|
* RelationFlushRelation
|
|
|
|
*
|
|
|
|
* Rebuild the relation if it is open (refcount > 0), else blow it away.
|
2014-02-07 01:37:58 +01:00
|
|
|
* This is used when we receive a cache invalidation event for the rel.
|
1999-10-04 01:55:40 +02:00
|
|
|
*/
|
|
|
|
static void
|
2001-01-02 05:33:24 +01:00
|
|
|
RelationFlushRelation(Relation relation)
|
1999-10-04 01:55:40 +02:00
|
|
|
{
|
2007-01-25 03:17:26 +01:00
|
|
|
if (relation->rd_createSubid != InvalidSubTransactionId ||
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid != InvalidSubTransactionId)
|
2000-01-31 05:35:57 +01:00
|
|
|
{
|
|
|
|
/*
|
2002-08-11 23:17:35 +02:00
|
|
|
* New relcache entries are always rebuilt, not flushed; else we'd
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* forget the "new" status of the relation. Ditto for the
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* new-relfilenumber status.
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
*
|
|
|
|
* The rel could have zero refcnt here, so temporarily increment the
|
|
|
|
* refcnt to ensure it's safe to rebuild it. We can assume that the
|
|
|
|
* current transaction has some lock on the rel already.
|
2000-01-31 05:35:57 +01:00
|
|
|
*/
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
RelationIncrementReferenceCount(relation);
|
|
|
|
RelationClearRelation(relation, true);
|
|
|
|
RelationDecrementReferenceCount(relation);
|
2000-01-31 05:35:57 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
2002-08-11 23:17:35 +02:00
|
|
|
* Pre-existing rels can be dropped from the relcache if not open.
|
2000-01-31 05:35:57 +01:00
|
|
|
*/
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
bool rebuild = !RelationHasReferenceCountZero(relation);
|
1999-10-04 01:55:40 +02:00
|
|
|
|
Fix a problem introduced by my patch of 2010-01-12 that revised the way
relcache reload works. In the patched code, a relcache entry in process of
being rebuilt doesn't get unhooked from the relcache hash table; which means
that if a cache flush occurs due to sinval queue overrun while we're
rebuilding it, the entry could get blown away by RelationCacheInvalidate,
resulting in crash or misbehavior. Fix by ensuring that an entry being
rebuilt has positive refcount, so it won't be seen as a target for removal
if a cache flush occurs. (This will mean that the entry gets rebuilt twice
in such a scenario, but that's okay.) It appears that the problem can only
arise within a transaction that has previously reassigned the relfilenode of
a pre-existing table, via TRUNCATE or a similar operation. Per bug #5412
from Rusty Conover.
Back-patch to 8.2, same as the patch that introduced the problem.
I think that the failure can't actually occur in 8.2, since it lacks the
rd_newRelfilenodeSubid optimization, but let's make it work like the later
branches anyway.
Patch by Heikki, slightly editorialized on by me.
2010-04-14 23:31:11 +02:00
|
|
|
RelationClearRelation(relation, rebuild);
|
|
|
|
}
|
1999-10-04 01:55:40 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* RelationForgetRelation - caller reports that it dropped the relation
|
1997-06-04 10:56:51 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationForgetRelation(Oid rid)
|
|
|
|
{
|
|
|
|
Relation relation;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1997-06-04 10:56:51 +02:00
|
|
|
RelationIdCacheLookup(rid, relation);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-05-22 17:57:40 +02:00
|
|
|
if (!PointerIsValid(relation))
|
|
|
|
return; /* not in cache, nothing to do */
|
|
|
|
|
|
|
|
if (!RelationHasReferenceCountZero(relation))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "relation %u is still open", rid);
|
2002-05-22 17:57:40 +02:00
|
|
|
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
Assert(relation->rd_droppedSubid == InvalidSubTransactionId);
|
|
|
|
if (relation->rd_createSubid != InvalidSubTransactionId ||
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid != InvalidSubTransactionId)
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* In the event of subtransaction rollback, we must not forget
|
|
|
|
* rd_*Subid. Mark the entry "dropped" so RelationClearRelation()
|
|
|
|
* invalidates it in lieu of destroying it. (If we're in a top
|
|
|
|
* transaction, we could opt to destroy the entry.)
|
|
|
|
*/
|
|
|
|
relation->rd_droppedSubid = GetCurrentSubTransactionId();
|
|
|
|
}
|
|
|
|
|
2002-05-22 17:57:40 +02:00
|
|
|
RelationClearRelation(relation, false);
|
1997-06-04 10:56:51 +02:00
|
|
|
}
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2004-02-10 02:55:27 +01:00
|
|
|
* RelationCacheInvalidateEntry
|
2000-01-31 05:35:57 +01:00
|
|
|
*
|
|
|
|
* This routine is invoked for SI cache flush messages.
|
|
|
|
*
|
2004-02-10 02:55:27 +01:00
|
|
|
* Any relcache entry matching the relid must be flushed. (Note: caller has
|
|
|
|
* already determined that the relid belongs to our database or is a shared
|
2005-01-10 21:02:24 +01:00
|
|
|
* relation.)
|
2004-02-10 02:55:27 +01:00
|
|
|
*
|
|
|
|
* We used to skip local relations, on the grounds that they could
|
|
|
|
* not be targets of cross-backend SI update messages; but it seems
|
|
|
|
* safer to process them, so that our *own* SI update messages will
|
|
|
|
* have the same effects during CommandCounterIncrement for both
|
|
|
|
* local and nonlocal relations.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
void
|
2005-01-10 21:02:24 +01:00
|
|
|
RelationCacheInvalidateEntry(Oid relationId)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
Relation relation;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
RelationIdCacheLookup(relationId, relation);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-01-31 05:35:57 +01:00
|
|
|
if (PointerIsValid(relation))
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
relcacheInvalsReceived++;
|
2001-01-02 05:33:24 +01:00
|
|
|
RelationFlushRelation(relation);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
2021-10-24 03:36:38 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < in_progress_list_len; i++)
|
|
|
|
if (in_progress_list[i].reloid == relationId)
|
|
|
|
in_progress_list[i].invalidated = true;
|
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RelationCacheInvalidate
|
2000-01-31 05:35:57 +01:00
|
|
|
* Blow away cached relation descriptors that have zero reference counts,
|
2004-02-10 02:55:27 +01:00
|
|
|
* and rebuild those with positive reference counts. Also reset the smgr
|
2010-02-07 21:48:13 +01:00
|
|
|
* relation cache and re-read relation mapping data.
|
2000-01-29 20:51:59 +01:00
|
|
|
*
|
2021-10-24 03:36:38 +02:00
|
|
|
* Apart from debug_discard_caches, this is currently used only to recover
|
|
|
|
* from SI message buffer overflow, so we do not touch relations having
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* new-in-transaction relfilenumbers; they cannot be targets of cross-backend
|
2021-10-24 03:36:38 +02:00
|
|
|
* SI updates (and our own updates now go through a separate linked list
|
|
|
|
* that isn't limited by the SI message buffer size).
|
2001-01-02 05:33:24 +01:00
|
|
|
*
|
|
|
|
* We do this in two phases: the first pass deletes deletable items, and
|
|
|
|
* the second one rebuilds the rebuildable items. This is essential for
|
2001-10-05 19:28:13 +02:00
|
|
|
* safety, because hash_seq_search only copes with concurrent deletion of
|
2001-01-02 05:33:24 +01:00
|
|
|
* the element it is currently visiting. If a second SI overflow were to
|
|
|
|
* occur while we are walking the table, resulting in recursive entry to
|
|
|
|
* this routine, we could crash because the inner invocation blows away
|
|
|
|
* the entry next to be visited by the outer scan. But this way is OK,
|
|
|
|
* because (a) during the first pass we won't process any more SI messages,
|
2001-10-05 19:28:13 +02:00
|
|
|
* so hash_seq_search will complete safely; (b) during the second pass we
|
2001-01-02 05:33:24 +01:00
|
|
|
* only hold onto pointers to nondeletable entries.
|
2003-09-24 20:54:02 +02:00
|
|
|
*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* The two-phase approach also makes it easy to update relfilenumbers for
|
2011-08-16 20:38:20 +02:00
|
|
|
* mapped relations before we do anything else, and to ensure that the
|
|
|
|
* second pass processes nailed-in-cache items before other nondeletable
|
|
|
|
* items. This should ensure that system catalogs are up to date before
|
|
|
|
* we attempt to use them to reload information about other open relations.
|
2021-10-24 03:36:38 +02:00
|
|
|
*
|
|
|
|
* After those two phases of work having immediate effects, we normally
|
|
|
|
* signal any RelationBuildDesc() on the stack to start over. However, we
|
|
|
|
* don't do this if called as part of debug_discard_caches. Otherwise,
|
|
|
|
* RelationBuildDesc() would become an infinite loop.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
void
|
2021-10-24 03:36:38 +02:00
|
|
|
RelationCacheInvalidate(bool debug_discard)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2001-10-05 19:28:13 +02:00
|
|
|
HASH_SEQ_STATUS status;
|
2002-03-26 20:17:02 +01:00
|
|
|
RelIdCacheEnt *idhentry;
|
2001-10-05 19:28:13 +02:00
|
|
|
Relation relation;
|
2003-09-24 20:54:02 +02:00
|
|
|
List *rebuildFirstList = NIL;
|
2001-01-02 05:33:24 +01:00
|
|
|
List *rebuildList = NIL;
|
2004-05-26 06:41:50 +02:00
|
|
|
ListCell *l;
|
2021-10-24 03:36:38 +02:00
|
|
|
int i;
|
2001-01-02 05:33:24 +01:00
|
|
|
|
2011-08-16 20:38:20 +02:00
|
|
|
/*
|
|
|
|
* Reload relation mapping data before starting to reconstruct cache.
|
|
|
|
*/
|
|
|
|
RelationMapInvalidateAll();
|
|
|
|
|
2001-01-02 05:33:24 +01:00
|
|
|
/* Phase 1 */
|
2002-03-26 20:17:02 +01:00
|
|
|
hash_seq_init(&status, RelationIdCache);
|
2001-01-02 05:33:24 +01:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
2001-01-02 05:33:24 +01:00
|
|
|
{
|
2002-03-26 20:17:02 +01:00
|
|
|
relation = idhentry->reldesc;
|
2001-01-02 05:33:24 +01:00
|
|
|
|
2004-02-10 02:55:27 +01:00
|
|
|
/* Must close all smgr references to avoid leaving dangling ptrs */
|
2005-01-10 21:02:24 +01:00
|
|
|
RelationCloseSmgr(relation);
|
2004-02-10 02:55:27 +01:00
|
|
|
|
2012-12-24 17:43:22 +01:00
|
|
|
/*
|
|
|
|
* Ignore new relations; no other backend will manipulate them before
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* we commit. Likewise, before replacing a relation's relfilelocator,
|
|
|
|
* we shall have acquired AccessExclusiveLock and drained any
|
|
|
|
* applicable pending invalidations.
|
2012-12-24 17:43:22 +01:00
|
|
|
*/
|
|
|
|
if (relation->rd_createSubid != InvalidSubTransactionId ||
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid != InvalidSubTransactionId)
|
2001-10-05 19:28:13 +02:00
|
|
|
continue;
|
2001-01-02 05:33:24 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
relcacheInvalsReceived++;
|
|
|
|
|
2002-08-11 23:17:35 +02:00
|
|
|
if (RelationHasReferenceCountZero(relation))
|
2001-10-05 19:28:13 +02:00
|
|
|
{
|
|
|
|
/* Delete this entry immediately */
|
2003-09-24 20:54:02 +02:00
|
|
|
Assert(!relation->rd_isnailed);
|
2001-10-05 19:28:13 +02:00
|
|
|
RelationClearRelation(relation, false);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2011-08-16 20:38:20 +02:00
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* If it's a mapped relation, immediately update its rd_locator in
|
|
|
|
* case its relfilenumber changed. We must do this during phase 1
|
2011-08-16 20:38:20 +02:00
|
|
|
* in case the relation is consulted during rebuild of other
|
|
|
|
* relcache entries in phase 2. It's safe since consulting the
|
|
|
|
* map doesn't involve any access to relcache entries.
|
|
|
|
*/
|
|
|
|
if (RelationIsMapped(relation))
|
|
|
|
RelationInitPhysicalAddr(relation);
|
|
|
|
|
2003-09-24 20:54:02 +02:00
|
|
|
/*
|
|
|
|
* Add this entry to list of stuff to rebuild in second pass.
|
2011-08-16 20:38:20 +02:00
|
|
|
* pg_class goes to the front of rebuildFirstList while
|
|
|
|
* pg_class_oid_index goes to the back of rebuildFirstList, so
|
|
|
|
* they are done first and second respectively. Other nailed
|
|
|
|
* relations go to the front of rebuildList, so they'll be done
|
|
|
|
* next in no particular order; and everything else goes to the
|
|
|
|
* back of rebuildList.
|
2003-09-24 20:54:02 +02:00
|
|
|
*/
|
2011-08-16 20:38:20 +02:00
|
|
|
if (RelationGetRelid(relation) == RelationRelationId)
|
|
|
|
rebuildFirstList = lcons(relation, rebuildFirstList);
|
|
|
|
else if (RelationGetRelid(relation) == ClassOidIndexId)
|
|
|
|
rebuildFirstList = lappend(rebuildFirstList, relation);
|
|
|
|
else if (relation->rd_isnailed)
|
2003-09-24 20:54:02 +02:00
|
|
|
rebuildList = lcons(relation, rebuildList);
|
2011-08-16 20:38:20 +02:00
|
|
|
else
|
|
|
|
rebuildList = lappend(rebuildList, relation);
|
2001-10-05 19:28:13 +02:00
|
|
|
}
|
2001-01-02 05:33:24 +01:00
|
|
|
}
|
2001-10-05 19:28:13 +02:00
|
|
|
|
2004-02-10 02:55:27 +01:00
|
|
|
/*
|
|
|
|
* Now zap any remaining smgr cache entries. This must happen before we
|
|
|
|
* start to rebuild entries, since that may involve catalog fetches which
|
|
|
|
* will re-open catalog files.
|
|
|
|
*/
|
|
|
|
smgrcloseall();
|
|
|
|
|
2001-10-05 19:28:13 +02:00
|
|
|
/* Phase 2: rebuild the items found to need rebuild in phase 1 */
|
2006-01-19 01:27:08 +01:00
|
|
|
foreach(l, rebuildFirstList)
|
|
|
|
{
|
|
|
|
relation = (Relation) lfirst(l);
|
|
|
|
RelationClearRelation(relation, true);
|
|
|
|
}
|
|
|
|
list_free(rebuildFirstList);
|
2001-10-05 19:28:13 +02:00
|
|
|
foreach(l, rebuildList)
|
2001-01-02 05:33:24 +01:00
|
|
|
{
|
2001-10-05 19:28:13 +02:00
|
|
|
relation = (Relation) lfirst(l);
|
|
|
|
RelationClearRelation(relation, true);
|
2001-01-02 05:33:24 +01:00
|
|
|
}
|
2004-05-31 01:40:41 +02:00
|
|
|
list_free(rebuildList);
|
2021-10-24 03:36:38 +02:00
|
|
|
|
|
|
|
if (!debug_discard)
|
|
|
|
/* Any RelationBuildDesc() on the stack must start over. */
|
|
|
|
for (i = 0; i < in_progress_list_len; i++)
|
|
|
|
in_progress_list[i].invalidated = true;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
/*
|
|
|
|
* RelationCloseSmgrByOid - close a relcache entry's smgr link
|
|
|
|
*
|
|
|
|
* Needed in some cases where we are changing a relation's physical mapping.
|
|
|
|
* The link will be automatically reopened on next use.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationCloseSmgrByOid(Oid relationId)
|
|
|
|
{
|
|
|
|
Relation relation;
|
|
|
|
|
|
|
|
RelationIdCacheLookup(relationId, relation);
|
|
|
|
|
|
|
|
if (!PointerIsValid(relation))
|
|
|
|
return; /* not in cache, nothing to do */
|
|
|
|
|
|
|
|
RelationCloseSmgr(relation);
|
|
|
|
}
|
|
|
|
|
2014-05-17 23:57:53 +02:00
|
|
|
static void
|
2014-04-06 17:13:43 +02:00
|
|
|
RememberToFreeTupleDescAtEOX(TupleDesc td)
|
|
|
|
{
|
|
|
|
if (EOXactTupleDescArray == NULL)
|
|
|
|
{
|
|
|
|
MemoryContext oldcxt;
|
2014-05-06 18:12:18 +02:00
|
|
|
|
2014-04-06 17:13:43 +02:00
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
|
|
|
|
EOXactTupleDescArray = (TupleDesc *) palloc(16 * sizeof(TupleDesc));
|
|
|
|
EOXactTupleDescArrayLen = 16;
|
|
|
|
NextEOXactTupleDescNum = 0;
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
|
|
|
else if (NextEOXactTupleDescNum >= EOXactTupleDescArrayLen)
|
|
|
|
{
|
|
|
|
int32 newlen = EOXactTupleDescArrayLen * 2;
|
|
|
|
|
|
|
|
Assert(EOXactTupleDescArrayLen > 0);
|
|
|
|
|
|
|
|
EOXactTupleDescArray = (TupleDesc *) repalloc(EOXactTupleDescArray,
|
|
|
|
newlen * sizeof(TupleDesc));
|
|
|
|
EOXactTupleDescArrayLen = newlen;
|
|
|
|
}
|
|
|
|
|
|
|
|
EOXactTupleDescArray[NextEOXactTupleDescNum++] = td;
|
|
|
|
}
|
|
|
|
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
#ifdef USE_ASSERT_CHECKING
|
|
|
|
static void
|
|
|
|
AssertPendingSyncConsistency(Relation relation)
|
|
|
|
{
|
|
|
|
bool relcache_verdict =
|
2021-03-23 01:22:48 +01:00
|
|
|
RelationIsPermanent(relation) &&
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
((relation->rd_createSubid != InvalidSubTransactionId &&
|
|
|
|
RELKIND_HAS_STORAGE(relation->rd_rel->relkind)) ||
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid != InvalidSubTransactionId);
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
Assert(relcache_verdict == RelFileLocatorSkippingWAL(relation->rd_locator));
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
|
|
|
|
if (relation->rd_droppedSubid != InvalidSubTransactionId)
|
|
|
|
Assert(!relation->rd_isvalid &&
|
|
|
|
(relation->rd_createSubid != InvalidSubTransactionId ||
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid != InvalidSubTransactionId));
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* AssertPendingSyncs_RelationCache
|
|
|
|
*
|
|
|
|
* Assert that relcache.c and storage.c agree on whether to skip WAL.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
AssertPendingSyncs_RelationCache(void)
|
|
|
|
{
|
|
|
|
HASH_SEQ_STATUS status;
|
|
|
|
LOCALLOCK *locallock;
|
|
|
|
Relation *rels;
|
|
|
|
int maxrels;
|
|
|
|
int nrels;
|
|
|
|
RelIdCacheEnt *idhentry;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Open every relation that this transaction has locked. If, for some
|
|
|
|
* relation, storage.c is skipping WAL and relcache.c is not skipping WAL,
|
|
|
|
* a CommandCounterIncrement() typically yields a local invalidation
|
|
|
|
* message that destroys the relcache entry. By recreating such entries
|
|
|
|
* here, we detect the problem.
|
|
|
|
*/
|
|
|
|
PushActiveSnapshot(GetTransactionSnapshot());
|
|
|
|
maxrels = 1;
|
|
|
|
rels = palloc(maxrels * sizeof(*rels));
|
|
|
|
nrels = 0;
|
|
|
|
hash_seq_init(&status, GetLockMethodLocalHash());
|
|
|
|
while ((locallock = (LOCALLOCK *) hash_seq_search(&status)) != NULL)
|
|
|
|
{
|
|
|
|
Oid relid;
|
|
|
|
Relation r;
|
|
|
|
|
|
|
|
if (locallock->nLocks <= 0)
|
|
|
|
continue;
|
|
|
|
if ((LockTagType) locallock->tag.lock.locktag_type !=
|
|
|
|
LOCKTAG_RELATION)
|
|
|
|
continue;
|
|
|
|
relid = ObjectIdGetDatum(locallock->tag.lock.locktag_field2);
|
|
|
|
r = RelationIdGetRelation(relid);
|
|
|
|
if (!RelationIsValid(r))
|
|
|
|
continue;
|
|
|
|
if (nrels >= maxrels)
|
|
|
|
{
|
|
|
|
maxrels *= 2;
|
|
|
|
rels = repalloc(rels, maxrels * sizeof(*rels));
|
|
|
|
}
|
|
|
|
rels[nrels++] = r;
|
|
|
|
}
|
|
|
|
|
|
|
|
hash_seq_init(&status, RelationIdCache);
|
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
|
|
|
AssertPendingSyncConsistency(idhentry->reldesc);
|
|
|
|
|
|
|
|
for (i = 0; i < nrels; i++)
|
|
|
|
RelationClose(rels[i]);
|
|
|
|
PopActiveSnapshot();
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
1999-09-04 20:42:15 +02:00
|
|
|
/*
|
2002-08-06 04:36:35 +02:00
|
|
|
* AtEOXact_RelationCache
|
1999-09-04 20:42:15 +02:00
|
|
|
*
|
2004-07-17 05:32:14 +02:00
|
|
|
* Clean up the relcache at main-transaction commit or abort.
|
2003-09-24 20:54:02 +02:00
|
|
|
*
|
|
|
|
* Note: this must be called *before* processing invalidation messages.
|
|
|
|
* In the case of abort, we don't want to try to rebuild any invalidated
|
|
|
|
* cache entries (since we can't safely do database accesses). Therefore
|
|
|
|
* we must reset refcnts before handling pending invalidations.
|
2005-08-08 21:17:23 +02:00
|
|
|
*
|
|
|
|
* As of PostgreSQL 8.1, relcache refcnts should get released by the
|
|
|
|
* ResourceOwner mechanism. This routine just does a debugging
|
|
|
|
* cross-check that no pins remain. However, we also need to do special
|
|
|
|
* cleanup when the current transaction created any relations or made use
|
|
|
|
* of forced index lists.
|
1999-09-04 20:42:15 +02:00
|
|
|
*/
|
|
|
|
void
|
2004-07-01 02:52:04 +02:00
|
|
|
AtEOXact_RelationCache(bool isCommit)
|
1999-09-04 20:42:15 +02:00
|
|
|
{
|
2001-10-05 19:28:13 +02:00
|
|
|
HASH_SEQ_STATUS status;
|
2002-03-26 20:17:02 +01:00
|
|
|
RelIdCacheEnt *idhentry;
|
2013-01-20 19:44:49 +01:00
|
|
|
int i;
|
1999-09-04 20:42:15 +02:00
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/*
|
|
|
|
* Forget in_progress_list. This is relevant when we're aborting due to
|
|
|
|
* an error during RelationBuildDesc().
|
|
|
|
*/
|
|
|
|
Assert(in_progress_list_len == 0 || !isCommit);
|
|
|
|
in_progress_list_len = 0;
|
|
|
|
|
2005-08-08 21:17:23 +02:00
|
|
|
/*
|
2013-01-20 19:44:49 +01:00
|
|
|
* Unless the eoxact_list[] overflowed, we only need to examine the rels
|
|
|
|
* listed in it. Otherwise fall back on a hash_seq_search scan.
|
|
|
|
*
|
|
|
|
* For simplicity, eoxact_list[] entries are not deleted till end of
|
|
|
|
* top-level transaction, even though we could remove them at
|
|
|
|
* subtransaction end in some cases, or remove relations from the list if
|
|
|
|
* they are cleared for other reasons. Therefore we should expect the
|
|
|
|
* case that list entries are not found in the hashtable; if not, there's
|
|
|
|
* nothing to do for them.
|
|
|
|
*/
|
|
|
|
if (eoxact_list_overflowed)
|
2001-10-05 19:28:13 +02:00
|
|
|
{
|
2013-01-20 19:44:49 +01:00
|
|
|
hash_seq_init(&status, RelationIdCache);
|
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
|
|
|
{
|
|
|
|
AtEOXact_cleanup(idhentry->reldesc, isCommit);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
for (i = 0; i < eoxact_list_len; i++)
|
|
|
|
{
|
|
|
|
idhentry = (RelIdCacheEnt *) hash_search(RelationIdCache,
|
2023-02-06 09:05:20 +01:00
|
|
|
&eoxact_list[i],
|
2013-01-20 19:44:49 +01:00
|
|
|
HASH_FIND,
|
|
|
|
NULL);
|
|
|
|
if (idhentry != NULL)
|
|
|
|
AtEOXact_cleanup(idhentry->reldesc, isCommit);
|
|
|
|
}
|
|
|
|
}
|
2005-08-08 21:17:23 +02:00
|
|
|
|
2014-04-06 17:13:43 +02:00
|
|
|
if (EOXactTupleDescArrayLen > 0)
|
|
|
|
{
|
|
|
|
Assert(EOXactTupleDescArray != NULL);
|
|
|
|
for (i = 0; i < NextEOXactTupleDescNum; i++)
|
|
|
|
FreeTupleDesc(EOXactTupleDescArray[i]);
|
|
|
|
pfree(EOXactTupleDescArray);
|
|
|
|
EOXactTupleDescArray = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Now we're out of the transaction and can clear the lists */
|
2013-01-20 19:44:49 +01:00
|
|
|
eoxact_list_len = 0;
|
|
|
|
eoxact_list_overflowed = false;
|
2014-04-06 17:13:43 +02:00
|
|
|
NextEOXactTupleDescNum = 0;
|
|
|
|
EOXactTupleDescArrayLen = 0;
|
2013-01-20 19:44:49 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* AtEOXact_cleanup
|
|
|
|
*
|
|
|
|
* Clean up a single rel at main-transaction commit or abort
|
|
|
|
*
|
|
|
|
* NB: this processing must be idempotent, because EOXactListAdd() doesn't
|
|
|
|
* bother to prevent duplicate entries in eoxact_list[].
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
AtEOXact_cleanup(Relation relation, bool isCommit)
|
|
|
|
{
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
bool clear_relcache = false;
|
|
|
|
|
2005-08-08 21:17:23 +02:00
|
|
|
/*
|
|
|
|
* The relcache entry's ref count should be back to its normal
|
|
|
|
* not-in-a-transaction state: 0 unless it's nailed in cache.
|
|
|
|
*
|
|
|
|
* In bootstrap mode, this is NOT true, so don't check it --- the
|
|
|
|
* bootstrap code expects relations to stay open across start/commit
|
|
|
|
* transaction calls. (That seems bogus, but it's not worth fixing.)
|
2013-01-20 19:44:49 +01:00
|
|
|
*
|
|
|
|
* Note: ideally this check would be applied to every relcache entry, not
|
|
|
|
* just those that have eoxact work to do. But it's not worth forcing a
|
|
|
|
* scan of the whole relcache just for this. (Moreover, doing so would
|
|
|
|
* mean that assert-enabled testing never tests the hash_search code path
|
|
|
|
* above, which seems a bad idea.)
|
2005-08-08 21:17:23 +02:00
|
|
|
*/
|
|
|
|
#ifdef USE_ASSERT_CHECKING
|
|
|
|
if (!IsBootstrapProcessingMode())
|
|
|
|
{
|
|
|
|
int expected_refcnt;
|
|
|
|
|
|
|
|
expected_refcnt = relation->rd_isnailed ? 1 : 0;
|
|
|
|
Assert(relation->rd_refcnt == expected_refcnt);
|
|
|
|
}
|
|
|
|
#endif
|
2002-08-03 00:36:05 +02:00
|
|
|
|
2002-08-06 04:36:35 +02:00
|
|
|
/*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* Is the relation live after this transaction ends?
|
2002-08-06 04:36:35 +02:00
|
|
|
*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* During commit, clear the relcache entry if it is preserved after
|
|
|
|
* relation drop, in order not to orphan the entry. During rollback,
|
|
|
|
* clear the relcache entry if the relation is created in the current
|
|
|
|
* transaction since it isn't interesting any longer once we are out of
|
|
|
|
* the transaction.
|
|
|
|
*/
|
|
|
|
clear_relcache =
|
|
|
|
(isCommit ?
|
|
|
|
relation->rd_droppedSubid != InvalidSubTransactionId :
|
|
|
|
relation->rd_createSubid != InvalidSubTransactionId);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Since we are now out of the transaction, reset the subids to zero. That
|
|
|
|
* also lets RelationClearRelation() drop the relcache entry.
|
2002-08-06 04:36:35 +02:00
|
|
|
*/
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
relation->rd_createSubid = InvalidSubTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
relation->rd_droppedSubid = InvalidSubTransactionId;
|
|
|
|
|
|
|
|
if (clear_relcache)
|
2013-05-29 22:58:43 +02:00
|
|
|
{
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
if (RelationHasReferenceCountZero(relation))
|
2002-08-06 04:36:35 +02:00
|
|
|
{
|
|
|
|
RelationClearRelation(relation, false);
|
2013-01-20 19:44:49 +01:00
|
|
|
return;
|
2002-08-06 04:36:35 +02:00
|
|
|
}
|
Fix subtransaction cleanup after an outer-subtransaction portal fails.
Formerly, we treated only portals created in the current subtransaction as
having failed during subtransaction abort. However, if the error occurred
while running a portal created in an outer subtransaction (ie, a cursor
declared before the last savepoint), that has to be considered broken too.
To allow reliable detection of which ones those are, add a bookkeeping
field to struct Portal that tracks the innermost subtransaction in which
each portal has actually been executed. (Without this, we'd end up
failing portals containing functions that had called the subtransaction,
thereby breaking plpgsql exception blocks completely.)
In addition, when we fail an outer-subtransaction Portal, transfer its
resources into the subtransaction's resource owner, so that they're
released early in cleanup of the subxact. This fixes a problem reported by
Jim Nasby in which a function executed in an outer-subtransaction cursor
could cause an Assert failure or crash by referencing a relation created
within the inner subtransaction.
The proximate cause of the Assert failure is that AtEOSubXact_RelationCache
assumed it could blow away a relcache entry without first checking that the
entry had zero refcount. That was a bad idea on its own terms, so add such
a check there, and to the similar coding in AtEOXact_RelationCache. This
provides an independent safety measure in case there are still ways to
provoke the situation despite the Portal-level changes.
This has been broken since subtransactions were invented, so back-patch
to all supported branches.
Tom Lane and Michael Paquier
2015-09-04 19:36:49 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Hmm, somewhere there's a (leaked?) reference to the relation.
|
|
|
|
* We daren't remove the entry for fear of dereferencing a
|
|
|
|
* dangling pointer later. Bleat, and mark it as not belonging to
|
|
|
|
* the current transaction. Hopefully it'll get cleaned up
|
|
|
|
* eventually. This must be just a WARNING to avoid
|
|
|
|
* error-during-error-recovery loops.
|
|
|
|
*/
|
|
|
|
elog(WARNING, "cannot remove relcache entry for \"%s\" because it has nonzero refcount",
|
|
|
|
RelationGetRelationName(relation));
|
|
|
|
}
|
2013-05-29 22:58:43 +02:00
|
|
|
}
|
1999-09-04 20:42:15 +02:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
|
2004-08-28 22:31:44 +02:00
|
|
|
/*
|
|
|
|
* AtEOSubXact_RelationCache
|
|
|
|
*
|
|
|
|
* Clean up the relcache at sub-transaction commit or abort.
|
|
|
|
*
|
|
|
|
* Note: this must be called *before* processing invalidation messages.
|
|
|
|
*/
|
|
|
|
void
|
2004-09-16 18:58:44 +02:00
|
|
|
AtEOSubXact_RelationCache(bool isCommit, SubTransactionId mySubid,
|
|
|
|
SubTransactionId parentSubid)
|
2004-08-28 22:31:44 +02:00
|
|
|
{
|
|
|
|
HASH_SEQ_STATUS status;
|
|
|
|
RelIdCacheEnt *idhentry;
|
2013-01-20 19:44:49 +01:00
|
|
|
int i;
|
2004-08-28 22:31:44 +02:00
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/*
|
|
|
|
* Forget in_progress_list. This is relevant when we're aborting due to
|
|
|
|
* an error during RelationBuildDesc(). We don't commit subtransactions
|
|
|
|
* during RelationBuildDesc().
|
|
|
|
*/
|
|
|
|
Assert(in_progress_list_len == 0 || !isCommit);
|
|
|
|
in_progress_list_len = 0;
|
|
|
|
|
2004-11-20 21:19:52 +01:00
|
|
|
/*
|
2013-01-20 19:44:49 +01:00
|
|
|
* Unless the eoxact_list[] overflowed, we only need to examine the rels
|
|
|
|
* listed in it. Otherwise fall back on a hash_seq_search scan. Same
|
|
|
|
* logic as in AtEOXact_RelationCache.
|
2004-11-20 21:19:52 +01:00
|
|
|
*/
|
2013-01-20 19:44:49 +01:00
|
|
|
if (eoxact_list_overflowed)
|
2004-08-28 22:31:44 +02:00
|
|
|
{
|
2013-01-20 19:44:49 +01:00
|
|
|
hash_seq_init(&status, RelationIdCache);
|
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
|
|
|
{
|
|
|
|
AtEOSubXact_cleanup(idhentry->reldesc, isCommit,
|
|
|
|
mySubid, parentSubid);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
for (i = 0; i < eoxact_list_len; i++)
|
|
|
|
{
|
|
|
|
idhentry = (RelIdCacheEnt *) hash_search(RelationIdCache,
|
2023-02-06 09:05:20 +01:00
|
|
|
&eoxact_list[i],
|
2013-01-20 19:44:49 +01:00
|
|
|
HASH_FIND,
|
|
|
|
NULL);
|
|
|
|
if (idhentry != NULL)
|
|
|
|
AtEOSubXact_cleanup(idhentry->reldesc, isCommit,
|
|
|
|
mySubid, parentSubid);
|
|
|
|
}
|
|
|
|
}
|
2004-08-28 22:31:44 +02:00
|
|
|
|
2013-01-20 19:44:49 +01:00
|
|
|
/* Don't reset the list; we still need more cleanup later */
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* AtEOSubXact_cleanup
|
|
|
|
*
|
|
|
|
* Clean up a single rel at subtransaction commit or abort
|
|
|
|
*
|
|
|
|
* NB: this processing must be idempotent, because EOXactListAdd() doesn't
|
|
|
|
* bother to prevent duplicate entries in eoxact_list[].
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
AtEOSubXact_cleanup(Relation relation, bool isCommit,
|
|
|
|
SubTransactionId mySubid, SubTransactionId parentSubid)
|
|
|
|
{
|
2004-08-28 22:31:44 +02:00
|
|
|
/*
|
|
|
|
* Is it a relation created in the current subtransaction?
|
|
|
|
*
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* During subcommit, mark it as belonging to the parent, instead, as long
|
|
|
|
* as it has not been dropped. Otherwise simply delete the relcache entry.
|
|
|
|
* --- it isn't interesting any longer.
|
2004-08-28 22:31:44 +02:00
|
|
|
*/
|
2004-09-16 18:58:44 +02:00
|
|
|
if (relation->rd_createSubid == mySubid)
|
2013-05-29 22:58:43 +02:00
|
|
|
{
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/*
|
|
|
|
* Valid rd_droppedSubid means the corresponding relation is dropped
|
|
|
|
* but the relcache entry is preserved for at-commit pending sync. We
|
|
|
|
* need to drop it explicitly here not to make the entry orphan.
|
|
|
|
*/
|
|
|
|
Assert(relation->rd_droppedSubid == mySubid ||
|
|
|
|
relation->rd_droppedSubid == InvalidSubTransactionId);
|
|
|
|
if (isCommit && relation->rd_droppedSubid == InvalidSubTransactionId)
|
2004-09-16 18:58:44 +02:00
|
|
|
relation->rd_createSubid = parentSubid;
|
Fix subtransaction cleanup after an outer-subtransaction portal fails.
Formerly, we treated only portals created in the current subtransaction as
having failed during subtransaction abort. However, if the error occurred
while running a portal created in an outer subtransaction (ie, a cursor
declared before the last savepoint), that has to be considered broken too.
To allow reliable detection of which ones those are, add a bookkeeping
field to struct Portal that tracks the innermost subtransaction in which
each portal has actually been executed. (Without this, we'd end up
failing portals containing functions that had called the subtransaction,
thereby breaking plpgsql exception blocks completely.)
In addition, when we fail an outer-subtransaction Portal, transfer its
resources into the subtransaction's resource owner, so that they're
released early in cleanup of the subxact. This fixes a problem reported by
Jim Nasby in which a function executed in an outer-subtransaction cursor
could cause an Assert failure or crash by referencing a relation created
within the inner subtransaction.
The proximate cause of the Assert failure is that AtEOSubXact_RelationCache
assumed it could blow away a relcache entry without first checking that the
entry had zero refcount. That was a bad idea on its own terms, so add such
a check there, and to the similar coding in AtEOXact_RelationCache. This
provides an independent safety measure in case there are still ways to
provoke the situation despite the Portal-level changes.
This has been broken since subtransactions were invented, so back-patch
to all supported branches.
Tom Lane and Michael Paquier
2015-09-04 19:36:49 +02:00
|
|
|
else if (RelationHasReferenceCountZero(relation))
|
2004-08-28 22:31:44 +02:00
|
|
|
{
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/* allow the entry to be removed */
|
|
|
|
relation->rd_createSubid = InvalidSubTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
relation->rd_droppedSubid = InvalidSubTransactionId;
|
2004-08-28 22:31:44 +02:00
|
|
|
RelationClearRelation(relation, false);
|
2013-01-20 19:44:49 +01:00
|
|
|
return;
|
2004-08-28 22:31:44 +02:00
|
|
|
}
|
Fix subtransaction cleanup after an outer-subtransaction portal fails.
Formerly, we treated only portals created in the current subtransaction as
having failed during subtransaction abort. However, if the error occurred
while running a portal created in an outer subtransaction (ie, a cursor
declared before the last savepoint), that has to be considered broken too.
To allow reliable detection of which ones those are, add a bookkeeping
field to struct Portal that tracks the innermost subtransaction in which
each portal has actually been executed. (Without this, we'd end up
failing portals containing functions that had called the subtransaction,
thereby breaking plpgsql exception blocks completely.)
In addition, when we fail an outer-subtransaction Portal, transfer its
resources into the subtransaction's resource owner, so that they're
released early in cleanup of the subxact. This fixes a problem reported by
Jim Nasby in which a function executed in an outer-subtransaction cursor
could cause an Assert failure or crash by referencing a relation created
within the inner subtransaction.
The proximate cause of the Assert failure is that AtEOSubXact_RelationCache
assumed it could blow away a relcache entry without first checking that the
entry had zero refcount. That was a bad idea on its own terms, so add such
a check there, and to the similar coding in AtEOXact_RelationCache. This
provides an independent safety measure in case there are still ways to
provoke the situation despite the Portal-level changes.
This has been broken since subtransactions were invented, so back-patch
to all supported branches.
Tom Lane and Michael Paquier
2015-09-04 19:36:49 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Hmm, somewhere there's a (leaked?) reference to the relation.
|
|
|
|
* We daren't remove the entry for fear of dereferencing a
|
|
|
|
* dangling pointer later. Bleat, and transfer it to the parent
|
|
|
|
* subtransaction so we can try again later. This must be just a
|
|
|
|
* WARNING to avoid error-during-error-recovery loops.
|
|
|
|
*/
|
|
|
|
relation->rd_createSubid = parentSubid;
|
|
|
|
elog(WARNING, "cannot remove relcache entry for \"%s\" because it has nonzero refcount",
|
|
|
|
RelationGetRelationName(relation));
|
|
|
|
}
|
2013-05-29 22:58:43 +02:00
|
|
|
}
|
2007-03-29 02:15:39 +02:00
|
|
|
|
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* Likewise, update or drop any new-relfilenumber-in-subtransaction record
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* or drop record.
|
2007-03-29 02:15:39 +02:00
|
|
|
*/
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
if (relation->rd_newRelfilelocatorSubid == mySubid)
|
2007-01-25 03:17:26 +01:00
|
|
|
{
|
|
|
|
if (isCommit)
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = parentSubid;
|
2007-01-25 03:17:26 +01:00
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
2007-01-25 03:17:26 +01:00
|
|
|
}
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
if (relation->rd_firstRelfilelocatorSubid == mySubid)
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
{
|
|
|
|
if (isCommit)
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid = parentSubid;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
if (relation->rd_droppedSubid == mySubid)
|
|
|
|
{
|
|
|
|
if (isCommit)
|
|
|
|
relation->rd_droppedSubid = parentSubid;
|
|
|
|
else
|
|
|
|
relation->rd_droppedSubid = InvalidSubTransactionId;
|
|
|
|
}
|
2004-08-28 22:31:44 +02:00
|
|
|
}
|
|
|
|
|
2007-03-29 02:15:39 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2001-06-29 23:08:25 +02:00
|
|
|
* RelationBuildLocalRelation
|
|
|
|
* Build a relcache entry for an about-to-be-created relation,
|
|
|
|
* and enter it into the relcache.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2001-06-29 23:08:25 +02:00
|
|
|
Relation
|
|
|
|
RelationBuildLocalRelation(const char *relname,
|
2002-03-26 20:17:02 +01:00
|
|
|
Oid relnamespace,
|
2001-06-29 23:08:25 +02:00
|
|
|
TupleDesc tupDesc,
|
2004-06-18 08:14:31 +02:00
|
|
|
Oid relid,
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
Oid accessmtd,
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelFileNumber relfilenumber,
|
2004-06-18 08:14:31 +02:00
|
|
|
Oid reltablespace,
|
2010-02-07 21:48:13 +01:00
|
|
|
bool shared_relation,
|
2010-12-13 18:34:26 +01:00
|
|
|
bool mapped_relation,
|
2012-06-14 15:47:30 +02:00
|
|
|
char relpersistence,
|
|
|
|
char relkind)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2001-06-29 23:08:25 +02:00
|
|
|
Relation rel;
|
1996-07-09 08:22:35 +02:00
|
|
|
MemoryContext oldcxt;
|
2001-06-29 23:08:25 +02:00
|
|
|
int natts = tupDesc->natts;
|
|
|
|
int i;
|
2002-11-15 18:18:49 +01:00
|
|
|
bool has_not_null;
|
2005-04-14 03:38:22 +02:00
|
|
|
bool nailit;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2022-10-28 09:19:06 +02:00
|
|
|
Assert(natts >= 0);
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2005-04-14 03:38:22 +02:00
|
|
|
/*
|
|
|
|
* check for creation of a rel that must be nailed in cache.
|
|
|
|
*
|
2009-08-12 22:53:31 +02:00
|
|
|
* XXX this list had better match the relations specially handled in
|
|
|
|
* RelationCacheInitializePhase2/3.
|
2005-04-14 03:38:22 +02:00
|
|
|
*/
|
|
|
|
switch (relid)
|
|
|
|
{
|
2009-08-12 22:53:31 +02:00
|
|
|
case DatabaseRelationId:
|
2010-04-21 01:48:47 +02:00
|
|
|
case AuthIdRelationId:
|
|
|
|
case AuthMemRelationId:
|
2005-04-14 03:38:22 +02:00
|
|
|
case RelationRelationId:
|
|
|
|
case AttributeRelationId:
|
|
|
|
case ProcedureRelationId:
|
|
|
|
case TypeRelationId:
|
|
|
|
nailit = true;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
nailit = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2006-07-31 22:09:10 +02:00
|
|
|
/*
|
|
|
|
* check that hardwired list of shared rels matches what's in the
|
|
|
|
* bootstrap .bki file. If you get a failure here during initdb, you
|
|
|
|
* probably need to fix IsSharedRelation() to match whatever you've done
|
|
|
|
* to the set of shared relations.
|
|
|
|
*/
|
|
|
|
if (shared_relation != IsSharedRelation(relid))
|
|
|
|
elog(ERROR, "shared_relation flag for \"%s\" does not match IsSharedRelation(%u)",
|
|
|
|
relname, relid);
|
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
/* Shared relations had better be mapped, too */
|
|
|
|
Assert(mapped_relation || !shared_relation);
|
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/*
|
|
|
|
* switch to the cache context to create the relcache entry.
|
|
|
|
*/
|
|
|
|
if (!CacheMemoryContext)
|
|
|
|
CreateCacheMemoryContext();
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2000-06-30 09:04:23 +02:00
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/*
|
2002-08-06 04:36:35 +02:00
|
|
|
* allocate a new relation descriptor and fill in basic state fields.
|
2001-06-29 23:08:25 +02:00
|
|
|
*/
|
2002-11-13 01:39:48 +01:00
|
|
|
rel = (Relation) palloc0(sizeof(RelationData));
|
2002-08-06 04:36:35 +02:00
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/* make sure relation is marked as having no open file yet */
|
2004-02-10 02:55:27 +01:00
|
|
|
rel->rd_smgr = NULL;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2005-04-14 03:38:22 +02:00
|
|
|
/* mark it nailed if appropriate */
|
|
|
|
rel->rd_isnailed = nailit;
|
|
|
|
|
2004-07-17 05:32:14 +02:00
|
|
|
rel->rd_refcnt = nailit ? 1 : 0;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2002-08-06 04:36:35 +02:00
|
|
|
/* it's being created in this transaction */
|
2004-09-16 18:58:44 +02:00
|
|
|
rel->rd_createSubid = GetCurrentSubTransactionId();
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
rel->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
rel->rd_droppedSubid = InvalidSubTransactionId;
|
2002-08-06 04:36:35 +02:00
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/*
|
2002-03-03 18:47:56 +01:00
|
|
|
* create a new tuple descriptor from the one passed in. We do this
|
|
|
|
* partly to copy it into the cache context, and partly because the new
|
|
|
|
* relation can't have any defaults or constraints yet; they have to be
|
|
|
|
* added in later steps, because they require additions to multiple system
|
|
|
|
* catalogs. We can copy attnotnull constraints here, however.
|
2001-06-29 23:08:25 +02:00
|
|
|
*/
|
2002-03-03 18:47:56 +01:00
|
|
|
rel->rd_att = CreateTupleDescCopy(tupDesc);
|
2006-06-16 20:42:24 +02:00
|
|
|
rel->rd_att->tdrefcount = 1; /* mark as refcounted */
|
2002-11-15 18:18:49 +01:00
|
|
|
has_not_null = false;
|
2002-03-03 18:47:56 +01:00
|
|
|
for (i = 0; i < natts; i++)
|
2002-11-15 18:18:49 +01:00
|
|
|
{
|
2017-08-20 20:19:07 +02:00
|
|
|
Form_pg_attribute satt = TupleDescAttr(tupDesc, i);
|
|
|
|
Form_pg_attribute datt = TupleDescAttr(rel->rd_att, i);
|
|
|
|
|
|
|
|
datt->attidentity = satt->attidentity;
|
2019-03-30 08:13:09 +01:00
|
|
|
datt->attgenerated = satt->attgenerated;
|
2017-08-20 20:19:07 +02:00
|
|
|
datt->attnotnull = satt->attnotnull;
|
|
|
|
has_not_null |= satt->attnotnull;
|
2002-11-15 18:18:49 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
if (has_not_null)
|
|
|
|
{
|
|
|
|
TupleConstr *constr = (TupleConstr *) palloc0(sizeof(TupleConstr));
|
|
|
|
|
|
|
|
constr->has_not_null = true;
|
|
|
|
rel->rd_att->constr = constr;
|
|
|
|
}
|
2001-06-29 23:08:25 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* initialize relation tuple form (caller may add/override data later)
|
|
|
|
*/
|
2002-11-13 01:39:48 +01:00
|
|
|
rel->rd_rel = (Form_pg_class) palloc0(CLASS_TUPLE_SIZE);
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
namestrcpy(&rel->rd_rel->relname, relname);
|
|
|
|
rel->rd_rel->relnamespace = relnamespace;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2012-06-14 15:47:30 +02:00
|
|
|
rel->rd_rel->relkind = relkind;
|
2001-06-29 23:08:25 +02:00
|
|
|
rel->rd_rel->relnatts = natts;
|
|
|
|
rel->rd_rel->reltype = InvalidOid;
|
2005-08-26 05:08:15 +02:00
|
|
|
/* needed when bootstrapping: */
|
|
|
|
rel->rd_rel->relowner = BOOTSTRAP_SUPERUSERID;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2012-12-18 02:15:32 +01:00
|
|
|
/* set up persistence and relcache fields dependent on it */
|
2010-12-13 18:34:26 +01:00
|
|
|
rel->rd_rel->relpersistence = relpersistence;
|
|
|
|
switch (relpersistence)
|
|
|
|
{
|
2010-12-29 12:48:53 +01:00
|
|
|
case RELPERSISTENCE_UNLOGGED:
|
2010-12-13 18:34:26 +01:00
|
|
|
case RELPERSISTENCE_PERMANENT:
|
|
|
|
rel->rd_backend = InvalidBackendId;
|
2012-12-18 02:15:32 +01:00
|
|
|
rel->rd_islocaltemp = false;
|
2010-12-13 18:34:26 +01:00
|
|
|
break;
|
|
|
|
case RELPERSISTENCE_TEMP:
|
2014-08-26 03:28:19 +02:00
|
|
|
Assert(isTempOrTempToastNamespace(relnamespace));
|
Improve the situation for parallel query versus temp relations.
Transmit the leader's temp-namespace state to workers. This is important
because without it, the workers do not really have the same search path
as the leader. For example, there is no good reason (and no extant code
either) to prevent a worker from executing a temp function that the
leader created previously; but as things stood it would fail to find the
temp function, and then either fail or execute the wrong function entirely.
We still prohibit a worker from creating a temp namespace on its own.
In effect, a worker can only see the session's temp namespace if the leader
had created it before starting the worker, which seems like the right
semantics.
Also, transmit the leader's BackendId to workers, and arrange for workers
to use that when determining the physical file path of a temp relation
belonging to their session. While the original intent was to prevent such
accesses entirely, there were a number of holes in that, notably in places
like dbsize.c which assume they can safely access temp rels of other
sessions anyway. We might as well get this right, as a small down payment
on someday allowing workers to access the leader's temp tables. (With
this change, directly using "MyBackendId" as a relation or buffer backend
ID is deprecated; you should use BackendIdForTempRelations() instead.
I left a couple of such uses alone though, as they're not going to be
reachable in parallel workers until we do something about localbuf.c.)
Move the thou-shalt-not-access-thy-leader's-temp-tables prohibition down
into localbuf.c, which is where it actually matters, instead of having it
in relation_open(). This amounts to recognizing that access to temp
tables' catalog entries is perfectly safe in a worker, it's only the data
in local buffers that is problematic.
Having done all that, we can get rid of the test in has_parallel_hazard()
that says that use of a temp table's rowtype is unsafe in parallel workers.
That test was unduly expensive, and if we really did need such a
prohibition, that was not even close to being a bulletproof guard for it.
(For example, any user-defined function executed in a parallel worker
might have attempted such access.)
2016-06-10 02:16:11 +02:00
|
|
|
rel->rd_backend = BackendIdForTempRelations();
|
2012-12-18 02:15:32 +01:00
|
|
|
rel->rd_islocaltemp = true;
|
2010-12-13 18:34:26 +01:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
elog(ERROR, "invalid relpersistence: %c", relpersistence);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2013-05-06 19:26:51 +02:00
|
|
|
/* if it's a materialized view, it's not populated initially */
|
|
|
|
if (relkind == RELKIND_MATVIEW)
|
|
|
|
rel->rd_rel->relispopulated = false;
|
|
|
|
else
|
|
|
|
rel->rd_rel->relispopulated = true;
|
|
|
|
|
Clean up the behavior and API of catalog.c's is-catalog-relation tests.
The right way for IsCatalogRelation/Class to behave is to return true
for OIDs less than FirstBootstrapObjectId (not FirstNormalObjectId),
without any of the ad-hoc fooling around with schema membership.
The previous code was wrong because (1) it claimed that
information_schema tables were not catalog relations but their toast
tables were, which is silly; and (2) if you dropped and recreated
information_schema, which is a supported operation, the behavior
changed. That's even sillier. With this definition, "catalog
relations" are exactly the ones traceable to the postgres.bki data,
which seems like what we want.
With this simplification, we don't actually need access to the pg_class
tuple to identify a catalog relation; we only need its OID. Hence,
replace IsCatalogClass with "IsCatalogRelationOid(oid)". But keep
IsCatalogRelation as a convenience function.
This allows fixing some arguably-wrong semantics in contrib/sepgsql and
ReindexRelationConcurrently, which were using an IsSystemNamespace test
where what they really should be using is IsCatalogRelationOid. The
previous coding failed to protect toast tables of system catalogs, and
also was not on board with the general principle that user-created tables
do not become catalogs just by virtue of being renamed into pg_catalog.
We can also get rid of a messy hack in ReindexMultipleTables.
While we're at it, also rename IsSystemNamespace to IsCatalogNamespace,
because the previous name invited confusion with the more expansive
semantics used by IsSystemRelation/Class.
Also improve the comments in catalog.c.
There are a few remaining places in replication-related code that are
special-casing OIDs below FirstNormalObjectId. I'm inclined to think
those are wrong too, and if there should be any special case it should
just extend to FirstBootstrapObjectId. But first we need to debate
whether a FOR ALL TABLES publication should include information_schema.
Discussion: https://postgr.es/m/21697.1557092753@sss.pgh.pa.us
Discussion: https://postgr.es/m/15150.1557257111@sss.pgh.pa.us
2019-05-09 05:27:29 +02:00
|
|
|
/* set replica identity -- system catalogs and non-tables don't have one */
|
|
|
|
if (!IsCatalogNamespace(relnamespace) &&
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
(relkind == RELKIND_RELATION ||
|
|
|
|
relkind == RELKIND_MATVIEW ||
|
|
|
|
relkind == RELKIND_PARTITIONED_TABLE))
|
2013-11-08 18:30:43 +01:00
|
|
|
rel->rd_rel->relreplident = REPLICA_IDENTITY_DEFAULT;
|
|
|
|
else
|
|
|
|
rel->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
|
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/*
|
2002-03-26 20:17:02 +01:00
|
|
|
* Insert relation physical and logical identifiers (OIDs) into the right
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* places. For a mapped relation, we set relfilenumber to zero and rely
|
|
|
|
* on RelationInitPhysicalAddr to consult the map.
|
2001-06-29 23:08:25 +02:00
|
|
|
*/
|
2004-06-18 08:14:31 +02:00
|
|
|
rel->rd_rel->relisshared = shared_relation;
|
2009-03-31 19:59:56 +02:00
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
RelationGetRelid(rel) = relid;
|
|
|
|
|
|
|
|
for (i = 0; i < natts; i++)
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(rel->rd_att, i)->attrelid = relid;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
rel->rd_rel->reltablespace = reltablespace;
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
if (mapped_relation)
|
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
rel->rd_rel->relfilenode = InvalidRelFileNumber;
|
2010-02-07 21:48:13 +01:00
|
|
|
/* Add it to the active mapping information */
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelationMapUpdateMap(relid, relfilenumber, shared_relation, true);
|
2010-02-07 21:48:13 +01:00
|
|
|
}
|
|
|
|
else
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
rel->rd_rel->relfilenode = relfilenumber;
|
2010-02-07 21:48:13 +01:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
RelationInitLockInfo(rel); /* see lmgr.c */
|
2001-06-29 23:08:25 +02:00
|
|
|
|
2004-06-18 08:14:31 +02:00
|
|
|
RelationInitPhysicalAddr(rel);
|
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
rel->rd_rel->relam = accessmtd;
|
|
|
|
|
2021-03-19 01:50:56 +01:00
|
|
|
/*
|
|
|
|
* RelationInitTableAccessMethod will do syscache lookups, so we mustn't
|
|
|
|
* run it in CacheMemoryContext. Fortunately, the remaining steps don't
|
|
|
|
* require a long-lived current context.
|
|
|
|
*/
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
2021-12-03 13:38:26 +01:00
|
|
|
if (RELKIND_HAS_TABLE_AM(relkind) || relkind == RELKIND_SEQUENCE)
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
RelationInitTableAccessMethod(rel);
|
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
/*
|
2014-05-19 00:17:55 +02:00
|
|
|
* Okay to insert into the relcache hash table.
|
|
|
|
*
|
|
|
|
* Ordinarily, there should certainly not be an existing hash entry for
|
|
|
|
* the same OID; but during bootstrap, when we create a "real" relcache
|
|
|
|
* entry for one of the bootstrap relations, we'll be overwriting the
|
|
|
|
* phony one created with formrdesc. So allow that to happen for nailed
|
|
|
|
* rels.
|
2001-06-29 23:08:25 +02:00
|
|
|
*/
|
2014-05-19 00:17:55 +02:00
|
|
|
RelationCacheInsert(rel, nailit);
|
1999-09-18 21:08:25 +02:00
|
|
|
|
2013-01-20 19:44:49 +01:00
|
|
|
/*
|
|
|
|
* Flag relation as needing eoxact cleanup (to clear rd_createSubid). We
|
|
|
|
* can't do this before storing relid in it.
|
|
|
|
*/
|
|
|
|
EOXactListAdd(rel);
|
|
|
|
|
2004-08-28 22:31:44 +02:00
|
|
|
/* It's fully valid */
|
|
|
|
rel->rd_isvalid = true;
|
|
|
|
|
2004-07-17 05:32:14 +02:00
|
|
|
/*
|
|
|
|
* Caller expects us to pin the returned entry.
|
|
|
|
*/
|
|
|
|
RelationIncrementReferenceCount(rel);
|
|
|
|
|
2001-06-29 23:08:25 +02:00
|
|
|
return rel;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2010-02-03 02:14:17 +01:00
|
|
|
|
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* RelationSetNewRelfilenumber
|
2010-02-03 02:14:17 +01:00
|
|
|
*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* Assign a new relfilenumber (physical file name), and possibly a new
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
* persistence setting, to the relation.
|
2010-02-03 02:14:17 +01:00
|
|
|
*
|
|
|
|
* This allows a full rewrite of the relation to be done with transactional
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* safety (since the filenumber assignment can be rolled back). Note however
|
2010-02-03 02:14:17 +01:00
|
|
|
* that there is no simple way to access the relation's old data for the
|
|
|
|
* remainder of the current transaction. This limits the usefulness to cases
|
|
|
|
* such as TRUNCATE or rebuilding an index from scratch.
|
|
|
|
*
|
|
|
|
* Caller must already hold exclusive lock on the relation.
|
|
|
|
*/
|
|
|
|
void
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelationSetNewRelfilenumber(Relation relation, char persistence)
|
2010-02-03 02:14:17 +01:00
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelFileNumber newrelfilenumber;
|
2010-02-03 02:14:17 +01:00
|
|
|
Relation pg_class;
|
|
|
|
HeapTuple tuple;
|
|
|
|
Form_pg_class classform;
|
2019-03-29 04:01:14 +01:00
|
|
|
MultiXactId minmulti = InvalidMultiXactId;
|
|
|
|
TransactionId freezeXid = InvalidTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelFileLocator newrlocator;
|
2010-02-03 02:14:17 +01:00
|
|
|
|
2022-07-28 22:03:42 +02:00
|
|
|
if (!IsBinaryUpgrade)
|
|
|
|
{
|
|
|
|
/* Allocate a new relfilenumber */
|
|
|
|
newrelfilenumber = GetNewRelFileNumber(relation->rd_rel->reltablespace,
|
2022-09-28 15:45:27 +02:00
|
|
|
NULL, persistence);
|
2022-07-28 22:03:42 +02:00
|
|
|
}
|
|
|
|
else if (relation->rd_rel->relkind == RELKIND_INDEX)
|
|
|
|
{
|
|
|
|
if (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenumber))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("index relfilenumber value not set when in binary upgrade mode")));
|
|
|
|
|
|
|
|
newrelfilenumber = binary_upgrade_next_index_pg_class_relfilenumber;
|
|
|
|
binary_upgrade_next_index_pg_class_relfilenumber = InvalidOid;
|
|
|
|
}
|
|
|
|
else if (relation->rd_rel->relkind == RELKIND_RELATION)
|
|
|
|
{
|
|
|
|
if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenumber))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("heap relfilenumber value not set when in binary upgrade mode")));
|
|
|
|
|
|
|
|
newrelfilenumber = binary_upgrade_next_heap_pg_class_relfilenumber;
|
|
|
|
binary_upgrade_next_heap_pg_class_relfilenumber = InvalidOid;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("unexpected request for new relfilenumber in binary upgrade mode")));
|
2010-02-03 02:14:17 +01:00
|
|
|
|
|
|
|
/*
|
2010-02-07 21:48:13 +01:00
|
|
|
* Get a writable copy of the pg_class tuple for the given relation.
|
2010-02-03 02:14:17 +01:00
|
|
|
*/
|
2019-01-21 19:32:19 +01:00
|
|
|
pg_class = table_open(RelationRelationId, RowExclusiveLock);
|
2010-02-03 02:14:17 +01:00
|
|
|
|
2010-02-14 19:42:19 +01:00
|
|
|
tuple = SearchSysCacheCopy1(RELOID,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
2010-02-03 02:14:17 +01:00
|
|
|
if (!HeapTupleIsValid(tuple))
|
|
|
|
elog(ERROR, "could not find tuple for relation %u",
|
|
|
|
RelationGetRelid(relation));
|
|
|
|
classform = (Form_pg_class) GETSTRUCT(tuple);
|
|
|
|
|
|
|
|
/*
|
2022-07-28 22:03:42 +02:00
|
|
|
* Schedule unlinking of the old storage at transaction commit, except
|
|
|
|
* when performing a binary upgrade, when we must do it immediately.
|
2010-02-03 02:14:17 +01:00
|
|
|
*/
|
2022-07-28 22:03:42 +02:00
|
|
|
if (IsBinaryUpgrade)
|
|
|
|
{
|
|
|
|
SMgrRelation srel;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* During a binary upgrade, we use this code path to ensure that
|
|
|
|
* pg_largeobject and its index have the same relfilenumbers as in the
|
|
|
|
* old cluster. This is necessary because pg_upgrade treats
|
|
|
|
* pg_largeobject like a user table, not a system table. It is however
|
|
|
|
* possible that a table or index may need to end up with the same
|
|
|
|
* relfilenumber in the new cluster as what it had in the old cluster.
|
|
|
|
* Hence, we can't wait until commit time to remove the old storage.
|
|
|
|
*
|
|
|
|
* In general, this function needs to have transactional semantics,
|
|
|
|
* and removing the old storage before commit time surely isn't.
|
|
|
|
* However, it doesn't really matter, because if a binary upgrade
|
|
|
|
* fails at this stage, the new cluster will need to be recreated
|
|
|
|
* anyway.
|
|
|
|
*/
|
|
|
|
srel = smgropen(relation->rd_locator, relation->rd_backend);
|
|
|
|
smgrdounlinkall(&srel, 1, false);
|
|
|
|
smgrclose(srel);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Not a binary upgrade, so just schedule it to happen later. */
|
|
|
|
RelationDropStorage(relation);
|
|
|
|
}
|
2010-02-03 02:14:17 +01:00
|
|
|
|
2019-03-29 04:01:14 +01:00
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* Create storage for the main fork of the new relfilenumber. If it's a
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
* table-like object, call into the table AM to do so, which'll also
|
|
|
|
* create the table's init fork if needed.
|
2019-03-29 04:01:14 +01:00
|
|
|
*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* NOTE: If relevant for the AM, any conflict in relfilenumber value will
|
|
|
|
* be caught here, if GetNewRelFileNumber messes up for any reason.
|
2019-03-29 04:01:14 +01:00
|
|
|
*/
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
newrlocator = relation->rd_locator;
|
|
|
|
newrlocator.relNumber = newrelfilenumber;
|
2019-03-29 04:01:14 +01:00
|
|
|
|
2021-12-03 13:38:26 +01:00
|
|
|
if (RELKIND_HAS_TABLE_AM(relation->rd_rel->relkind))
|
2019-03-29 04:01:14 +01:00
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
table_relation_set_new_filelocator(relation, &newrlocator,
|
|
|
|
persistence,
|
|
|
|
&freezeXid, &minmulti);
|
2021-12-03 13:38:26 +01:00
|
|
|
}
|
|
|
|
else if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
|
|
|
|
{
|
|
|
|
/* handle these directly, at least for now */
|
|
|
|
SMgrRelation srel;
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
srel = RelationCreateStorage(newrlocator, persistence, true);
|
2021-12-03 13:38:26 +01:00
|
|
|
smgrclose(srel);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* we shouldn't be called for anything else */
|
|
|
|
elog(ERROR, "relation \"%s\" does not have storage",
|
|
|
|
RelationGetRelationName(relation));
|
2019-03-29 04:01:14 +01:00
|
|
|
}
|
2010-02-07 21:48:13 +01:00
|
|
|
|
2019-04-30 04:28:05 +02:00
|
|
|
/*
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
* If we're dealing with a mapped index, pg_class.relfilenode doesn't
|
|
|
|
* change; instead we have to send the update to the relation mapper.
|
|
|
|
*
|
|
|
|
* For mapped indexes, we don't actually change the pg_class entry at all;
|
|
|
|
* this is essential when reindexing pg_class itself. That leaves us with
|
|
|
|
* possibly-inaccurate values of relpages etc, but those will be fixed up
|
|
|
|
* later.
|
2019-04-30 04:28:05 +02:00
|
|
|
*/
|
|
|
|
if (RelationIsMapped(relation))
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
{
|
|
|
|
/* This case is only supported for indexes */
|
|
|
|
Assert(relation->rd_rel->relkind == RELKIND_INDEX);
|
|
|
|
|
|
|
|
/* Since we're not updating pg_class, these had better not change */
|
|
|
|
Assert(classform->relfrozenxid == freezeXid);
|
|
|
|
Assert(classform->relminmxid == minmulti);
|
|
|
|
Assert(classform->relpersistence == persistence);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In some code paths it's possible that the tuple update we'd
|
|
|
|
* otherwise do here is the only thing that would assign an XID for
|
|
|
|
* the current transaction. However, we must have an XID to delete
|
|
|
|
* files, so make sure one is assigned.
|
|
|
|
*/
|
|
|
|
(void) GetCurrentTransactionId();
|
|
|
|
|
|
|
|
/* Do the deed */
|
2019-04-30 04:28:05 +02:00
|
|
|
RelationMapUpdateMap(RelationGetRelid(relation),
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
newrelfilenumber,
|
2019-04-30 04:28:05 +02:00
|
|
|
relation->rd_rel->relisshared,
|
|
|
|
false);
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
|
|
|
|
/* Since we're not updating pg_class, must trigger inval manually */
|
|
|
|
CacheInvalidateRelcache(relation);
|
|
|
|
}
|
2019-04-30 04:28:05 +02:00
|
|
|
else
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
{
|
|
|
|
/* Normal case, update the pg_class entry */
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
classform->relfilenode = newrelfilenumber;
|
2019-04-30 04:28:05 +02:00
|
|
|
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
/* relpages etc. never change for sequences */
|
|
|
|
if (relation->rd_rel->relkind != RELKIND_SEQUENCE)
|
|
|
|
{
|
|
|
|
classform->relpages = 0; /* it's empty until further notice */
|
Redefine pg_class.reltuples to be -1 before the first VACUUM or ANALYZE.
Historically, we've considered the state with relpages and reltuples
both zero as indicating that we do not know the table's tuple density.
This is problematic because it's impossible to distinguish "never yet
vacuumed" from "vacuumed and seen to be empty". In particular, a user
cannot use VACUUM or ANALYZE to override the planner's normal heuristic
that an empty table should not be believed to be empty because it is
probably about to get populated. That heuristic is a good safety
measure, so I don't care to abandon it, but there should be a way to
override it if the table is indeed intended to stay empty.
Hence, represent the initial state of ignorance by setting reltuples
to -1 (relpages is still set to zero), and apply the minimum-ten-pages
heuristic only when reltuples is still -1. If the table is empty,
VACUUM or ANALYZE (but not CREATE INDEX) will override that to
reltuples = relpages = 0, and then we'll plan on that basis.
This requires a bunch of fiddly little changes, but we can get rid of
some ugly kluges that were formerly needed to maintain the old definition.
One notable point is that FDWs' GetForeignRelSize methods will see
baserel->tuples = -1 when no ANALYZE has been done on the foreign table.
That seems like a net improvement, since those methods were formerly
also in the dark about what baserel->tuples = 0 really meant. Still,
it is an API change.
I bumped catversion because code predating this change would get confused
by seeing reltuples = -1.
Discussion: https://postgr.es/m/F02298E0-6EF4-49A1-BCB6-C484794D9ACC@thebuild.com
2020-08-30 18:21:51 +02:00
|
|
|
classform->reltuples = -1;
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
classform->relallvisible = 0;
|
|
|
|
}
|
|
|
|
classform->relfrozenxid = freezeXid;
|
|
|
|
classform->relminmxid = minmulti;
|
|
|
|
classform->relpersistence = persistence;
|
2010-02-07 21:48:13 +01:00
|
|
|
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
CatalogTupleUpdate(pg_class, &tuple->t_self, tuple);
|
|
|
|
}
|
2010-02-03 02:14:17 +01:00
|
|
|
|
|
|
|
heap_freetuple(tuple);
|
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(pg_class, RowExclusiveLock);
|
2010-02-03 02:14:17 +01:00
|
|
|
|
|
|
|
/*
|
Fix reindexing of pg_class indexes some more.
Commits 3dbb317d3 et al failed under CLOBBER_CACHE_ALWAYS testing.
Investigation showed that to reindex pg_class_oid_index, we must
suppress accesses to the index (via SetReindexProcessing) before we call
RelationSetNewRelfilenode, or at least before we do CommandCounterIncrement
therein; otherwise, relcache reloads happening within the CCI may try to
fetch pg_class rows using the index's new relfilenode value, which is as
yet an empty file.
Of course, the point of 3dbb317d3 was that that ordering didn't work
either, because then RelationSetNewRelfilenode's own update of the index's
pg_class row cannot access the index, should it need to.
There are various ways we might have got around that, but Andres Freund
came up with a brilliant solution: for a mapped index, we can really just
skip the pg_class update altogether. The only fields it was actually
changing were relpages etc, but it was just setting them to zeroes which
is useless make-work. (Correct new values will be installed at the end
of index build.) All pg_class indexes are mapped and probably always will
be, so this eliminates the problem by removing work rather than adding it,
always a pleasant outcome. Having taught RelationSetNewRelfilenode to do
it that way, we can revert the code reordering in reindex_index. (But
I left the moved setup code where it was; there seems no reason why it
has to run without use of the old index. If you're trying to fix a
busted pg_class index, you'll have had to disable system index use
altogether to get this far.)
Moreover, this means we don't need RelationSetIndexList at all, because
reindex_relation's hacking to make "REINDEX TABLE pg_class" work is
likewise now unnecessary. We'll leave that code in place in the back
branches, but a follow-on patch will remove it in HEAD.
In passing, do some minor cleanup for commit 5c1560606 (in HEAD only),
notably removing a duplicate newrnode assignment.
Patch by me, using a core idea due to Andres Freund. Back-patch to all
supported branches, as 3dbb317d3 was.
Discussion: https://postgr.es/m/28926.1556664156@sss.pgh.pa.us
2019-05-03 01:11:28 +02:00
|
|
|
* Make the pg_class row change or relation map change visible. This will
|
|
|
|
* cause the relcache entry to get updated, too.
|
2010-02-03 02:14:17 +01:00
|
|
|
*/
|
|
|
|
CommandCounterIncrement();
|
|
|
|
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelationAssumeNewRelfilelocator(relation);
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* RelationAssumeNewRelfilelocator
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
*
|
|
|
|
* Code that modifies pg_class.reltablespace or pg_class.relfilenode must call
|
|
|
|
* this. The call shall precede any code that might insert WAL records whose
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* replay would modify bytes in the new RelFileLocator, and the call shall follow
|
|
|
|
* any WAL modifying bytes in the prior RelFileLocator. See struct RelationData.
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
* Ideally, call this as near as possible to the CommandCounterIncrement()
|
|
|
|
* that makes the pg_class change visible (before it or after it); that
|
|
|
|
* minimizes the chance of future development adding a forbidden WAL insertion
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
* between RelationAssumeNewRelfilelocator() and CommandCounterIncrement().
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
*/
|
|
|
|
void
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
RelationAssumeNewRelfilelocator(Relation relation)
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
{
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
relation->rd_newRelfilelocatorSubid = GetCurrentSubTransactionId();
|
|
|
|
if (relation->rd_firstRelfilelocatorSubid == InvalidSubTransactionId)
|
|
|
|
relation->rd_firstRelfilelocatorSubid = relation->rd_newRelfilelocatorSubid;
|
2013-01-20 19:44:49 +01:00
|
|
|
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
/* Flag relation as needing eoxact cleanup (to clear these fields) */
|
2013-01-20 19:44:49 +01:00
|
|
|
EOXactListAdd(relation);
|
2010-02-03 02:14:17 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2000-08-06 06:40:08 +02:00
|
|
|
* RelationCacheInitialize
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
2002-02-19 21:11:20 +01:00
|
|
|
* This initializes the relation descriptor cache. At the time
|
|
|
|
* that this is invoked, we can't do database access yet (mainly
|
2006-05-04 20:51:36 +02:00
|
|
|
* because the transaction subsystem is not up); all we are doing
|
|
|
|
* is making an empty cache hashtable. This must be done before
|
|
|
|
* starting the initialization transaction, because otherwise
|
|
|
|
* AtEOXact_RelationCache would crash if that transaction aborts
|
|
|
|
* before we can get the relcache set up.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
|
|
|
|
#define INITRELCACHESIZE 400
|
|
|
|
|
|
|
|
void
|
2000-08-06 06:40:08 +02:00
|
|
|
RelationCacheInitialize(void)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
|
|
|
HASHCTL ctl;
|
2021-10-24 03:36:38 +02:00
|
|
|
int allocsize;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2009-12-27 19:55:52 +01:00
|
|
|
* make sure cache memory context exists
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2000-06-28 05:33:33 +02:00
|
|
|
if (!CacheMemoryContext)
|
|
|
|
CreateCacheMemoryContext();
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2006-05-06 17:51:07 +02:00
|
|
|
* create hashtable that indexes the relcache
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
ctl.keysize = sizeof(Oid);
|
2001-10-01 07:36:17 +02:00
|
|
|
ctl.entrysize = sizeof(RelIdCacheEnt);
|
2001-10-05 19:28:13 +02:00
|
|
|
RelationIdCache = hash_create("Relcache by OID", INITRELCACHESIZE,
|
Improve hash_create's API for selecting simple-binary-key hash functions.
Previously, if you wanted anything besides C-string hash keys, you had to
specify a custom hashing function to hash_create(). Nearly all such
callers were specifying tag_hash or oid_hash; which is tedious, and rather
error-prone, since a caller could easily miss the opportunity to optimize
by using hash_uint32 when appropriate. Replace this with a design whereby
callers using simple binary-data keys just specify HASH_BLOBS and don't
need to mess with specific support functions. hash_create() itself will
take care of optimizing when the key size is four bytes.
This nets out saving a few hundred bytes of code space, and offers
a measurable performance improvement in tidbitmap.c (which was not
exploiting the opportunity to use hash_uint32 for its 4-byte keys).
There might be some wins elsewhere too, I didn't analyze closely.
In future we could look into offering a similar optimized hashing function
for 8-byte keys. Under this design that could be done in a centralized
and machine-independent fashion, whereas getting it right for keys of
platform-dependent sizes would've been notationally painful before.
For the moment, the old way still works fine, so as not to break source
code compatibility for loadable modules. Eventually we might want to
remove tag_hash and friends from the exported API altogether, since there's
no real need for them to be explicitly referenced from outside dynahash.c.
Teodor Sigaev and Tom Lane
2014-12-18 19:36:29 +01:00
|
|
|
&ctl, HASH_ELEM | HASH_BLOBS);
|
2010-02-07 21:48:13 +01:00
|
|
|
|
2021-10-24 03:36:38 +02:00
|
|
|
/*
|
|
|
|
* reserve enough in_progress_list slots for many cases
|
|
|
|
*/
|
|
|
|
allocsize = 4;
|
|
|
|
in_progress_list =
|
|
|
|
MemoryContextAlloc(CacheMemoryContext,
|
|
|
|
allocsize * sizeof(*in_progress_list));
|
|
|
|
in_progress_list_maxlen = allocsize;
|
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
/*
|
2010-11-15 05:10:45 +01:00
|
|
|
* relation mapper needs to be initialized too
|
2010-02-07 21:48:13 +01:00
|
|
|
*/
|
|
|
|
RelationMapInitialize();
|
2006-05-04 20:51:36 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RelationCacheInitializePhase2
|
|
|
|
*
|
2010-04-21 01:48:47 +02:00
|
|
|
* This is called to prepare for access to shared catalogs during startup.
|
|
|
|
* We must at least set up nailed reldescs for pg_database, pg_authid,
|
2016-01-05 18:50:53 +01:00
|
|
|
* pg_auth_members, and pg_shseclabel. Ideally we'd like to have reldescs
|
|
|
|
* for their indexes, too. We attempt to load this information from the
|
|
|
|
* shared relcache init file. If that's missing or broken, just make
|
|
|
|
* phony entries for the catalogs themselves.
|
|
|
|
* RelationCacheInitializePhase3 will clean up as needed.
|
2006-05-04 20:51:36 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationCacheInitializePhase2(void)
|
2009-08-12 22:53:31 +02:00
|
|
|
{
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
/*
|
|
|
|
* relation mapper needs initialized too
|
|
|
|
*/
|
|
|
|
RelationMapInitializePhase2();
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
2010-04-21 01:48:47 +02:00
|
|
|
* In bootstrap mode, the shared catalogs aren't there yet anyway, so do
|
|
|
|
* nothing.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
|
|
|
if (IsBootstrapProcessingMode())
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* switch to cache memory context
|
|
|
|
*/
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to load the shared relcache cache file. If unsuccessful, bootstrap
|
2010-04-21 01:48:47 +02:00
|
|
|
* the cache with pre-made descriptors for the critical shared catalogs.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
|
|
|
if (!load_relcache_init_file(true))
|
|
|
|
{
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc("pg_database", DatabaseRelation_Rowtype_Id, true,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_database, Desc_pg_database);
|
2010-04-21 01:48:47 +02:00
|
|
|
formrdesc("pg_authid", AuthIdRelation_Rowtype_Id, true,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_authid, Desc_pg_authid);
|
2010-04-21 01:48:47 +02:00
|
|
|
formrdesc("pg_auth_members", AuthMemRelation_Rowtype_Id, true,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_auth_members, Desc_pg_auth_members);
|
2016-01-05 18:50:53 +01:00
|
|
|
formrdesc("pg_shseclabel", SharedSecLabelRelation_Rowtype_Id, true,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_shseclabel, Desc_pg_shseclabel);
|
2017-01-19 18:00:00 +01:00
|
|
|
formrdesc("pg_subscription", SubscriptionRelation_Rowtype_Id, true,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_subscription, Desc_pg_subscription);
|
2009-08-12 22:53:31 +02:00
|
|
|
|
2017-01-19 18:00:00 +01:00
|
|
|
#define NUM_CRITICAL_SHARED_RELS 5 /* fix if you change list above */
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RelationCacheInitializePhase3
|
|
|
|
*
|
|
|
|
* This is called as soon as the catcache and transaction system
|
|
|
|
* are functional and we have determined MyDatabaseId. At this point
|
|
|
|
* we can actually read data from the database's system catalogs.
|
|
|
|
* We first try to read pre-computed relcache entries from the local
|
|
|
|
* relcache init file. If that's missing or broken, make phony entries
|
|
|
|
* for the minimum set of nailed-in-cache relations. Then (unless
|
|
|
|
* bootstrapping) make sure we have entries for the critical system
|
|
|
|
* indexes. Once we've done all this, we have enough infrastructure to
|
|
|
|
* open any system catalog or use any catcache. The last step is to
|
|
|
|
* rewrite the cache files if needed.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationCacheInitializePhase3(void)
|
2006-05-04 20:51:36 +02:00
|
|
|
{
|
|
|
|
HASH_SEQ_STATUS status;
|
|
|
|
RelIdCacheEnt *idhentry;
|
|
|
|
MemoryContext oldcxt;
|
2009-08-12 22:53:31 +02:00
|
|
|
bool needNewCacheFile = !criticalSharedRelcachesBuilt;
|
2006-05-04 20:51:36 +02:00
|
|
|
|
2010-02-07 21:48:13 +01:00
|
|
|
/*
|
|
|
|
* relation mapper needs initialized too
|
|
|
|
*/
|
|
|
|
RelationMapInitializePhase3();
|
|
|
|
|
2001-02-22 19:39:20 +01:00
|
|
|
/*
|
2006-05-04 20:51:36 +02:00
|
|
|
* switch to cache memory context
|
|
|
|
*/
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
|
|
|
|
/*
|
2009-08-12 22:53:31 +02:00
|
|
|
* Try to load the local relcache cache file. If unsuccessful, bootstrap
|
|
|
|
* the cache with pre-made descriptors for the critical "nailed-in" system
|
|
|
|
* catalogs.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
if (IsBootstrapProcessingMode() ||
|
2009-08-12 22:53:31 +02:00
|
|
|
!load_relcache_init_file(false))
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
2006-05-06 17:51:07 +02:00
|
|
|
needNewCacheFile = true;
|
|
|
|
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc("pg_class", RelationRelation_Rowtype_Id, false,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_class, Desc_pg_class);
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc("pg_attribute", AttributeRelation_Rowtype_Id, false,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_attribute, Desc_pg_attribute);
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc("pg_proc", ProcedureRelation_Rowtype_Id, false,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_proc, Desc_pg_proc);
|
2009-09-27 01:08:22 +02:00
|
|
|
formrdesc("pg_type", TypeRelation_Rowtype_Id, false,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Natts_pg_type, Desc_pg_type);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
#define NUM_CRITICAL_LOCAL_RELS 4 /* fix if you change list above */
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2006-05-04 20:51:36 +02:00
|
|
|
/* In bootstrap mode, the faked-up formrdesc info is all we'll have */
|
2002-02-19 21:11:20 +01:00
|
|
|
if (IsBootstrapProcessingMode())
|
|
|
|
return;
|
|
|
|
|
2000-08-06 06:40:08 +02:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* If we didn't get the critical system indexes loaded into relcache, do
|
2006-12-31 21:32:04 +01:00
|
|
|
* so now. These are critical because the catcache and/or opclass cache
|
|
|
|
* depend on them for fetches done during relcache load. Thus, we have an
|
2002-02-19 21:11:20 +01:00
|
|
|
* infinite-recursion problem. We can break the recursion by doing
|
|
|
|
* heapscans instead of indexscans at certain key spots. To avoid hobbling
|
|
|
|
* performance, we only want to do that until we have the critical indexes
|
|
|
|
* loaded into relcache. Thus, the flag criticalRelcachesBuilt is used to
|
|
|
|
* decide whether to do heapscan or indexscan at the key spots, and we set
|
|
|
|
* it true after we've loaded the critical indexes.
|
|
|
|
*
|
|
|
|
* The critical indexes are marked as "nailed in cache", partly to make it
|
|
|
|
* easy for load_relcache_init_file to count them, but mainly because we
|
|
|
|
* cannot flush and rebuild them once we've set criticalRelcachesBuilt to
|
|
|
|
* true. (NOTE: perhaps it would be possible to reload them by
|
|
|
|
* temporarily setting criticalRelcachesBuilt to false again. For now,
|
|
|
|
* though, we just nail 'em in.)
|
2006-01-08 21:04:41 +01:00
|
|
|
*
|
|
|
|
* RewriteRelRulenameIndexId and TriggerRelidNameIndexId are not critical
|
|
|
|
* in the same way as the others, because the critical catalogs don't
|
|
|
|
* (currently) have any rules or triggers, and so these indexes can be
|
|
|
|
* rebuilt without inducing recursion. However they are used during
|
|
|
|
* relcache load when a rel does have rules or triggers, so we choose to
|
|
|
|
* nail them for performance reasons.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
if (!criticalRelcachesBuilt)
|
|
|
|
{
|
2010-01-14 00:07:08 +01:00
|
|
|
load_critical_index(ClassOidIndexId,
|
|
|
|
RelationRelationId);
|
|
|
|
load_critical_index(AttributeRelidNumIndexId,
|
|
|
|
AttributeRelationId);
|
|
|
|
load_critical_index(IndexRelidIndexId,
|
|
|
|
IndexRelationId);
|
|
|
|
load_critical_index(OpclassOidIndexId,
|
|
|
|
OperatorClassRelationId);
|
|
|
|
load_critical_index(AccessMethodProcedureIndexId,
|
|
|
|
AccessMethodProcedureRelationId);
|
|
|
|
load_critical_index(RewriteRelRulenameIndexId,
|
|
|
|
RewriteRelationId);
|
|
|
|
load_critical_index(TriggerRelidNameIndexId,
|
|
|
|
TriggerRelationId);
|
2009-08-12 22:53:31 +02:00
|
|
|
|
2010-11-29 18:29:42 +01:00
|
|
|
#define NUM_CRITICAL_LOCAL_INDEXES 7 /* fix if you change list above */
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
criticalRelcachesBuilt = true;
|
|
|
|
}
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* Process critical shared indexes too.
|
|
|
|
*
|
|
|
|
* DatabaseNameIndexId isn't critical for relcache loading, but rather for
|
|
|
|
* initial lookup of MyDatabaseId, without which we'll never find any
|
|
|
|
* non-shared catalogs at all. Autovacuum calls InitPostgres with a
|
2010-04-21 01:48:47 +02:00
|
|
|
* database OID, so it instead depends on DatabaseOidIndexId. We also
|
|
|
|
* need to nail up some indexes on pg_authid and pg_auth_members for use
|
2016-01-05 18:50:53 +01:00
|
|
|
* during client authentication. SharedSecLabelObjectIndexId isn't
|
|
|
|
* critical for the core system, but authentication hooks might be
|
|
|
|
* interested in it.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
|
|
|
if (!criticalSharedRelcachesBuilt)
|
|
|
|
{
|
2010-01-14 00:07:08 +01:00
|
|
|
load_critical_index(DatabaseNameIndexId,
|
|
|
|
DatabaseRelationId);
|
|
|
|
load_critical_index(DatabaseOidIndexId,
|
|
|
|
DatabaseRelationId);
|
2010-04-21 01:48:47 +02:00
|
|
|
load_critical_index(AuthIdRolnameIndexId,
|
|
|
|
AuthIdRelationId);
|
|
|
|
load_critical_index(AuthIdOidIndexId,
|
|
|
|
AuthIdRelationId);
|
|
|
|
load_critical_index(AuthMemMemRoleIndexId,
|
|
|
|
AuthMemRelationId);
|
2016-01-05 18:50:53 +01:00
|
|
|
load_critical_index(SharedSecLabelObjectIndexId,
|
|
|
|
SharedSecLabelRelationId);
|
2010-04-21 01:48:47 +02:00
|
|
|
|
2016-01-05 18:50:53 +01:00
|
|
|
#define NUM_CRITICAL_SHARED_INDEXES 6 /* fix if you change list above */
|
2009-08-12 22:53:31 +02:00
|
|
|
|
|
|
|
criticalSharedRelcachesBuilt = true;
|
|
|
|
}
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Now, scan all the relcache entries and update anything that might be
|
|
|
|
* wrong in the results from formrdesc or the relcache cache file. If we
|
|
|
|
* faked up relcache entries using formrdesc, then read the real pg_class
|
|
|
|
* rows and replace the fake entries with them. Also, if any of the
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
* relcache entries have rules, triggers, or security policies, load that
|
|
|
|
* info the hard way since it isn't recorded in the cache file.
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
*
|
|
|
|
* Whenever we access the catalogs to read data, there is a possibility of
|
|
|
|
* a shared-inval cache flush causing relcache entries to be removed.
|
|
|
|
* Since hash_seq_search only guarantees to still work after the *current*
|
|
|
|
* entry is removed, it's unsafe to continue the hashtable scan afterward.
|
|
|
|
* We handle this by restarting the scan from scratch after each access.
|
|
|
|
* This is theoretically O(N^2), but the number of entries that actually
|
|
|
|
* need to be fixed is small enough that it doesn't matter.
|
2000-08-06 06:40:08 +02:00
|
|
|
*/
|
2002-03-26 20:17:02 +01:00
|
|
|
hash_seq_init(&status, RelationIdCache);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
2000-08-06 06:40:08 +02:00
|
|
|
{
|
2002-03-26 20:17:02 +01:00
|
|
|
Relation relation = idhentry->reldesc;
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
bool restart = false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure *this* entry doesn't get flushed while we work with it.
|
|
|
|
*/
|
|
|
|
RelationIncrementReferenceCount(relation);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2001-01-06 02:48:59 +01:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* If it's a faked-up entry, read the real pg_class tuple.
|
2001-01-06 02:48:59 +01:00
|
|
|
*/
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
if (relation->rd_rel->relowner == InvalidOid)
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
|
|
|
HeapTuple htup;
|
|
|
|
Form_pg_class relp;
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2010-02-14 19:42:19 +01:00
|
|
|
htup = SearchSysCache1(RELOID,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
2002-02-19 21:11:20 +01:00
|
|
|
if (!HeapTupleIsValid(htup))
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(FATAL, "cache lookup failed for relation %u",
|
|
|
|
RelationGetRelid(relation));
|
2002-02-19 21:11:20 +01:00
|
|
|
relp = (Form_pg_class) GETSTRUCT(htup);
|
2002-09-04 22:31:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Copy tuple to relation->rd_rel. (See notes in
|
|
|
|
* AllocateRelationDesc())
|
|
|
|
*/
|
|
|
|
memcpy((char *) relation->rd_rel, (char *) relp, CLASS_TUPLE_SIZE);
|
2004-04-01 23:28:47 +02:00
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
/* Update rd_options while we have the tuple */
|
|
|
|
if (relation->rd_options)
|
|
|
|
pfree(relation->rd_options);
|
|
|
|
RelationParseRelOptions(relation, htup);
|
|
|
|
|
2004-04-01 23:28:47 +02:00
|
|
|
/*
|
2009-09-27 01:08:22 +02:00
|
|
|
* Check the values in rd_att were set up correctly. (We cannot
|
|
|
|
* just copy them over now: formrdesc must have set up the rd_att
|
|
|
|
* data correctly to start with, because it may already have been
|
|
|
|
* copied into one or more catcache entries.)
|
2004-04-01 23:28:47 +02:00
|
|
|
*/
|
2009-09-27 01:08:22 +02:00
|
|
|
Assert(relation->rd_att->tdtypeid == relp->reltype);
|
|
|
|
Assert(relation->rd_att->tdtypmod == -1);
|
2001-01-06 02:48:59 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
ReleaseSysCache(htup);
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
|
|
|
|
/* relowner had better be OK now, else we'll loop forever */
|
|
|
|
if (relation->rd_rel->relowner == InvalidOid)
|
|
|
|
elog(ERROR, "invalid relowner in pg_class entry for \"%s\"",
|
|
|
|
RelationGetRelationName(relation));
|
|
|
|
|
|
|
|
restart = true;
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Fix data that isn't saved in relcache cache file.
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
*
|
|
|
|
* relhasrules or relhastriggers could possibly be wrong or out of
|
|
|
|
* date. If we don't actually find any rules or triggers, clear the
|
|
|
|
* local copy of the flag so that we don't get into an infinite loop
|
|
|
|
* here. We don't make any attempt to fix the pg_class entry, though.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
if (relation->rd_rel->relhasrules && relation->rd_rules == NULL)
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
RelationBuildRuleLock(relation);
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
if (relation->rd_rules == NULL)
|
|
|
|
relation->rd_rel->relhasrules = false;
|
|
|
|
restart = true;
|
|
|
|
}
|
2008-11-09 22:24:33 +01:00
|
|
|
if (relation->rd_rel->relhastriggers && relation->trigdesc == NULL)
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
RelationBuildTriggers(relation);
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
if (relation->trigdesc == NULL)
|
|
|
|
relation->rd_rel->relhastriggers = false;
|
|
|
|
restart = true;
|
|
|
|
}
|
|
|
|
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
/*
|
|
|
|
* Re-load the row security policies if the relation has them, since
|
|
|
|
* they are not preserved in the cache. Note that we can never NOT
|
Code review for row security.
Buildfarm member tick identified an issue where the policies in the
relcache for a relation were were being replaced underneath a running
query, leading to segfaults while processing the policies to be added
to a query. Similar to how TupleDesc RuleLocks are handled, add in a
equalRSDesc() function to check if the policies have actually changed
and, if not, swap back the rsdesc field (using the original instead of
the temporairly built one; the whole structure is swapped and then
specific fields swapped back). This now passes a CLOBBER_CACHE_ALWAYS
for me and should resolve the buildfarm error.
In addition to addressing this, add a new chapter in Data Definition
under Privileges which explains row security and provides examples of
its usage, change \d to always list policies (even if row security is
disabled- but note that it is disabled, or enabled with no policies),
rework check_role_for_policy (it really didn't need the entire policy,
but it did need to be using has_privs_of_role()), and change the field
in pg_class to relrowsecurity from relhasrowsecurity, based on
Heikki's suggestion. Also from Heikki, only issue SET ROW_SECURITY in
pg_restore when talking to a 9.5+ server, list Bypass RLS in \du, and
document --enable-row-security options for pg_dump and pg_restore.
Lastly, fix a number of minor whitespace and typo issues from Heikki,
Dimitri, add a missing #include, per Peter E, fix a few minor
variable-assigned-but-not-used and resource leak issues from Coverity
and add tab completion for role attribute bypassrls as well.
2014-09-24 22:32:22 +02:00
|
|
|
* have a policy while relrowsecurity is true,
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
* RelationBuildRowSecurity will create a single default-deny policy
|
Rename pg_rowsecurity -> pg_policy and other fixes
As pointed out by Robert, we should really have named pg_rowsecurity
pg_policy, as the objects stored in that catalog are policies. This
patch fixes that and updates the column names to start with 'pol' to
match the new catalog name.
The security consideration for COPY with row level security, also
pointed out by Robert, has also been addressed by remembering and
re-checking the OID of the relation initially referenced during COPY
processing, to make sure it hasn't changed under us by the time we
finish planning out the query which has been built.
Robert and Alvaro also commented on missing OCLASS and OBJECT entries
for POLICY (formerly ROWSECURITY or POLICY, depending) in various
places. This patch fixes that too, which also happens to add the
ability to COMMENT on policies.
In passing, attempt to improve the consistency of messages, comments,
and documentation as well. This removes various incarnations of
'row-security', 'row-level security', 'Row-security', etc, in favor
of 'policy', 'row level security' or 'row_security' as appropriate.
Happy Thanksgiving!
2014-11-27 07:06:36 +01:00
|
|
|
* if there is no policy defined in pg_policy.
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
*/
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
if (relation->rd_rel->relrowsecurity && relation->rd_rsdesc == NULL)
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
{
|
|
|
|
RelationBuildRowSecurity(relation);
|
|
|
|
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
Assert(relation->rd_rsdesc != NULL);
|
Row-Level Security Policies (RLS)
Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table. Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.
New top-level commands are CREATE/ALTER/DROP POLICY and are
controlled by the table owner. Row Security is able to be enabled
and disabled by the owner on a per-table basis using
ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.
Per discussion, ROW SECURITY is disabled on tables by default and
must be enabled for policies on the table to be used. If no
policies exist on a table with ROW SECURITY enabled, a default-deny
policy is used and no records will be visible.
By default, row security is applied at all times except for the
table owner and the superuser. A new GUC, row_security, is added
which can be set to ON, OFF, or FORCE. When set to FORCE, row
security will be applied even for the table owner and superusers.
When set to OFF, row security will be disabled when allowed and an
error will be thrown if the user does not have rights to bypass row
security.
Per discussion, pg_dump sets row_security = OFF by default to ensure
that exports and backups will have all data in the table or will
error if there are insufficient privileges to bypass row security.
A new option has been added to pg_dump, --enable-row-security, to
ask pg_dump to export with row security enabled.
A new role capability, BYPASSRLS, which can only be set by the
superuser, is added to allow other users to be able to bypass row
security using row_security = OFF.
Many thanks to the various individuals who have helped with the
design, particularly Robert Haas for his feedback.
Authors include Craig Ringer, KaiGai Kohei, Adam Brightwell, Dean
Rasheed, with additional changes and rework by me.
Reviewers have included all of the above, Greg Smith,
Jeff McCormick, and Robert Haas.
2014-09-19 17:18:35 +02:00
|
|
|
restart = true;
|
|
|
|
}
|
|
|
|
|
Load relcache entries' partitioning data on-demand, not immediately.
Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested. (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)
This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote). In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.
Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero. In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.) Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.
In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does. Avoid doing get_rel_relkind in the
critical section, too.
Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too. Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.
Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com
2019-12-25 20:43:13 +01:00
|
|
|
/* Reload tableam data if needed */
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
if (relation->rd_tableam == NULL &&
|
2021-12-03 13:38:26 +01:00
|
|
|
(RELKIND_HAS_TABLE_AM(relation->rd_rel->relkind) || relation->rd_rel->relkind == RELKIND_SEQUENCE))
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
{
|
|
|
|
RelationInitTableAccessMethod(relation);
|
|
|
|
Assert(relation->rd_tableam != NULL);
|
|
|
|
|
|
|
|
restart = true;
|
|
|
|
}
|
|
|
|
|
Fix RelationCacheInitializePhase2 (Phase3, in HEAD) to cope with the
possibility of shared-inval messages causing a relcache flush while it tries
to fill in missing data in preloaded relcache entries. There are actually
two distinct failure modes here:
1. The flush could delete the next-to-be-processed cache entry, causing
the subsequent hash_seq_search calls to go off into the weeds. This is
the problem reported by Michael Brown, and I believe it also accounts
for bug #5074. The simplest fix is to restart the hashtable scan after
we've read any new data from the catalogs. It appears that pre-8.4
branches have not suffered from this failure, because by chance there were
no other catalogs sharing the same hash chains with the catalogs that
RelationCacheInitializePhase2 had work to do for. However that's obviously
pretty fragile, and it seems possible that derivative versions with
additional system catalogs might be vulnerable, so I'm back-patching this
part of the fix anyway.
2. The flush could delete the *current* cache entry, in which case the
pointer to the newly-loaded data would end up being stored into an
already-deleted Relation struct. As long as it was still deleted, the only
consequence would be some leaked space in CacheMemoryContext. But it seems
possible that the Relation struct could already have been recycled, in
which case this represents a hard-to-reproduce clobber of cached data
structures, with unforeseeable consequences. The fix here is to pin the
entry while we work on it.
In passing, also change RelationCacheInitializePhase2 to Assert that
formrdesc() set up the relation's cached TupleDesc (rd_att) with the
correct type OID and hasoids values. This is more appropriate than
silently updating the values, because the original tupdesc might already
have been copied into the catcache. However this part of the patch is
not in HEAD because it fails due to some questionable recent changes in
formrdesc :-(. That will be cleaned up in a subsequent patch.
2009-09-26 20:24:49 +02:00
|
|
|
/* Release hold on the relation */
|
|
|
|
RelationDecrementReferenceCount(relation);
|
|
|
|
|
|
|
|
/* Now, restart the hashtable scan if needed */
|
|
|
|
if (restart)
|
|
|
|
{
|
|
|
|
hash_seq_term(&status);
|
|
|
|
hash_seq_init(&status, RelationIdCache);
|
|
|
|
}
|
2000-08-06 06:40:08 +02:00
|
|
|
}
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2006-05-04 20:51:36 +02:00
|
|
|
/*
|
2009-08-12 22:53:31 +02:00
|
|
|
* Lastly, write out new relcache cache files if needed. We don't bother
|
|
|
|
* to distinguish cases where only one of the two needs an update.
|
2006-05-04 20:51:36 +02:00
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
if (needNewCacheFile)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Force all the catcaches to finish initializing and thereby open the
|
|
|
|
* catalogs and indexes they use. This will preload the relcache with
|
|
|
|
* entries for all the most important system catalogs and indexes, so
|
2009-08-12 22:53:31 +02:00
|
|
|
* that the init files will be most useful for future backends.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
InitCatalogCachePhase2();
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/* now write the files */
|
|
|
|
write_relcache_init_file(true);
|
|
|
|
write_relcache_init_file(false);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* Load one critical system index into the relcache
|
2010-01-14 00:07:08 +01:00
|
|
|
*
|
|
|
|
* indexoid is the OID of the target index, heapoid is the OID of the catalog
|
|
|
|
* it belongs to.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
|
|
|
static void
|
2010-01-14 00:07:08 +01:00
|
|
|
load_critical_index(Oid indexoid, Oid heapoid)
|
2009-08-12 22:53:31 +02:00
|
|
|
{
|
|
|
|
Relation ird;
|
|
|
|
|
2010-01-14 00:07:08 +01:00
|
|
|
/*
|
|
|
|
* We must lock the underlying catalog before locking the index to avoid
|
|
|
|
* deadlock, since RelationBuildDesc might well need to read the catalog,
|
|
|
|
* and if anyone else is exclusive-locking this catalog and index they'll
|
|
|
|
* be doing it in that order.
|
|
|
|
*/
|
|
|
|
LockRelationOid(heapoid, AccessShareLock);
|
2009-08-12 22:53:31 +02:00
|
|
|
LockRelationOid(indexoid, AccessShareLock);
|
2010-01-12 19:12:18 +01:00
|
|
|
ird = RelationBuildDesc(indexoid, true);
|
2009-08-12 22:53:31 +02:00
|
|
|
if (ird == NULL)
|
|
|
|
elog(PANIC, "could not open critical system index %u", indexoid);
|
|
|
|
ird->rd_isnailed = true;
|
|
|
|
ird->rd_refcnt = 1;
|
|
|
|
UnlockRelationOid(indexoid, AccessShareLock);
|
2010-01-14 00:07:08 +01:00
|
|
|
UnlockRelationOid(heapoid, AccessShareLock);
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
|
|
|
|
(void) RelationGetIndexAttOptions(ird, false);
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
|
|
|
|
2005-03-29 02:17:27 +02:00
|
|
|
/*
|
2006-07-04 00:45:41 +02:00
|
|
|
* GetPgClassDescriptor -- get a predefined tuple descriptor for pg_class
|
2005-03-29 02:17:27 +02:00
|
|
|
* GetPgIndexDescriptor -- get a predefined tuple descriptor for pg_index
|
|
|
|
*
|
|
|
|
* We need this kluge because we have to be able to access non-fixed-width
|
2006-07-04 00:45:41 +02:00
|
|
|
* fields of pg_class and pg_index before we have the standard catalog caches
|
|
|
|
* available. We use predefined data that's set up in just the same way as
|
|
|
|
* the bootstrapped reldescs used by formrdesc(). The resulting tupdesc is
|
|
|
|
* not 100% kosher: it does not have the correct rowtype OID in tdtypeid, nor
|
|
|
|
* does it have a TupleConstr field. But it's good enough for the purpose of
|
|
|
|
* extracting fields.
|
2005-03-29 02:17:27 +02:00
|
|
|
*/
|
|
|
|
static TupleDesc
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
BuildHardcodedDescriptor(int natts, const FormData_pg_attribute *attrs)
|
2005-03-29 02:17:27 +02:00
|
|
|
{
|
2006-07-04 00:45:41 +02:00
|
|
|
TupleDesc result;
|
2005-03-29 02:17:27 +02:00
|
|
|
MemoryContext oldcxt;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
result = CreateTemplateTupleDesc(natts);
|
2006-07-04 00:45:41 +02:00
|
|
|
result->tdtypeid = RECORDOID; /* not right, but we don't care */
|
|
|
|
result->tdtypmod = -1;
|
2005-03-29 02:17:27 +02:00
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
for (i = 0; i < natts; i++)
|
2005-03-29 02:17:27 +02:00
|
|
|
{
|
2017-08-20 20:19:07 +02:00
|
|
|
memcpy(TupleDescAttr(result, i), &attrs[i], ATTRIBUTE_FIXED_PART_SIZE);
|
2005-03-29 02:17:27 +02:00
|
|
|
/* make sure attcacheoff is valid */
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(result, i)->attcacheoff = -1;
|
2005-03-29 02:17:27 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/* initialize first attribute's attcacheoff, cf RelationBuildTupleDesc */
|
2017-08-20 20:19:07 +02:00
|
|
|
TupleDescAttr(result, 0)->attcacheoff = 0;
|
2005-03-29 02:17:27 +02:00
|
|
|
|
|
|
|
/* Note: we don't bother to set up a TupleConstr entry */
|
|
|
|
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
static TupleDesc
|
|
|
|
GetPgClassDescriptor(void)
|
|
|
|
{
|
|
|
|
static TupleDesc pgclassdesc = NULL;
|
|
|
|
|
|
|
|
/* Already done? */
|
|
|
|
if (pgclassdesc == NULL)
|
|
|
|
pgclassdesc = BuildHardcodedDescriptor(Natts_pg_class,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Desc_pg_class);
|
2006-07-04 00:45:41 +02:00
|
|
|
|
|
|
|
return pgclassdesc;
|
|
|
|
}
|
|
|
|
|
|
|
|
static TupleDesc
|
|
|
|
GetPgIndexDescriptor(void)
|
|
|
|
{
|
|
|
|
static TupleDesc pgindexdesc = NULL;
|
|
|
|
|
|
|
|
/* Already done? */
|
|
|
|
if (pgindexdesc == NULL)
|
|
|
|
pgindexdesc = BuildHardcodedDescriptor(Natts_pg_index,
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
Desc_pg_index);
|
2006-07-04 00:45:41 +02:00
|
|
|
|
2005-03-29 02:17:27 +02:00
|
|
|
return pgindexdesc;
|
|
|
|
}
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* Load any default attribute value definitions for the relation.
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
*
|
|
|
|
* ndef is the number of attributes that were marked atthasdef.
|
|
|
|
*
|
|
|
|
* Note: we don't make it a hard error to be missing some pg_attrdef records.
|
|
|
|
* We can limp along as long as nothing needs to use the default value. Code
|
|
|
|
* that fails to find an expected AttrDefault record should throw an error.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
1997-08-21 03:36:09 +02:00
|
|
|
static void
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
AttrDefaultFetch(Relation relation, int ndef)
|
1997-08-21 03:36:09 +02:00
|
|
|
{
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
AttrDefault *attrdef;
|
1997-08-21 03:36:09 +02:00
|
|
|
Relation adrel;
|
2002-02-19 21:11:20 +01:00
|
|
|
SysScanDesc adscan;
|
1997-08-21 03:36:09 +02:00
|
|
|
ScanKeyData skey;
|
2000-02-18 10:30:20 +01:00
|
|
|
HeapTuple htup;
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
int found = 0;
|
|
|
|
|
|
|
|
/* Allocate array with room for as many entries as expected */
|
|
|
|
attrdef = (AttrDefault *)
|
|
|
|
MemoryContextAllocZero(CacheMemoryContext,
|
|
|
|
ndef * sizeof(AttrDefault));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/* Search pg_attrdef for relevant entries */
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey,
|
|
|
|
Anum_pg_attrdef_adrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
adrel = table_open(AttrDefaultRelationId, AccessShareLock);
|
2005-04-14 22:03:27 +02:00
|
|
|
adscan = systable_beginscan(adrel, AttrDefaultIndexId, true,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 1, &skey);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(adscan)))
|
1997-08-21 03:36:09 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
Form_pg_attrdef adform = (Form_pg_attrdef) GETSTRUCT(htup);
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
Datum val;
|
|
|
|
bool isnull;
|
1998-09-01 06:40:42 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/* protect limited size of array */
|
|
|
|
if (found >= ndef)
|
1997-08-21 03:36:09 +02:00
|
|
|
{
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
elog(WARNING, "unexpected pg_attrdef record found for attribute %d of relation \"%s\"",
|
|
|
|
adform->adnum, RelationGetRelationName(relation));
|
1997-08-22 05:35:44 +02:00
|
|
|
break;
|
1997-08-21 03:36:09 +02:00
|
|
|
}
|
1998-09-01 06:40:42 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
val = fastgetattr(htup,
|
|
|
|
Anum_pg_attrdef_adbin,
|
|
|
|
adrel->rd_att, &isnull);
|
|
|
|
if (isnull)
|
|
|
|
elog(WARNING, "null adbin for attribute %d of relation \"%s\"",
|
2003-07-25 22:18:01 +02:00
|
|
|
adform->adnum, RelationGetRelationName(relation));
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/* detoast and convert to cstring in caller's context */
|
|
|
|
char *s = TextDatumGetCString(val);
|
|
|
|
|
|
|
|
attrdef[found].adnum = adform->adnum;
|
|
|
|
attrdef[found].adbin = MemoryContextStrdup(CacheMemoryContext, s);
|
|
|
|
pfree(s);
|
|
|
|
found++;
|
|
|
|
}
|
1997-08-21 03:36:09 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
systable_endscan(adscan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(adrel, AccessShareLock);
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
|
|
|
|
if (found != ndef)
|
|
|
|
elog(WARNING, "%d pg_attrdef record(s) missing for relation \"%s\"",
|
|
|
|
ndef - found, RelationGetRelationName(relation));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Sort the AttrDefault entries by adnum, for the convenience of
|
|
|
|
* equalTupleDescs(). (Usually, they already will be in order, but this
|
|
|
|
* might not be so if systable_getnext isn't using an index.)
|
|
|
|
*/
|
|
|
|
if (found > 1)
|
|
|
|
qsort(attrdef, found, sizeof(AttrDefault), AttrDefaultCmp);
|
|
|
|
|
|
|
|
/* Install array only after it's fully valid */
|
|
|
|
relation->rd_att->constr->defval = attrdef;
|
|
|
|
relation->rd_att->constr->num_defval = found;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* qsort comparator to sort AttrDefault entries by adnum
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
AttrDefaultCmp(const void *a, const void *b)
|
|
|
|
{
|
|
|
|
const AttrDefault *ada = (const AttrDefault *) a;
|
|
|
|
const AttrDefault *adb = (const AttrDefault *) b;
|
|
|
|
|
|
|
|
return ada->adnum - adb->adnum;
|
1997-08-21 03:36:09 +02:00
|
|
|
}
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/*
|
|
|
|
* Load any check constraints for the relation.
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
*
|
|
|
|
* As with defaults, if we don't find the expected number of them, just warn
|
|
|
|
* here. The executor should throw an error if an INSERT/UPDATE is attempted.
|
2009-08-12 22:53:31 +02:00
|
|
|
*/
|
1997-08-22 05:35:44 +02:00
|
|
|
static void
|
2002-07-12 20:43:19 +02:00
|
|
|
CheckConstraintFetch(Relation relation)
|
1997-08-22 05:35:44 +02:00
|
|
|
{
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
ConstrCheck *check;
|
|
|
|
int ncheck = relation->rd_rel->relchecks;
|
2002-07-12 20:43:19 +02:00
|
|
|
Relation conrel;
|
|
|
|
SysScanDesc conscan;
|
|
|
|
ScanKeyData skey[1];
|
2000-02-18 10:30:20 +01:00
|
|
|
HeapTuple htup;
|
2002-07-12 20:43:19 +02:00
|
|
|
int found = 0;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/* Allocate array with room for as many entries as expected */
|
|
|
|
check = (ConstrCheck *)
|
|
|
|
MemoryContextAllocZero(CacheMemoryContext,
|
|
|
|
ncheck * sizeof(ConstrCheck));
|
|
|
|
|
|
|
|
/* Search pg_constraint for relevant entries */
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey[0],
|
|
|
|
Anum_pg_constraint_conrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
conrel = table_open(ConstraintRelationId, AccessShareLock);
|
Fully enforce uniqueness of constraint names.
It's been true for a long time that we expect names of table and domain
constraints to be unique among the constraints of that table or domain.
However, the enforcement of that has been pretty haphazard, and it missed
some corner cases such as creating a CHECK constraint and then an index
constraint of the same name (as per recent report from André Hänsel).
Also, due to the lack of an actual unique index enforcing this, duplicates
could be created through race conditions.
Moreover, the code that searches pg_constraint has been quite inconsistent
about how to handle duplicate names if one did occur: some places checked
and threw errors if there was more than one match, while others just
processed the first match they came to.
To fix, create a unique index on (conrelid, contypid, conname). Since
either conrelid or contypid is zero, this will separately enforce
uniqueness of constraint names among constraints of any one table and any
one domain. (If we ever implement SQL assertions, and put them into this
catalog, more thought might be needed. But it'd be at least as reasonable
to put them into a new catalog; having overloaded this one catalog with
two kinds of constraints was a mistake already IMO.) This index can replace
the existing non-unique index on conrelid, though we need to keep the one
on contypid for query performance reasons.
Having done that, we can simplify the logic in various places that either
coped with duplicates or neglected to, as well as potentially improve
lookup performance when searching for a constraint by name.
Also, as per our usual practice, install a preliminary check so that you
get something more friendly than a unique-index violation report in the
case complained of by André. And teach ChooseIndexName to avoid choosing
autogenerated names that would draw such a failure.
While it's not possible to make such a change in the back branches,
it doesn't seem quite too late to put this into v11, so do so.
Discussion: https://postgr.es/m/0c1001d4428f$0942b430$1bc81c90$@webkr.de
2018-09-04 19:45:35 +02:00
|
|
|
conscan = systable_beginscan(conrel, ConstraintRelidTypidNameIndexId, true,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 1, skey);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-07-12 20:43:19 +02:00
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(conscan)))
|
1997-08-22 05:35:44 +02:00
|
|
|
{
|
2002-07-12 20:43:19 +02:00
|
|
|
Form_pg_constraint conform = (Form_pg_constraint) GETSTRUCT(htup);
|
2014-08-24 17:56:52 +02:00
|
|
|
Datum val;
|
|
|
|
bool isnull;
|
2002-07-12 20:43:19 +02:00
|
|
|
|
|
|
|
/* We want check constraints only */
|
|
|
|
if (conform->contype != CONSTRAINT_CHECK)
|
|
|
|
continue;
|
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/* protect limited size of array */
|
2003-07-25 22:18:01 +02:00
|
|
|
if (found >= ncheck)
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
{
|
|
|
|
elog(WARNING, "unexpected pg_constraint record found for relation \"%s\"",
|
1999-11-08 00:08:36 +01:00
|
|
|
RelationGetRelationName(relation));
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
break;
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2011-06-02 00:43:50 +02:00
|
|
|
check[found].ccvalid = conform->convalidated;
|
2012-04-21 04:46:20 +02:00
|
|
|
check[found].ccnoinherit = conform->connoinherit;
|
2000-06-30 09:04:23 +02:00
|
|
|
check[found].ccname = MemoryContextStrdup(CacheMemoryContext,
|
2002-07-12 20:43:19 +02:00
|
|
|
NameStr(conform->conname));
|
|
|
|
|
|
|
|
/* Grab and test conbin is actually set */
|
2000-07-06 01:12:09 +02:00
|
|
|
val = fastgetattr(htup,
|
2002-07-12 20:43:19 +02:00
|
|
|
Anum_pg_constraint_conbin,
|
|
|
|
conrel->rd_att, &isnull);
|
1997-08-22 05:35:44 +02:00
|
|
|
if (isnull)
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
elog(WARNING, "null conbin for relation \"%s\"",
|
1999-11-08 00:08:36 +01:00
|
|
|
RelationGetRelationName(relation));
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/* detoast and convert to cstring in caller's context */
|
|
|
|
char *s = TextDatumGetCString(val);
|
2002-07-12 20:43:19 +02:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
check[found].ccbin = MemoryContextStrdup(CacheMemoryContext, s);
|
|
|
|
pfree(s);
|
|
|
|
found++;
|
|
|
|
}
|
1997-08-22 05:35:44 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-07-12 20:43:19 +02:00
|
|
|
systable_endscan(conscan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(conrel, AccessShareLock);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
if (found != ncheck)
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
elog(WARNING, "%d pg_constraint record(s) missing for relation \"%s\"",
|
2002-02-19 21:11:20 +01:00
|
|
|
ncheck - found, RelationGetRelationName(relation));
|
Apply table and domain CHECK constraints in name order.
Previously, CHECK constraints of the same scope were checked in whatever
order they happened to be read from pg_constraint. (Usually, but not
reliably, this would be creation order for domain constraints and reverse
creation order for table constraints, because of differing implementation
details.) Nondeterministic results of this sort are problematic at least
for testing purposes, and in discussion it was agreed to be a violation of
the principle of least astonishment. Therefore, borrow the principle
already established for triggers, and apply such checks in name order
(using strcmp() sort rules). This lets users control the check order
if they have a mind to.
Domain CHECK constraints still follow the rule of checking lower nested
domains' constraints first; the name sort only applies to multiple
constraints attached to the same domain.
In passing, I failed to resist the temptation to wordsmith a bit in
create_domain.sgml.
Apply to HEAD only, since this could result in a behavioral change in
existing applications, and the potential regression test failures have
not actually been observed in our buildfarm.
2015-03-23 21:59:29 +01:00
|
|
|
|
Clean up treatment of missing default and CHECK-constraint records.
Andrew Gierth reported that it's possible to crash the backend if no
pg_attrdef record is found to match an attribute that has atthasdef set.
AttrDefaultFetch warns about this situation, but then leaves behind
a relation tupdesc that has null "adbin" pointer(s), which most places
don't guard against.
We considered promoting the warning to an error, but throwing errors
during relcache load is pretty drastic: it effectively locks one out
of using the relation at all. What seems better is to leave the
load-time behavior as a warning, but then throw an error in any code
path that wants to use a default and can't find it. This confines
the error to a subset of INSERT/UPDATE operations on the table, and
in particular will at least allow a pg_dump to succeed.
Also, we should fix AttrDefaultFetch to not leave any null pointers
in the tupdesc, because that just creates an untested bug hazard.
While at it, apply the same philosophy of "warn at load, throw error
only upon use of the known-missing info" to CHECK constraints.
CheckConstraintFetch is very nearly the same logic as AttrDefaultFetch,
but for reasons lost in the mists of time, it was throwing ERROR for
the same cases that AttrDefaultFetch treats as WARNING. Make the two
functions more nearly alike.
In passing, get rid of potentially-O(N^2) loops in equalTupleDesc
by making AttrDefaultFetch sort the entries after fetching them,
so that equalTupleDesc can assume that entries in two equal tupdescs
must be in matching order. (CheckConstraintFetch already was sorting
CHECK constraints, but equalTupleDesc hadn't been told about it.)
There's some argument for back-patching this, but with such a small
number of field reports, I'm content to fix it in HEAD.
Discussion: https://postgr.es/m/87pmzaq4gx.fsf@news-spur.riddles.org.uk
2021-04-06 16:34:37 +02:00
|
|
|
/*
|
|
|
|
* Sort the records by name. This ensures that CHECKs are applied in a
|
|
|
|
* deterministic order, and it also makes equalTupleDescs() faster.
|
|
|
|
*/
|
|
|
|
if (found > 1)
|
|
|
|
qsort(check, found, sizeof(ConstrCheck), CheckConstraintCmp);
|
|
|
|
|
|
|
|
/* Install array only after it's fully valid */
|
|
|
|
relation->rd_att->constr->check = check;
|
|
|
|
relation->rd_att->constr->num_check = found;
|
Apply table and domain CHECK constraints in name order.
Previously, CHECK constraints of the same scope were checked in whatever
order they happened to be read from pg_constraint. (Usually, but not
reliably, this would be creation order for domain constraints and reverse
creation order for table constraints, because of differing implementation
details.) Nondeterministic results of this sort are problematic at least
for testing purposes, and in discussion it was agreed to be a violation of
the principle of least astonishment. Therefore, borrow the principle
already established for triggers, and apply such checks in name order
(using strcmp() sort rules). This lets users control the check order
if they have a mind to.
Domain CHECK constraints still follow the rule of checking lower nested
domains' constraints first; the name sort only applies to multiple
constraints attached to the same domain.
In passing, I failed to resist the temptation to wordsmith a bit in
create_domain.sgml.
Apply to HEAD only, since this could result in a behavioral change in
existing applications, and the potential regression test failures have
not actually been observed in our buildfarm.
2015-03-23 21:59:29 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* qsort comparator to sort ConstrCheck entries by name
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
CheckConstraintCmp(const void *a, const void *b)
|
|
|
|
{
|
|
|
|
const ConstrCheck *ca = (const ConstrCheck *) a;
|
|
|
|
const ConstrCheck *cb = (const ConstrCheck *) b;
|
|
|
|
|
|
|
|
return strcmp(ca->ccname, cb->ccname);
|
1997-08-22 05:35:44 +02:00
|
|
|
}
|
|
|
|
|
2016-06-18 21:22:34 +02:00
|
|
|
/*
|
|
|
|
* RelationGetFKeyList -- get a list of foreign key info for the relation
|
|
|
|
*
|
|
|
|
* Returns a list of ForeignKeyCacheInfo structs, one per FK constraining
|
|
|
|
* the given relation. This data is a direct copy of relevant fields from
|
|
|
|
* pg_constraint. The list items are in no particular order.
|
|
|
|
*
|
|
|
|
* CAUTION: the returned list is part of the relcache's data, and could
|
|
|
|
* vanish in a relcache entry reset. Callers must inspect or copy it
|
|
|
|
* before doing anything that might trigger a cache flush, such as
|
|
|
|
* system catalog accesses. copyObject() can be used if desired.
|
|
|
|
* (We define it this way because current callers want to filter and
|
|
|
|
* modify the list entries anyway, so copying would be a waste of time.)
|
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetFKeyList(Relation relation)
|
|
|
|
{
|
|
|
|
List *result;
|
|
|
|
Relation conrel;
|
|
|
|
SysScanDesc conscan;
|
|
|
|
ScanKeyData skey;
|
|
|
|
HeapTuple htup;
|
|
|
|
List *oldlist;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the list. */
|
|
|
|
if (relation->rd_fkeyvalid)
|
|
|
|
return relation->rd_fkeylist;
|
|
|
|
|
Correct attach/detach logic for FKs in partitions
There was no code to handle foreign key constraints on partitioned
tables in the case of ALTER TABLE DETACH; and if you happened to ATTACH
a partition that already had an equivalent constraint, that one was
ignored and a new constraint was created. Adding this to the fact that
foreign key cloning reuses the constraint name on the partition instead
of generating a new name (as it probably should, to cater to SQL
standard rules about constraint naming within schemas), the result was a
pretty poor user experience -- the most visible failure was that just
detaching a partition and re-attaching it failed with an error such as
ERROR: duplicate key value violates unique constraint "pg_constraint_conrelid_contypid_conname_index"
DETAIL: Key (conrelid, contypid, conname)=(26702, 0, test_result_asset_id_fkey) already exists.
because it would try to create an identically-named constraint in the
partition. To make matters worse, if you tried to drop the constraint
in the now-independent partition, that would fail because the constraint
was still seen as dependent on the constraint in its former parent
partitioned table:
ERROR: cannot drop inherited constraint "test_result_asset_id_fkey" of relation "test_result_cbsystem_0001_0050_monthly_2018_09"
This fix attacks the problem from two angles: first, when the partition
is detached, the constraint is also marked as independent, so the drop
now works. Second, when the partition is re-attached, we scan existing
constraints searching for one matching the FK in the parent, and if one
exists, we link that one to the parent constraint. So we don't end up
with a duplicate -- and better yet, we don't need to scan the referenced
table to verify that the constraint holds.
To implement this I made a small change to previously planner-only
struct ForeignKeyCacheInfo to contain the constraint OID; also relcache
now maintains the list of FKs for partitioned tables too.
Backpatch to 11.
Reported-by: Michael Vitale (bug #15425)
Discussion: https://postgr.es/m/15425-2dbc9d2aa999f816@postgresql.org
2018-10-12 17:36:26 +02:00
|
|
|
/* Fast path: non-partitioned tables without triggers can't have FKs */
|
|
|
|
if (!relation->rd_rel->relhastriggers &&
|
|
|
|
relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
|
2016-06-18 21:22:34 +02:00
|
|
|
return NIL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We build the list we intend to return (in the caller's context) while
|
|
|
|
* doing the scan. After successfully completing the scan, we copy that
|
|
|
|
* list into the relcache entry. This avoids cache-context memory leakage
|
|
|
|
* if we get some sort of error partway through.
|
|
|
|
*/
|
|
|
|
result = NIL;
|
|
|
|
|
|
|
|
/* Prepare to scan pg_constraint for entries having conrelid = this rel. */
|
|
|
|
ScanKeyInit(&skey,
|
|
|
|
Anum_pg_constraint_conrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
conrel = table_open(ConstraintRelationId, AccessShareLock);
|
Fully enforce uniqueness of constraint names.
It's been true for a long time that we expect names of table and domain
constraints to be unique among the constraints of that table or domain.
However, the enforcement of that has been pretty haphazard, and it missed
some corner cases such as creating a CHECK constraint and then an index
constraint of the same name (as per recent report from André Hänsel).
Also, due to the lack of an actual unique index enforcing this, duplicates
could be created through race conditions.
Moreover, the code that searches pg_constraint has been quite inconsistent
about how to handle duplicate names if one did occur: some places checked
and threw errors if there was more than one match, while others just
processed the first match they came to.
To fix, create a unique index on (conrelid, contypid, conname). Since
either conrelid or contypid is zero, this will separately enforce
uniqueness of constraint names among constraints of any one table and any
one domain. (If we ever implement SQL assertions, and put them into this
catalog, more thought might be needed. But it'd be at least as reasonable
to put them into a new catalog; having overloaded this one catalog with
two kinds of constraints was a mistake already IMO.) This index can replace
the existing non-unique index on conrelid, though we need to keep the one
on contypid for query performance reasons.
Having done that, we can simplify the logic in various places that either
coped with duplicates or neglected to, as well as potentially improve
lookup performance when searching for a constraint by name.
Also, as per our usual practice, install a preliminary check so that you
get something more friendly than a unique-index violation report in the
case complained of by André. And teach ChooseIndexName to avoid choosing
autogenerated names that would draw such a failure.
While it's not possible to make such a change in the back branches,
it doesn't seem quite too late to put this into v11, so do so.
Discussion: https://postgr.es/m/0c1001d4428f$0942b430$1bc81c90$@webkr.de
2018-09-04 19:45:35 +02:00
|
|
|
conscan = systable_beginscan(conrel, ConstraintRelidTypidNameIndexId, true,
|
2016-06-18 21:22:34 +02:00
|
|
|
NULL, 1, &skey);
|
|
|
|
|
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(conscan)))
|
|
|
|
{
|
|
|
|
Form_pg_constraint constraint = (Form_pg_constraint) GETSTRUCT(htup);
|
|
|
|
ForeignKeyCacheInfo *info;
|
|
|
|
|
|
|
|
/* consider only foreign keys */
|
|
|
|
if (constraint->contype != CONSTRAINT_FOREIGN)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
info = makeNode(ForeignKeyCacheInfo);
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
info->conoid = constraint->oid;
|
2016-06-18 21:22:34 +02:00
|
|
|
info->conrelid = constraint->conrelid;
|
|
|
|
info->confrelid = constraint->confrelid;
|
|
|
|
|
2019-01-18 18:40:13 +01:00
|
|
|
DeconstructFkConstraintRow(htup, &info->nkeys,
|
|
|
|
info->conkey,
|
|
|
|
info->confkey,
|
|
|
|
info->conpfeqop,
|
2021-12-08 11:09:44 +01:00
|
|
|
NULL, NULL, NULL, NULL);
|
2016-06-18 21:22:34 +02:00
|
|
|
|
|
|
|
/* Add FK's node to the result list */
|
|
|
|
result = lappend(result, info);
|
|
|
|
}
|
|
|
|
|
|
|
|
systable_endscan(conscan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(conrel, AccessShareLock);
|
2016-06-18 21:22:34 +02:00
|
|
|
|
|
|
|
/* Now save a copy of the completed list in the relcache entry. */
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
oldlist = relation->rd_fkeylist;
|
|
|
|
relation->rd_fkeylist = copyObject(result);
|
|
|
|
relation->rd_fkeyvalid = true;
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
/* Don't leak the old list, if there is one */
|
|
|
|
list_free_deep(oldlist);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2000-06-17 23:49:04 +02:00
|
|
|
/*
|
|
|
|
* RelationGetIndexList -- get a list of OIDs of indexes on this relation
|
|
|
|
*
|
|
|
|
* The index list is created only if someone requests it. We scan pg_index
|
|
|
|
* to find relevant indexes, and add the list to the relcache entry so that
|
|
|
|
* we won't have to compute it again. Note that shared cache inval of a
|
2019-05-03 16:26:14 +02:00
|
|
|
* relcache entry will delete the old list and set rd_indexvalid to false,
|
2000-06-17 23:49:04 +02:00
|
|
|
* so that we must recompute the index list on next request. This handles
|
|
|
|
* creation or deletion of an index.
|
|
|
|
*
|
2018-12-27 10:07:46 +01:00
|
|
|
* Indexes that are marked not indislive are omitted from the returned list.
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
* Such indexes are expected to be dropped momentarily, and should not be
|
|
|
|
* touched at all by any caller of this function.
|
|
|
|
*
|
2000-06-20 01:40:48 +02:00
|
|
|
* The returned list is guaranteed to be sorted in order by OID. This is
|
|
|
|
* needed by the executor, since for index types that we obtain exclusive
|
|
|
|
* locks on when updating the index, all backends must lock the indexes in
|
|
|
|
* the same order or we will get deadlocks (see ExecOpenIndices()). Any
|
|
|
|
* consistent ordering would do, but ordering by OID is easy.
|
|
|
|
*
|
2000-06-17 23:49:04 +02:00
|
|
|
* Since shared cache inval causes the relcache's copy of the list to go away,
|
|
|
|
* we return a copy of the list palloc'd in the caller's context. The caller
|
2005-09-16 06:13:18 +02:00
|
|
|
* may list_free() the returned list after scanning it. This is necessary
|
2000-06-17 23:49:04 +02:00
|
|
|
* since the caller will typically be doing syscache lookups on the relevant
|
|
|
|
* indexes, and syscache lookup could cause SI messages to be processed!
|
2005-08-12 03:36:05 +02:00
|
|
|
*
|
2017-02-06 17:31:23 +01:00
|
|
|
* In exactly the same way, we update rd_pkindex, which is the OID of the
|
|
|
|
* relation's primary key index if any, else InvalidOid; and rd_replidindex,
|
|
|
|
* which is the pg_class OID of an index to be used as the relation's
|
|
|
|
* replication identity index, or InvalidOid if there is no such index.
|
2000-06-17 23:49:04 +02:00
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetIndexList(Relation relation)
|
|
|
|
{
|
|
|
|
Relation indrel;
|
2002-02-19 21:11:20 +01:00
|
|
|
SysScanDesc indscan;
|
2000-06-17 23:49:04 +02:00
|
|
|
ScanKeyData skey;
|
2002-02-19 21:11:20 +01:00
|
|
|
HeapTuple htup;
|
2000-06-17 23:49:04 +02:00
|
|
|
List *result;
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
List *oldlist;
|
2013-11-08 18:30:43 +01:00
|
|
|
char replident = relation->rd_rel->relreplident;
|
|
|
|
Oid pkeyIndex = InvalidOid;
|
|
|
|
Oid candidateIndex = InvalidOid;
|
2000-06-17 23:49:04 +02:00
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the list. */
|
2019-05-03 16:26:14 +02:00
|
|
|
if (relation->rd_indexvalid)
|
2004-05-31 01:40:41 +02:00
|
|
|
return list_copy(relation->rd_indexlist);
|
2000-06-17 23:49:04 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We build the list we intend to return (in the caller's context) while
|
|
|
|
* doing the scan. After successfully completing the scan, we copy that
|
|
|
|
* list into the relcache entry. This avoids cache-context memory leakage
|
|
|
|
* if we get some sort of error partway through.
|
|
|
|
*/
|
|
|
|
result = NIL;
|
2001-03-22 05:01:46 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* Prepare to scan pg_index for entries having indrelid = this rel. */
|
2003-11-12 22:15:59 +01:00
|
|
|
ScanKeyInit(&skey,
|
|
|
|
Anum_pg_index_indrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
2000-06-17 23:49:04 +02:00
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
indrel = table_open(IndexRelationId, AccessShareLock);
|
2005-04-14 22:03:27 +02:00
|
|
|
indscan = systable_beginscan(indrel, IndexIndrelidIndexId, true,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 1, &skey);
|
2000-06-17 23:49:04 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(indscan)))
|
|
|
|
{
|
|
|
|
Form_pg_index index = (Form_pg_index) GETSTRUCT(htup);
|
2000-06-17 23:49:04 +02:00
|
|
|
|
2012-04-06 11:21:40 +02:00
|
|
|
/*
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
* Ignore any indexes that are currently being dropped. This will
|
|
|
|
* prevent them from being searched, inserted into, or considered in
|
|
|
|
* HOT-safety decisions. It's unsafe to touch such an index at all
|
|
|
|
* since its catalog entries could disappear at any instant.
|
2012-04-06 11:21:40 +02:00
|
|
|
*/
|
2018-12-27 10:07:46 +01:00
|
|
|
if (!index->indislive)
|
2012-04-06 11:21:40 +02:00
|
|
|
continue;
|
|
|
|
|
Clean up some ad-hoc code for sorting and de-duplicating Lists.
heap.c and relcache.c contained nearly identical copies of logic
to insert OIDs into an OID list while preserving the list's OID
ordering (and rejecting duplicates, in one case but not the other).
The comments argue that this is faster than qsort for small numbers
of OIDs, which is at best unproven, and seems even less likely to be
true now that lappend_cell_oid has to move data around. In any case
it's ugly and hard-to-follow code, and if we do have a lot of OIDs
to consider, it's O(N^2).
Hence, replace with simply lappend'ing OIDs to a List, then list_sort
the completed List, then remove adjacent duplicates if necessary.
This is demonstrably O(N log N) and it's much simpler for the
callers. It's possible that this would be somewhat inefficient
if there were a very large number of duplicates, but that seems
unlikely in the existing usage.
This adds list_deduplicate_oid and list_oid_cmp infrastructure
to list.c. I didn't bother with equivalent functionality for
integer or pointer Lists, but such could always be added later
if we find a use for it.
Discussion: https://postgr.es/m/26193.1563228600@sss.pgh.pa.us
2019-07-16 18:04:06 +02:00
|
|
|
/* add index's OID to result list */
|
|
|
|
result = lappend_oid(result, index->indexrelid);
|
2005-08-12 03:36:05 +02:00
|
|
|
|
2013-11-08 18:30:43 +01:00
|
|
|
/*
|
Catalog not-null constraints
We now create contype='n' pg_constraint rows for not-null constraints.
We propagate these constraints to other tables during operations such as
adding inheritance relationships, creating and attaching partitions and
creating tables LIKE other tables. We also spawn not-null constraints
for inheritance child tables when their parents have primary keys.
These related constraints mostly follow the well-known rules of
conislocal and coninhcount that we have for CHECK constraints, with some
adaptations: for example, as opposed to CHECK constraints, we don't
match not-null ones by name when descending a hierarchy to alter it,
instead matching by column name that they apply to. This means we don't
require the constraint names to be identical across a hierarchy.
For now, we omit them for system catalogs. Maybe this is worth
reconsidering. We don't support NOT VALID nor DEFERRABLE clauses
either; these can be added as separate features later (this patch is
already large and complicated enough.)
psql shows these constraints in \d+.
pg_dump requires some ad-hoc hacks, particularly when dumping a primary
key. We now create one "throwaway" not-null constraint for each column
in the PK together with the CREATE TABLE command, and once the PK is
created, all those throwaway constraints are removed. This avoids
having to check each tuple for nullness when the dump restores the
primary key creation.
pg_upgrading from an older release requires a somewhat brittle procedure
to create a constraint state that matches what would be created if the
database were being created fresh in Postgres 17. I have tested all the
scenarios I could think of, and it works correctly as far as I can tell,
but I could have neglected weird cases.
This patch has been very long in the making. The first patch was
written by Bernd Helmle in 2010 to add a new pg_constraint.contype value
('n'), which I (Álvaro) then hijacked in 2011 and 2012, until that one
was killed by the realization that we ought to use contype='c' instead:
manufactured CHECK constraints. However, later SQL standard
development, as well as nonobvious emergent properties of that design
(mostly, failure to distinguish them from "normal" CHECK constraints as
well as the performance implication of having to test the CHECK
expression) led us to reconsider this choice, so now the current
implementation uses contype='n' again. During Postgres 16 this had
already been introduced by commit e056c557aef4, but there were some
problems mainly with the pg_upgrade procedure that couldn't be fixed in
reasonable time, so it was reverted.
In 2016 Vitaly Burovoy also worked on this feature[1] but found no
consensus for his proposed approach, which was claimed to be closer to
the letter of the standard, requiring an additional pg_attribute column
to track the OID of the not-null constraint for that column.
[1] https://postgr.es/m/CAKOSWNkN6HSyatuys8xZxzRCR-KL1OkHS5-b9qd9bf1Rad3PLA@mail.gmail.com
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Author: Bernd Helmle <mailings@oopsware.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
2023-08-25 13:31:24 +02:00
|
|
|
* Non-unique, non-immediate or predicate indexes aren't interesting
|
|
|
|
* for either oid indexes or replication identity indexes, so don't
|
|
|
|
* check them.
|
2013-11-08 18:30:43 +01:00
|
|
|
*/
|
Catalog not-null constraints
We now create contype='n' pg_constraint rows for not-null constraints.
We propagate these constraints to other tables during operations such as
adding inheritance relationships, creating and attaching partitions and
creating tables LIKE other tables. We also spawn not-null constraints
for inheritance child tables when their parents have primary keys.
These related constraints mostly follow the well-known rules of
conislocal and coninhcount that we have for CHECK constraints, with some
adaptations: for example, as opposed to CHECK constraints, we don't
match not-null ones by name when descending a hierarchy to alter it,
instead matching by column name that they apply to. This means we don't
require the constraint names to be identical across a hierarchy.
For now, we omit them for system catalogs. Maybe this is worth
reconsidering. We don't support NOT VALID nor DEFERRABLE clauses
either; these can be added as separate features later (this patch is
already large and complicated enough.)
psql shows these constraints in \d+.
pg_dump requires some ad-hoc hacks, particularly when dumping a primary
key. We now create one "throwaway" not-null constraint for each column
in the PK together with the CREATE TABLE command, and once the PK is
created, all those throwaway constraints are removed. This avoids
having to check each tuple for nullness when the dump restores the
primary key creation.
pg_upgrading from an older release requires a somewhat brittle procedure
to create a constraint state that matches what would be created if the
database were being created fresh in Postgres 17. I have tested all the
scenarios I could think of, and it works correctly as far as I can tell,
but I could have neglected weird cases.
This patch has been very long in the making. The first patch was
written by Bernd Helmle in 2010 to add a new pg_constraint.contype value
('n'), which I (Álvaro) then hijacked in 2011 and 2012, until that one
was killed by the realization that we ought to use contype='c' instead:
manufactured CHECK constraints. However, later SQL standard
development, as well as nonobvious emergent properties of that design
(mostly, failure to distinguish them from "normal" CHECK constraints as
well as the performance implication of having to test the CHECK
expression) led us to reconsider this choice, so now the current
implementation uses contype='n' again. During Postgres 16 this had
already been introduced by commit e056c557aef4, but there were some
problems mainly with the pg_upgrade procedure that couldn't be fixed in
reasonable time, so it was reverted.
In 2016 Vitaly Burovoy also worked on this feature[1] but found no
consensus for his proposed approach, which was claimed to be closer to
the letter of the standard, requiring an additional pg_attribute column
to track the OID of the not-null constraint for that column.
[1] https://postgr.es/m/CAKOSWNkN6HSyatuys8xZxzRCR-KL1OkHS5-b9qd9bf1Rad3PLA@mail.gmail.com
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Author: Bernd Helmle <mailings@oopsware.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
2023-08-25 13:31:24 +02:00
|
|
|
if (!index->indisunique ||
|
2013-11-08 18:30:43 +01:00
|
|
|
!index->indimmediate ||
|
2018-03-28 02:13:52 +02:00
|
|
|
!heap_attisnull(htup, Anum_pg_index_indpred, NULL))
|
2013-11-08 18:30:43 +01:00
|
|
|
continue;
|
|
|
|
|
Catalog not-null constraints
We now create contype='n' pg_constraint rows for not-null constraints.
We propagate these constraints to other tables during operations such as
adding inheritance relationships, creating and attaching partitions and
creating tables LIKE other tables. We also spawn not-null constraints
for inheritance child tables when their parents have primary keys.
These related constraints mostly follow the well-known rules of
conislocal and coninhcount that we have for CHECK constraints, with some
adaptations: for example, as opposed to CHECK constraints, we don't
match not-null ones by name when descending a hierarchy to alter it,
instead matching by column name that they apply to. This means we don't
require the constraint names to be identical across a hierarchy.
For now, we omit them for system catalogs. Maybe this is worth
reconsidering. We don't support NOT VALID nor DEFERRABLE clauses
either; these can be added as separate features later (this patch is
already large and complicated enough.)
psql shows these constraints in \d+.
pg_dump requires some ad-hoc hacks, particularly when dumping a primary
key. We now create one "throwaway" not-null constraint for each column
in the PK together with the CREATE TABLE command, and once the PK is
created, all those throwaway constraints are removed. This avoids
having to check each tuple for nullness when the dump restores the
primary key creation.
pg_upgrading from an older release requires a somewhat brittle procedure
to create a constraint state that matches what would be created if the
database were being created fresh in Postgres 17. I have tested all the
scenarios I could think of, and it works correctly as far as I can tell,
but I could have neglected weird cases.
This patch has been very long in the making. The first patch was
written by Bernd Helmle in 2010 to add a new pg_constraint.contype value
('n'), which I (Álvaro) then hijacked in 2011 and 2012, until that one
was killed by the realization that we ought to use contype='c' instead:
manufactured CHECK constraints. However, later SQL standard
development, as well as nonobvious emergent properties of that design
(mostly, failure to distinguish them from "normal" CHECK constraints as
well as the performance implication of having to test the CHECK
expression) led us to reconsider this choice, so now the current
implementation uses contype='n' again. During Postgres 16 this had
already been introduced by commit e056c557aef4, but there were some
problems mainly with the pg_upgrade procedure that couldn't be fixed in
reasonable time, so it was reverted.
In 2016 Vitaly Burovoy also worked on this feature[1] but found no
consensus for his proposed approach, which was claimed to be closer to
the letter of the standard, requiring an additional pg_attribute column
to track the OID of the not-null constraint for that column.
[1] https://postgr.es/m/CAKOSWNkN6HSyatuys8xZxzRCR-KL1OkHS5-b9qd9bf1Rad3PLA@mail.gmail.com
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Author: Bernd Helmle <mailings@oopsware.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
2023-08-25 13:31:24 +02:00
|
|
|
/*
|
|
|
|
* Remember primary key index, if any. We do this only if the index
|
|
|
|
* is valid; but if the table is partitioned, then we do it even if
|
|
|
|
* it's invalid.
|
|
|
|
*
|
|
|
|
* The reason for returning invalid primary keys for foreign tables is
|
|
|
|
* because of pg_dump of NOT NULL constraints, and the fact that PKs
|
|
|
|
* remain marked invalid until the partitions' PKs are attached to it.
|
|
|
|
* If we make rd_pkindex invalid, then the attnotnull flag is reset
|
|
|
|
* after the PK is created, which causes the ALTER INDEX ATTACH
|
|
|
|
* PARTITION to fail with 'column ... is not marked NOT NULL'. With
|
|
|
|
* this, dropconstraint_internal() will believe that the columns must
|
|
|
|
* not have attnotnull reset, so the PKs-on-partitions can be attached
|
|
|
|
* correctly, until finally the PK-on-parent is marked valid.
|
|
|
|
*
|
|
|
|
* Also, this doesn't harm anything, because rd_pkindex is not a
|
|
|
|
* "real" index anyway, but a RELKIND_PARTITIONED_INDEX.
|
|
|
|
*/
|
|
|
|
if (index->indisprimary &&
|
|
|
|
(index->indisvalid ||
|
|
|
|
relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
|
2013-11-08 18:30:43 +01:00
|
|
|
pkeyIndex = index->indexrelid;
|
|
|
|
|
Catalog not-null constraints
We now create contype='n' pg_constraint rows for not-null constraints.
We propagate these constraints to other tables during operations such as
adding inheritance relationships, creating and attaching partitions and
creating tables LIKE other tables. We also spawn not-null constraints
for inheritance child tables when their parents have primary keys.
These related constraints mostly follow the well-known rules of
conislocal and coninhcount that we have for CHECK constraints, with some
adaptations: for example, as opposed to CHECK constraints, we don't
match not-null ones by name when descending a hierarchy to alter it,
instead matching by column name that they apply to. This means we don't
require the constraint names to be identical across a hierarchy.
For now, we omit them for system catalogs. Maybe this is worth
reconsidering. We don't support NOT VALID nor DEFERRABLE clauses
either; these can be added as separate features later (this patch is
already large and complicated enough.)
psql shows these constraints in \d+.
pg_dump requires some ad-hoc hacks, particularly when dumping a primary
key. We now create one "throwaway" not-null constraint for each column
in the PK together with the CREATE TABLE command, and once the PK is
created, all those throwaway constraints are removed. This avoids
having to check each tuple for nullness when the dump restores the
primary key creation.
pg_upgrading from an older release requires a somewhat brittle procedure
to create a constraint state that matches what would be created if the
database were being created fresh in Postgres 17. I have tested all the
scenarios I could think of, and it works correctly as far as I can tell,
but I could have neglected weird cases.
This patch has been very long in the making. The first patch was
written by Bernd Helmle in 2010 to add a new pg_constraint.contype value
('n'), which I (Álvaro) then hijacked in 2011 and 2012, until that one
was killed by the realization that we ought to use contype='c' instead:
manufactured CHECK constraints. However, later SQL standard
development, as well as nonobvious emergent properties of that design
(mostly, failure to distinguish them from "normal" CHECK constraints as
well as the performance implication of having to test the CHECK
expression) led us to reconsider this choice, so now the current
implementation uses contype='n' again. During Postgres 16 this had
already been introduced by commit e056c557aef4, but there were some
problems mainly with the pg_upgrade procedure that couldn't be fixed in
reasonable time, so it was reverted.
In 2016 Vitaly Burovoy also worked on this feature[1] but found no
consensus for his proposed approach, which was claimed to be closer to
the letter of the standard, requiring an additional pg_attribute column
to track the OID of the not-null constraint for that column.
[1] https://postgr.es/m/CAKOSWNkN6HSyatuys8xZxzRCR-KL1OkHS5-b9qd9bf1Rad3PLA@mail.gmail.com
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Author: Bernd Helmle <mailings@oopsware.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
2023-08-25 13:31:24 +02:00
|
|
|
if (!index->indisvalid)
|
|
|
|
continue;
|
|
|
|
|
2014-05-14 20:55:48 +02:00
|
|
|
/* remember explicitly chosen replica index */
|
2013-11-08 18:30:43 +01:00
|
|
|
if (index->indisreplident)
|
|
|
|
candidateIndex = index->indexrelid;
|
2000-06-17 23:49:04 +02:00
|
|
|
}
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
systable_endscan(indscan);
|
2013-11-08 18:30:43 +01:00
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(indrel, AccessShareLock);
|
2000-06-17 23:49:04 +02:00
|
|
|
|
Clean up some ad-hoc code for sorting and de-duplicating Lists.
heap.c and relcache.c contained nearly identical copies of logic
to insert OIDs into an OID list while preserving the list's OID
ordering (and rejecting duplicates, in one case but not the other).
The comments argue that this is faster than qsort for small numbers
of OIDs, which is at best unproven, and seems even less likely to be
true now that lappend_cell_oid has to move data around. In any case
it's ugly and hard-to-follow code, and if we do have a lot of OIDs
to consider, it's O(N^2).
Hence, replace with simply lappend'ing OIDs to a List, then list_sort
the completed List, then remove adjacent duplicates if necessary.
This is demonstrably O(N log N) and it's much simpler for the
callers. It's possible that this would be somewhat inefficient
if there were a very large number of duplicates, but that seems
unlikely in the existing usage.
This adds list_deduplicate_oid and list_oid_cmp infrastructure
to list.c. I didn't bother with equivalent functionality for
integer or pointer Lists, but such could always be added later
if we find a use for it.
Discussion: https://postgr.es/m/26193.1563228600@sss.pgh.pa.us
2019-07-16 18:04:06 +02:00
|
|
|
/* Sort the result list into OID order, per API spec. */
|
|
|
|
list_sort(result, list_oid_cmp);
|
|
|
|
|
2000-06-20 01:40:48 +02:00
|
|
|
/* Now save a copy of the completed list in the relcache entry. */
|
2000-06-28 05:33:33 +02:00
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
oldlist = relation->rd_indexlist;
|
2004-05-31 01:40:41 +02:00
|
|
|
relation->rd_indexlist = list_copy(result);
|
2017-01-19 18:00:00 +01:00
|
|
|
relation->rd_pkindex = pkeyIndex;
|
2014-05-14 20:55:48 +02:00
|
|
|
if (replident == REPLICA_IDENTITY_DEFAULT && OidIsValid(pkeyIndex))
|
|
|
|
relation->rd_replidindex = pkeyIndex;
|
|
|
|
else if (replident == REPLICA_IDENTITY_INDEX && OidIsValid(candidateIndex))
|
|
|
|
relation->rd_replidindex = candidateIndex;
|
|
|
|
else
|
|
|
|
relation->rd_replidindex = InvalidOid;
|
2019-05-03 16:26:14 +02:00
|
|
|
relation->rd_indexvalid = true;
|
2000-06-17 23:49:04 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
/* Don't leak the old list, if there is one */
|
|
|
|
list_free(oldlist);
|
|
|
|
|
2000-06-17 23:49:04 +02:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
/*
|
|
|
|
* RelationGetStatExtList
|
2017-05-14 16:54:47 +02:00
|
|
|
* get a list of OIDs of statistics objects on this relation
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
*
|
|
|
|
* The statistics list is created only if someone requests it, in a way
|
|
|
|
* similar to RelationGetIndexList(). We scan pg_statistic_ext to find
|
|
|
|
* relevant statistics, and add the list to the relcache entry so that we
|
|
|
|
* won't have to compute it again. Note that shared cache inval of a
|
|
|
|
* relcache entry will delete the old list and set rd_statvalid to 0,
|
|
|
|
* so that we must recompute the statistics list on next request. This
|
2017-05-14 16:54:47 +02:00
|
|
|
* handles creation or deletion of a statistics object.
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
*
|
|
|
|
* The returned list is guaranteed to be sorted in order by OID, although
|
|
|
|
* this is not currently needed.
|
|
|
|
*
|
|
|
|
* Since shared cache inval causes the relcache's copy of the list to go away,
|
|
|
|
* we return a copy of the list palloc'd in the caller's context. The caller
|
|
|
|
* may list_free() the returned list after scanning it. This is necessary
|
|
|
|
* since the caller will typically be doing syscache lookups on the relevant
|
|
|
|
* statistics, and syscache lookup could cause SI messages to be processed!
|
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetStatExtList(Relation relation)
|
|
|
|
{
|
|
|
|
Relation indrel;
|
|
|
|
SysScanDesc indscan;
|
|
|
|
ScanKeyData skey;
|
|
|
|
HeapTuple htup;
|
|
|
|
List *result;
|
|
|
|
List *oldlist;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the list. */
|
|
|
|
if (relation->rd_statvalid != 0)
|
|
|
|
return list_copy(relation->rd_statlist);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We build the list we intend to return (in the caller's context) while
|
|
|
|
* doing the scan. After successfully completing the scan, we copy that
|
|
|
|
* list into the relcache entry. This avoids cache-context memory leakage
|
|
|
|
* if we get some sort of error partway through.
|
|
|
|
*/
|
|
|
|
result = NIL;
|
|
|
|
|
2017-04-17 23:34:29 +02:00
|
|
|
/*
|
|
|
|
* Prepare to scan pg_statistic_ext for entries having stxrelid = this
|
|
|
|
* rel.
|
|
|
|
*/
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
ScanKeyInit(&skey,
|
2017-04-17 23:34:29 +02:00
|
|
|
Anum_pg_statistic_ext_stxrelid,
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(RelationGetRelid(relation)));
|
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
indrel = table_open(StatisticExtRelationId, AccessShareLock);
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
indscan = systable_beginscan(indrel, StatisticExtRelidIndexId, true,
|
|
|
|
NULL, 1, &skey);
|
|
|
|
|
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(indscan)))
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
{
|
2019-01-15 18:07:10 +01:00
|
|
|
Oid oid = ((Form_pg_statistic_ext) GETSTRUCT(htup))->oid;
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
|
Clean up some ad-hoc code for sorting and de-duplicating Lists.
heap.c and relcache.c contained nearly identical copies of logic
to insert OIDs into an OID list while preserving the list's OID
ordering (and rejecting duplicates, in one case but not the other).
The comments argue that this is faster than qsort for small numbers
of OIDs, which is at best unproven, and seems even less likely to be
true now that lappend_cell_oid has to move data around. In any case
it's ugly and hard-to-follow code, and if we do have a lot of OIDs
to consider, it's O(N^2).
Hence, replace with simply lappend'ing OIDs to a List, then list_sort
the completed List, then remove adjacent duplicates if necessary.
This is demonstrably O(N log N) and it's much simpler for the
callers. It's possible that this would be somewhat inefficient
if there were a very large number of duplicates, but that seems
unlikely in the existing usage.
This adds list_deduplicate_oid and list_oid_cmp infrastructure
to list.c. I didn't bother with equivalent functionality for
integer or pointer Lists, but such could always be added later
if we find a use for it.
Discussion: https://postgr.es/m/26193.1563228600@sss.pgh.pa.us
2019-07-16 18:04:06 +02:00
|
|
|
result = lappend_oid(result, oid);
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
}
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
|
|
|
|
systable_endscan(indscan);
|
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(indrel, AccessShareLock);
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
|
Clean up some ad-hoc code for sorting and de-duplicating Lists.
heap.c and relcache.c contained nearly identical copies of logic
to insert OIDs into an OID list while preserving the list's OID
ordering (and rejecting duplicates, in one case but not the other).
The comments argue that this is faster than qsort for small numbers
of OIDs, which is at best unproven, and seems even less likely to be
true now that lappend_cell_oid has to move data around. In any case
it's ugly and hard-to-follow code, and if we do have a lot of OIDs
to consider, it's O(N^2).
Hence, replace with simply lappend'ing OIDs to a List, then list_sort
the completed List, then remove adjacent duplicates if necessary.
This is demonstrably O(N log N) and it's much simpler for the
callers. It's possible that this would be somewhat inefficient
if there were a very large number of duplicates, but that seems
unlikely in the existing usage.
This adds list_deduplicate_oid and list_oid_cmp infrastructure
to list.c. I didn't bother with equivalent functionality for
integer or pointer Lists, but such could always be added later
if we find a use for it.
Discussion: https://postgr.es/m/26193.1563228600@sss.pgh.pa.us
2019-07-16 18:04:06 +02:00
|
|
|
/* Sort the result list into OID order, per API spec. */
|
|
|
|
list_sort(result, list_oid_cmp);
|
|
|
|
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
/* Now save a copy of the completed list in the relcache entry. */
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
oldlist = relation->rd_statlist;
|
|
|
|
relation->rd_statlist = list_copy(result);
|
|
|
|
|
|
|
|
relation->rd_statvalid = true;
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
/* Don't leak the old list, if there is one */
|
|
|
|
list_free(oldlist);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2017-01-19 18:00:00 +01:00
|
|
|
/*
|
|
|
|
* RelationGetPrimaryKeyIndex -- get OID of the relation's primary key index
|
|
|
|
*
|
|
|
|
* Returns InvalidOid if there is no such index.
|
|
|
|
*/
|
|
|
|
Oid
|
|
|
|
RelationGetPrimaryKeyIndex(Relation relation)
|
|
|
|
{
|
|
|
|
List *ilist;
|
|
|
|
|
2019-05-03 16:26:14 +02:00
|
|
|
if (!relation->rd_indexvalid)
|
2017-01-19 18:00:00 +01:00
|
|
|
{
|
|
|
|
/* RelationGetIndexList does the heavy lifting. */
|
|
|
|
ilist = RelationGetIndexList(relation);
|
|
|
|
list_free(ilist);
|
2019-05-03 16:26:14 +02:00
|
|
|
Assert(relation->rd_indexvalid);
|
2017-01-19 18:00:00 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
return relation->rd_pkindex;
|
|
|
|
}
|
|
|
|
|
2014-05-14 20:55:48 +02:00
|
|
|
/*
|
|
|
|
* RelationGetReplicaIndex -- get OID of the relation's replica identity index
|
|
|
|
*
|
|
|
|
* Returns InvalidOid if there is no such index.
|
|
|
|
*/
|
|
|
|
Oid
|
|
|
|
RelationGetReplicaIndex(Relation relation)
|
|
|
|
{
|
|
|
|
List *ilist;
|
|
|
|
|
2019-05-03 16:26:14 +02:00
|
|
|
if (!relation->rd_indexvalid)
|
2014-05-14 20:55:48 +02:00
|
|
|
{
|
|
|
|
/* RelationGetIndexList does the heavy lifting. */
|
|
|
|
ilist = RelationGetIndexList(relation);
|
|
|
|
list_free(ilist);
|
2019-05-03 16:26:14 +02:00
|
|
|
Assert(relation->rd_indexvalid);
|
2014-05-14 20:55:48 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
return relation->rd_replidindex;
|
|
|
|
}
|
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
/*
|
|
|
|
* RelationGetIndexExpressions -- get the index expressions for an index
|
|
|
|
*
|
|
|
|
* We cache the result of transforming pg_index.indexprs into a node tree.
|
|
|
|
* If the rel is not an index or has no expressional columns, we return NIL.
|
|
|
|
* Otherwise, the returned tree is copied into the caller's memory context.
|
|
|
|
* (We don't want to return a pointer to the relcache copy, since it could
|
|
|
|
* disappear due to relcache invalidation.)
|
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetIndexExpressions(Relation relation)
|
|
|
|
{
|
|
|
|
List *result;
|
|
|
|
Datum exprsDatum;
|
|
|
|
bool isnull;
|
|
|
|
char *exprsString;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the result. */
|
|
|
|
if (relation->rd_indexprs)
|
2017-03-09 21:18:59 +01:00
|
|
|
return copyObject(relation->rd_indexprs);
|
2003-05-28 18:04:02 +02:00
|
|
|
|
|
|
|
/* Quick exit if there is nothing to do. */
|
|
|
|
if (relation->rd_indextuple == NULL ||
|
2018-03-28 02:13:52 +02:00
|
|
|
heap_attisnull(relation->rd_indextuple, Anum_pg_index_indexprs, NULL))
|
2003-05-28 18:04:02 +02:00
|
|
|
return NIL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We build the tree we intend to return in the caller's context. After
|
|
|
|
* successfully completing the work, we copy it into the relcache entry.
|
|
|
|
* This avoids problems if we get some sort of error partway through.
|
2005-03-29 02:17:27 +02:00
|
|
|
*/
|
|
|
|
exprsDatum = heap_getattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indexprs,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
2003-05-28 18:04:02 +02:00
|
|
|
Assert(!isnull);
|
2008-03-25 23:42:46 +01:00
|
|
|
exprsString = TextDatumGetCString(exprsDatum);
|
2003-05-28 18:04:02 +02:00
|
|
|
result = (List *) stringToNode(exprsString);
|
|
|
|
pfree(exprsString);
|
|
|
|
|
|
|
|
/*
|
2005-03-28 02:58:26 +02:00
|
|
|
* Run the expressions through eval_const_expressions. This is not just an
|
|
|
|
* optimization, but is necessary, because the planner will be comparing
|
|
|
|
* them to similarly-processed qual clauses, and may fail to detect valid
|
Fix improper uses of canonicalize_qual().
One of the things canonicalize_qual() does is to remove constant-NULL
subexpressions of top-level AND/OR clauses. It does that on the assumption
that what it's given is a top-level WHERE clause, so that NULL can be
treated like FALSE. Although this is documented down inside a subroutine
of canonicalize_qual(), it wasn't mentioned in the documentation of that
function itself, and some callers hadn't gotten that memo.
Notably, commit d007a9505 caused get_relation_constraints() to apply
canonicalize_qual() to CHECK constraints. That allowed constraint
exclusion to misoptimize situations in which a CHECK constraint had a
provably-NULL subclause, as seen in the regression test case added here,
in which a child table that should be scanned is not. (Although this
thinko is ancient, the test case doesn't fail before 9.2, for reasons
I've not bothered to track down in detail. There may be related cases
that do fail before that.)
More recently, commit f0e44751d added an independent bug by applying
canonicalize_qual() to index expressions, which is even sillier since
those might not even be boolean. If they are, though, I think this
could lead to making incorrect index entries for affected index
expressions in v10. I haven't attempted to prove that though.
To fix, add an "is_check" parameter to canonicalize_qual() to specify
whether it should assume WHERE or CHECK semantics, and make it perform
NULL-elimination accordingly. Adjust the callers to apply the right
semantics, or remove the call entirely in cases where it's not known
that the expression has one or the other semantics. I also removed
the call in some cases involving partition expressions, where it should
be a no-op because such expressions should be canonical already ...
and was a no-op, independently of whether it could in principle have
done something, because it was being handed the qual in implicit-AND
format which isn't what it expects. In HEAD, add an Assert to catch
that type of mistake in future.
This represents an API break for external callers of canonicalize_qual().
While that's intentional in HEAD to make such callers think about which
case applies to them, it seems like something we probably wouldn't be
thanked for in released branches. Hence, in released branches, the
extra parameter is added to a new function canonicalize_qual_ext(),
and canonicalize_qual() is a wrapper that retains its old behavior.
Patch by me with suggestions from Dean Rasheed. Back-patch to all
supported branches.
Discussion: https://postgr.es/m/24475.1520635069@sss.pgh.pa.us
2018-03-11 23:10:42 +01:00
|
|
|
* matches without this. We must not use canonicalize_qual, however,
|
|
|
|
* since these aren't qual expressions.
|
2003-05-28 18:04:02 +02:00
|
|
|
*/
|
2008-04-01 02:48:33 +02:00
|
|
|
result = (List *) eval_const_expressions(NULL, (Node *) result);
|
2003-05-28 18:04:02 +02:00
|
|
|
|
|
|
|
/* May as well fix opfuncids too */
|
|
|
|
fix_opfuncids((Node *) result);
|
|
|
|
|
|
|
|
/* Now save a copy of the completed tree in the relcache entry. */
|
2010-01-12 19:12:18 +01:00
|
|
|
oldcxt = MemoryContextSwitchTo(relation->rd_indexcxt);
|
2017-03-09 21:18:59 +01:00
|
|
|
relation->rd_indexprs = copyObject(result);
|
2003-05-28 18:04:02 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
Fix misbehavior with expression indexes on ON COMMIT DELETE ROWS tables.
We implement ON COMMIT DELETE ROWS by truncating tables marked that
way, which requires also truncating/rebuilding their indexes. But
RelationTruncateIndexes asks the relcache for up-to-date copies of any
index expressions, which may cause execution of eval_const_expressions
on them, which can result in actual execution of subexpressions.
This is a bad thing to have happening during ON COMMIT. Manuel Rigger
reported that use of a SQL function resulted in crashes due to
expectations that ActiveSnapshot would be set, which it isn't.
The most obvious fix perhaps would be to push a snapshot during
PreCommit_on_commit_actions, but I think that would just open the door
to more problems: CommitTransaction explicitly expects that no
user-defined code can be running at this point.
Fortunately, since we know that no tuples exist to be indexed, there
seems no need to use the real index expressions or predicates during
RelationTruncateIndexes. We can set up dummy index expressions
instead (we do need something that will expose the right data type,
as there are places that build index tupdescs based on this), and
just ignore predicates and exclusion constraints.
In a green field it'd likely be better to reimplement ON COMMIT DELETE
ROWS using the same "init fork" infrastructure used for unlogged
relations. That seems impractical without catalog changes though,
and even without that it'd be too big a change to back-patch.
So for now do it like this.
Per private report from Manuel Rigger. This has been broken forever,
so back-patch to all supported branches.
2019-12-01 19:09:26 +01:00
|
|
|
/*
|
|
|
|
* RelationGetDummyIndexExpressions -- get dummy expressions for an index
|
|
|
|
*
|
|
|
|
* Return a list of dummy expressions (just Const nodes) with the same
|
|
|
|
* types/typmods/collations as the index's real expressions. This is
|
|
|
|
* useful in situations where we don't want to run any user-defined code.
|
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetDummyIndexExpressions(Relation relation)
|
|
|
|
{
|
|
|
|
List *result;
|
|
|
|
Datum exprsDatum;
|
|
|
|
bool isnull;
|
|
|
|
char *exprsString;
|
|
|
|
List *rawExprs;
|
|
|
|
ListCell *lc;
|
|
|
|
|
|
|
|
/* Quick exit if there is nothing to do. */
|
|
|
|
if (relation->rd_indextuple == NULL ||
|
|
|
|
heap_attisnull(relation->rd_indextuple, Anum_pg_index_indexprs, NULL))
|
|
|
|
return NIL;
|
|
|
|
|
|
|
|
/* Extract raw node tree(s) from index tuple. */
|
|
|
|
exprsDatum = heap_getattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indexprs,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
|
|
|
Assert(!isnull);
|
|
|
|
exprsString = TextDatumGetCString(exprsDatum);
|
|
|
|
rawExprs = (List *) stringToNode(exprsString);
|
|
|
|
pfree(exprsString);
|
|
|
|
|
|
|
|
/* Construct null Consts; the typlen and typbyval are arbitrary. */
|
|
|
|
result = NIL;
|
|
|
|
foreach(lc, rawExprs)
|
|
|
|
{
|
|
|
|
Node *rawExpr = (Node *) lfirst(lc);
|
|
|
|
|
|
|
|
result = lappend(result,
|
|
|
|
makeConst(exprType(rawExpr),
|
|
|
|
exprTypmod(rawExpr),
|
|
|
|
exprCollation(rawExpr),
|
|
|
|
1,
|
|
|
|
(Datum) 0,
|
|
|
|
true,
|
|
|
|
true));
|
|
|
|
}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
/*
|
|
|
|
* RelationGetIndexPredicate -- get the index predicate for an index
|
|
|
|
*
|
2003-12-28 22:57:37 +01:00
|
|
|
* We cache the result of transforming pg_index.indpred into an implicit-AND
|
2017-03-26 17:36:46 +02:00
|
|
|
* node tree (suitable for use in planning).
|
2003-05-28 18:04:02 +02:00
|
|
|
* If the rel is not an index or has no predicate, we return NIL.
|
|
|
|
* Otherwise, the returned tree is copied into the caller's memory context.
|
|
|
|
* (We don't want to return a pointer to the relcache copy, since it could
|
|
|
|
* disappear due to relcache invalidation.)
|
|
|
|
*/
|
|
|
|
List *
|
|
|
|
RelationGetIndexPredicate(Relation relation)
|
|
|
|
{
|
|
|
|
List *result;
|
|
|
|
Datum predDatum;
|
|
|
|
bool isnull;
|
|
|
|
char *predString;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the result. */
|
|
|
|
if (relation->rd_indpred)
|
2017-03-09 21:18:59 +01:00
|
|
|
return copyObject(relation->rd_indpred);
|
2003-05-28 18:04:02 +02:00
|
|
|
|
|
|
|
/* Quick exit if there is nothing to do. */
|
|
|
|
if (relation->rd_indextuple == NULL ||
|
2018-03-28 02:13:52 +02:00
|
|
|
heap_attisnull(relation->rd_indextuple, Anum_pg_index_indpred, NULL))
|
2003-05-28 18:04:02 +02:00
|
|
|
return NIL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We build the tree we intend to return in the caller's context. After
|
|
|
|
* successfully completing the work, we copy it into the relcache entry.
|
|
|
|
* This avoids problems if we get some sort of error partway through.
|
2005-03-29 02:17:27 +02:00
|
|
|
*/
|
|
|
|
predDatum = heap_getattr(relation->rd_indextuple,
|
|
|
|
Anum_pg_index_indpred,
|
|
|
|
GetPgIndexDescriptor(),
|
|
|
|
&isnull);
|
2003-05-28 18:04:02 +02:00
|
|
|
Assert(!isnull);
|
2008-03-25 23:42:46 +01:00
|
|
|
predString = TextDatumGetCString(predDatum);
|
2003-05-28 18:04:02 +02:00
|
|
|
result = (List *) stringToNode(predString);
|
|
|
|
pfree(predString);
|
|
|
|
|
|
|
|
/*
|
2005-03-28 02:58:26 +02:00
|
|
|
* Run the expression through const-simplification and canonicalization.
|
|
|
|
* This is not just an optimization, but is necessary, because the planner
|
|
|
|
* will be comparing it to similarly-processed qual clauses, and may fail
|
|
|
|
* to detect valid matches without this. This must match the processing
|
|
|
|
* done to qual clauses in preprocess_expression()! (We can skip the
|
|
|
|
* stuff involving subqueries, however, since we don't allow any in index
|
|
|
|
* predicates.)
|
2003-05-28 18:04:02 +02:00
|
|
|
*/
|
2008-04-01 02:48:33 +02:00
|
|
|
result = (List *) eval_const_expressions(NULL, (Node *) result);
|
2003-05-28 18:04:02 +02:00
|
|
|
|
Fix improper uses of canonicalize_qual().
One of the things canonicalize_qual() does is to remove constant-NULL
subexpressions of top-level AND/OR clauses. It does that on the assumption
that what it's given is a top-level WHERE clause, so that NULL can be
treated like FALSE. Although this is documented down inside a subroutine
of canonicalize_qual(), it wasn't mentioned in the documentation of that
function itself, and some callers hadn't gotten that memo.
Notably, commit d007a9505 caused get_relation_constraints() to apply
canonicalize_qual() to CHECK constraints. That allowed constraint
exclusion to misoptimize situations in which a CHECK constraint had a
provably-NULL subclause, as seen in the regression test case added here,
in which a child table that should be scanned is not. (Although this
thinko is ancient, the test case doesn't fail before 9.2, for reasons
I've not bothered to track down in detail. There may be related cases
that do fail before that.)
More recently, commit f0e44751d added an independent bug by applying
canonicalize_qual() to index expressions, which is even sillier since
those might not even be boolean. If they are, though, I think this
could lead to making incorrect index entries for affected index
expressions in v10. I haven't attempted to prove that though.
To fix, add an "is_check" parameter to canonicalize_qual() to specify
whether it should assume WHERE or CHECK semantics, and make it perform
NULL-elimination accordingly. Adjust the callers to apply the right
semantics, or remove the call entirely in cases where it's not known
that the expression has one or the other semantics. I also removed
the call in some cases involving partition expressions, where it should
be a no-op because such expressions should be canonical already ...
and was a no-op, independently of whether it could in principle have
done something, because it was being handed the qual in implicit-AND
format which isn't what it expects. In HEAD, add an Assert to catch
that type of mistake in future.
This represents an API break for external callers of canonicalize_qual().
While that's intentional in HEAD to make such callers think about which
case applies to them, it seems like something we probably wouldn't be
thanked for in released branches. Hence, in released branches, the
extra parameter is added to a new function canonicalize_qual_ext(),
and canonicalize_qual() is a wrapper that retains its old behavior.
Patch by me with suggestions from Dean Rasheed. Back-patch to all
supported branches.
Discussion: https://postgr.es/m/24475.1520635069@sss.pgh.pa.us
2018-03-11 23:10:42 +01:00
|
|
|
result = (List *) canonicalize_qual((Expr *) result, false);
|
2005-03-28 02:58:26 +02:00
|
|
|
|
2003-12-28 22:57:37 +01:00
|
|
|
/* Also convert to implicit-AND format */
|
|
|
|
result = make_ands_implicit((Expr *) result);
|
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
/* May as well fix opfuncids too */
|
|
|
|
fix_opfuncids((Node *) result);
|
|
|
|
|
|
|
|
/* Now save a copy of the completed tree in the relcache entry. */
|
2010-01-12 19:12:18 +01:00
|
|
|
oldcxt = MemoryContextSwitchTo(relation->rd_indexcxt);
|
2017-03-09 21:18:59 +01:00
|
|
|
relation->rd_indpred = copyObject(result);
|
2003-05-28 18:04:02 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/*
|
|
|
|
* RelationGetIndexAttrBitmap -- get a bitmap of index attribute numbers
|
|
|
|
*
|
|
|
|
* The result has a bit set for each attribute used anywhere in the index
|
|
|
|
* definitions of all the indexes on this relation. (This includes not only
|
|
|
|
* simple index keys, but attributes used in expressions and partial-index
|
|
|
|
* predicates.)
|
|
|
|
*
|
2023-08-10 12:04:07 +02:00
|
|
|
* Depending on attrKind, a bitmap covering attnums for certain columns is
|
|
|
|
* returned:
|
|
|
|
* INDEX_ATTR_BITMAP_KEY Columns in non-partial unique indexes not
|
|
|
|
* in expressions (i.e., usable for FKs)
|
|
|
|
* INDEX_ATTR_BITMAP_PRIMARY_KEY Columns in the table's primary key
|
|
|
|
* INDEX_ATTR_BITMAP_IDENTITY_KEY Columns in the table's replica identity
|
|
|
|
* index (empty if FULL)
|
|
|
|
* INDEX_ATTR_BITMAP_HOT_BLOCKING Columns that block updates from being HOT
|
|
|
|
* INDEX_ATTR_BITMAP_SUMMARIZED Columns included in summarizing indexes
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
*
|
2007-09-20 19:56:33 +02:00
|
|
|
* Attribute numbers are offset by FirstLowInvalidHeapAttributeNumber so that
|
|
|
|
* we can include system attributes (e.g., OID) in the bitmap representation.
|
|
|
|
*
|
2011-03-22 18:00:24 +01:00
|
|
|
* Caller had better hold at least RowExclusiveLock on the target relation
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
* to ensure it is safe (deadlock-free) for us to take locks on the relation's
|
|
|
|
* indexes. Note that since the introduction of CREATE INDEX CONCURRENTLY,
|
|
|
|
* that lock level doesn't guarantee a stable set of indexes, so we have to
|
|
|
|
* be prepared to retry here in case of a change in the set of indexes.
|
2011-03-22 18:00:24 +01:00
|
|
|
*
|
2007-09-20 19:56:33 +02:00
|
|
|
* The returned result is palloc'd in the caller's memory context and should
|
|
|
|
* be bms_free'd when not needed anymore.
|
|
|
|
*/
|
|
|
|
Bitmapset *
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
|
2007-09-20 19:56:33 +02:00
|
|
|
{
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
Bitmapset *uindexattrs; /* columns in unique indexes */
|
2017-01-19 18:00:00 +01:00
|
|
|
Bitmapset *pkindexattrs; /* columns in the primary index */
|
2013-12-13 18:58:48 +01:00
|
|
|
Bitmapset *idindexattrs; /* columns in the replica identity */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */
|
|
|
|
Bitmapset *summarizedattrs; /* columns with summarizing indexes */
|
2007-09-20 19:56:33 +02:00
|
|
|
List *indexoidlist;
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
List *newindexoidlist;
|
2017-01-19 18:00:00 +01:00
|
|
|
Oid relpkindex;
|
2014-05-14 20:55:48 +02:00
|
|
|
Oid relreplindex;
|
2007-09-20 19:56:33 +02:00
|
|
|
ListCell *l;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the result. */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
if (relation->rd_attrsvalid)
|
2014-05-14 20:55:48 +02:00
|
|
|
{
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
switch (attrKind)
|
|
|
|
{
|
2014-05-14 20:55:48 +02:00
|
|
|
case INDEX_ATTR_BITMAP_KEY:
|
|
|
|
return bms_copy(relation->rd_keyattr);
|
2017-01-19 18:00:00 +01:00
|
|
|
case INDEX_ATTR_BITMAP_PRIMARY_KEY:
|
|
|
|
return bms_copy(relation->rd_pkattr);
|
2014-05-14 20:55:48 +02:00
|
|
|
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
|
|
|
|
return bms_copy(relation->rd_idattr);
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
case INDEX_ATTR_BITMAP_HOT_BLOCKING:
|
|
|
|
return bms_copy(relation->rd_hotblockingattr);
|
|
|
|
case INDEX_ATTR_BITMAP_SUMMARIZED:
|
|
|
|
return bms_copy(relation->rd_summarizedattr);
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
default:
|
|
|
|
elog(ERROR, "unknown attrKind %u", attrKind);
|
|
|
|
}
|
2014-05-14 20:55:48 +02:00
|
|
|
}
|
2007-09-20 19:56:33 +02:00
|
|
|
|
|
|
|
/* Fast path if definitely no indexes */
|
|
|
|
if (!RelationGetForm(relation)->relhasindex)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
/*
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
* Get cached list of index OIDs. If we have to start over, we do so here.
|
2007-09-20 19:56:33 +02:00
|
|
|
*/
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
restart:
|
2007-09-20 19:56:33 +02:00
|
|
|
indexoidlist = RelationGetIndexList(relation);
|
|
|
|
|
|
|
|
/* Fall out if no indexes (but relhasindex was set) */
|
|
|
|
if (indexoidlist == NIL)
|
|
|
|
return NULL;
|
|
|
|
|
2014-05-14 20:55:48 +02:00
|
|
|
/*
|
2017-02-06 17:31:23 +01:00
|
|
|
* Copy the rd_pkindex and rd_replidindex values computed by
|
2017-01-19 18:00:00 +01:00
|
|
|
* RelationGetIndexList before proceeding. This is needed because a
|
|
|
|
* relcache flush could occur inside index_open below, resetting the
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
* fields managed by RelationGetIndexList. We need to do the work with
|
|
|
|
* stable values of these fields.
|
2014-05-14 20:55:48 +02:00
|
|
|
*/
|
2017-01-19 18:00:00 +01:00
|
|
|
relpkindex = relation->rd_pkindex;
|
2014-05-14 20:55:48 +02:00
|
|
|
relreplindex = relation->rd_replidindex;
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/*
|
2022-06-16 15:02:48 +02:00
|
|
|
* For each index, add referenced attributes to indexattrs.
|
Fix assorted bugs in CREATE/DROP INDEX CONCURRENTLY.
Commit 8cb53654dbdb4c386369eb988062d0bbb6de725e, which introduced DROP
INDEX CONCURRENTLY, managed to break CREATE INDEX CONCURRENTLY via a poor
choice of catalog state representation. The pg_index state for an index
that's reached the final pre-drop stage was the same as the state for an
index just created by CREATE INDEX CONCURRENTLY. This meant that the
(necessary) change to make RelationGetIndexList ignore about-to-die indexes
also made it ignore freshly-created indexes; which is catastrophic because
the latter do need to be considered in HOT-safety decisions. Failure to
do so leads to incorrect index entries and subsequently wrong results from
queries depending on the concurrently-created index.
To fix, add an additional boolean column "indislive" to pg_index, so that
the freshly-created and about-to-die states can be distinguished. (This
change obviously is only possible in HEAD. This patch will need to be
back-patched, but in 9.2 we'll use a kluge consisting of overloading the
formerly-impossible state of indisvalid = true and indisready = false.)
In addition, change CREATE/DROP INDEX CONCURRENTLY so that the pg_index
flag changes they make without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update. The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption. This is a pre-existing bug in CREATE INDEX
CONCURRENTLY, which was copied into the DROP code.
In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate. These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.
Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation. Previously we
could have been left with stale values of some fields in an index relcache
entry. It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.
In addition, do some code and docs review for DROP INDEX CONCURRENTLY;
some cosmetic code cleanup but mostly addition and revision of comments.
This will need to be back-patched, but in a noticeably different form,
so I'm committing it to HEAD before working on the back-patch.
Problem reported by Amit Kapila, diagnosis by Pavan Deolassee,
fix by Tom Lane and Andres Freund.
2012-11-29 03:25:27 +01:00
|
|
|
*
|
|
|
|
* Note: we consider all indexes returned by RelationGetIndexList, even if
|
|
|
|
* they are not indisready or indisvalid. This is important because an
|
|
|
|
* index for which CREATE INDEX CONCURRENTLY has just started must be
|
|
|
|
* included in HOT-safety decisions (see README.HOT). If a DROP INDEX
|
|
|
|
* CONCURRENTLY is far enough along that we should ignore the index, it
|
|
|
|
* won't be returned at all by RelationGetIndexList.
|
2007-09-20 19:56:33 +02:00
|
|
|
*/
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
uindexattrs = NULL;
|
2017-01-19 18:00:00 +01:00
|
|
|
pkindexattrs = NULL;
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
idindexattrs = NULL;
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
hotblockingattrs = NULL;
|
|
|
|
summarizedattrs = NULL;
|
2007-09-20 19:56:33 +02:00
|
|
|
foreach(l, indexoidlist)
|
|
|
|
{
|
|
|
|
Oid indexOid = lfirst_oid(l);
|
|
|
|
Relation indexDesc;
|
2019-01-28 22:09:33 +01:00
|
|
|
Datum datum;
|
|
|
|
bool isnull;
|
|
|
|
Node *indexExpressions;
|
|
|
|
Node *indexPredicate;
|
2007-09-20 19:56:33 +02:00
|
|
|
int i;
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
bool isKey; /* candidate key */
|
2017-01-19 18:00:00 +01:00
|
|
|
bool isPK; /* primary key */
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
bool isIDKey; /* replica identity index */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
Bitmapset **attrs;
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
indexDesc = index_open(indexOid, AccessShareLock);
|
|
|
|
|
2019-01-28 22:09:33 +01:00
|
|
|
/*
|
|
|
|
* Extract index expressions and index predicate. Note: Don't use
|
|
|
|
* RelationGetIndexExpressions()/RelationGetIndexPredicate(), because
|
|
|
|
* those might run constant expressions evaluation, which needs a
|
|
|
|
* snapshot, which we might not have here. (Also, it's probably more
|
|
|
|
* sound to collect the bitmaps before any transformations that might
|
|
|
|
* eliminate columns, but the practical impact of this is limited.)
|
|
|
|
*/
|
|
|
|
|
|
|
|
datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indexprs,
|
|
|
|
GetPgIndexDescriptor(), &isnull);
|
|
|
|
if (!isnull)
|
|
|
|
indexExpressions = stringToNode(TextDatumGetCString(datum));
|
|
|
|
else
|
|
|
|
indexExpressions = NULL;
|
|
|
|
|
|
|
|
datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indpred,
|
|
|
|
GetPgIndexDescriptor(), &isnull);
|
|
|
|
if (!isnull)
|
|
|
|
indexPredicate = stringToNode(TextDatumGetCString(datum));
|
|
|
|
else
|
|
|
|
indexPredicate = NULL;
|
2007-09-20 19:56:33 +02:00
|
|
|
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
/* Can this index be referenced by a foreign key? */
|
2019-01-28 22:09:33 +01:00
|
|
|
isKey = indexDesc->rd_index->indisunique &&
|
|
|
|
indexExpressions == NULL &&
|
|
|
|
indexPredicate == NULL;
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
|
2017-01-19 18:00:00 +01:00
|
|
|
/* Is this a primary key? */
|
|
|
|
isPK = (indexOid == relpkindex);
|
|
|
|
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
/* Is this index the configured (or default) replica identity? */
|
2014-05-14 20:55:48 +02:00
|
|
|
isIDKey = (indexOid == relreplindex);
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
/*
|
|
|
|
* If the index is summarizing, it doesn't block HOT updates, but we
|
|
|
|
* may still need to update it (if the attributes were modified). So
|
|
|
|
* decide which bitmap we'll update in the following loop.
|
|
|
|
*/
|
|
|
|
if (indexDesc->rd_indam->amsummarizing)
|
|
|
|
attrs = &summarizedattrs;
|
|
|
|
else
|
|
|
|
attrs = &hotblockingattrs;
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/* Collect simple attribute references */
|
2019-01-28 22:09:33 +01:00
|
|
|
for (i = 0; i < indexDesc->rd_index->indnatts; i++)
|
2007-09-20 19:56:33 +02:00
|
|
|
{
|
2019-01-28 22:09:33 +01:00
|
|
|
int attrnum = indexDesc->rd_index->indkey.values[i];
|
2007-09-20 19:56:33 +02:00
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
/*
|
|
|
|
* Since we have covering indexes with non-key columns, we must
|
|
|
|
* handle them accurately here. non-key columns must be added into
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
* hotblockingattrs or summarizedattrs, since they are in index,
|
|
|
|
* and update shouldn't miss them.
|
|
|
|
*
|
|
|
|
* Summarizing indexes do not block HOT, but do need to be updated
|
|
|
|
* when the column value changes, thus require a separate
|
|
|
|
* attribute bitmapset.
|
|
|
|
*
|
|
|
|
* Obviously, non-key columns couldn't be referenced by foreign
|
2018-04-07 22:00:39 +02:00
|
|
|
* key or identity key. Hence we do not include them into
|
|
|
|
* uindexattrs, pkindexattrs and idindexattrs bitmaps.
|
|
|
|
*/
|
2007-09-20 19:56:33 +02:00
|
|
|
if (attrnum != 0)
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
{
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
*attrs = bms_add_member(*attrs,
|
|
|
|
attrnum - FirstLowInvalidHeapAttributeNumber);
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
|
2019-01-28 22:09:33 +01:00
|
|
|
if (isKey && i < indexDesc->rd_index->indnkeyatts)
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
uindexattrs = bms_add_member(uindexattrs,
|
|
|
|
attrnum - FirstLowInvalidHeapAttributeNumber);
|
2014-05-14 20:55:48 +02:00
|
|
|
|
2019-01-28 22:09:33 +01:00
|
|
|
if (isPK && i < indexDesc->rd_index->indnkeyatts)
|
2017-01-19 18:00:00 +01:00
|
|
|
pkindexattrs = bms_add_member(pkindexattrs,
|
|
|
|
attrnum - FirstLowInvalidHeapAttributeNumber);
|
|
|
|
|
2019-01-28 22:09:33 +01:00
|
|
|
if (isIDKey && i < indexDesc->rd_index->indnkeyatts)
|
2014-05-14 20:55:48 +02:00
|
|
|
idindexattrs = bms_add_member(idindexattrs,
|
|
|
|
attrnum - FirstLowInvalidHeapAttributeNumber);
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
}
|
2007-09-20 19:56:33 +02:00
|
|
|
}
|
|
|
|
|
2019-01-15 18:07:10 +01:00
|
|
|
/* Collect all attributes used in expressions, too */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
pull_varattnos(indexExpressions, 1, attrs);
|
2019-01-15 18:07:10 +01:00
|
|
|
|
2022-06-16 15:02:48 +02:00
|
|
|
/* Collect all attributes in the index predicate, too */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
pull_varattnos(indexPredicate, 1, attrs);
|
2007-09-20 19:56:33 +02:00
|
|
|
|
|
|
|
index_close(indexDesc, AccessShareLock);
|
|
|
|
}
|
|
|
|
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
/*
|
|
|
|
* During one of the index_opens in the above loop, we might have received
|
|
|
|
* a relcache flush event on this relcache entry, which might have been
|
|
|
|
* signaling a change in the rel's index list. If so, we'd better start
|
|
|
|
* over to ensure we deliver up-to-date attribute bitmaps.
|
|
|
|
*/
|
|
|
|
newindexoidlist = RelationGetIndexList(relation);
|
|
|
|
if (equal(indexoidlist, newindexoidlist) &&
|
|
|
|
relpkindex == relation->rd_pkindex &&
|
|
|
|
relreplindex == relation->rd_replidindex)
|
|
|
|
{
|
|
|
|
/* Still the same index set, so proceed */
|
|
|
|
list_free(newindexoidlist);
|
|
|
|
list_free(indexoidlist);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Gotta do it over ... might as well not leak memory */
|
|
|
|
list_free(newindexoidlist);
|
|
|
|
list_free(indexoidlist);
|
|
|
|
bms_free(uindexattrs);
|
|
|
|
bms_free(pkindexattrs);
|
|
|
|
bms_free(idindexattrs);
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
bms_free(hotblockingattrs);
|
|
|
|
bms_free(summarizedattrs);
|
Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap().
The problem with the original coding here is that we might receive (and
clear) a relcache invalidation signal for the target relation down inside
one of the index_open calls we're doing. Since the target is open, we
would not drop the relcache entry, just reset its rd_indexvalid and
rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and
would eventually cache and return potentially-obsolete attribute bitmaps.
The case where this matters is where the inval signal was from a CREATE
INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed
column. (In all other cases, the lock we hold on the target rel should
prevent any concurrent change in index state.) Even just returning the
stale attribute bitmap is not such a problem, because it shouldn't matter
during the transaction in which we receive the signal. What hurts is
caching the stale data, because it can survive into later transactions,
breaking CREATE INDEX CONCURRENTLY's expectation that later transactions
will not create new broken HOT chains. The upshot is that there's a window
for building corrupted indexes during CREATE INDEX CONCURRENTLY.
This patch fixes the problem by rechecking that the set of index OIDs
is still the same at the end of RelationGetIndexAttrBitmap() as it was
at the start. If not, we loop back and try again. That's a little
more than is strictly necessary to fix the bug --- in principle, we
could return the stale data but not cache it --- but it seems like a
bad idea on general principles for relcache to return data it knows
is stale.
There might be more hazards of the same ilk, or there might be a better
way to fix this one, but this patch definitely improves matters and seems
unlikely to make anything worse. So let's push it into today's releases
even as we continue to study the problem.
Pavan Deolasee and myself
Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com
2017-02-06 19:19:50 +01:00
|
|
|
|
|
|
|
goto restart;
|
|
|
|
}
|
2007-09-20 19:56:33 +02:00
|
|
|
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
/* Don't leak the old values of these bitmaps, if any */
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
relation->rd_attrsvalid = false;
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
bms_free(relation->rd_keyattr);
|
|
|
|
relation->rd_keyattr = NULL;
|
2017-01-19 18:00:00 +01:00
|
|
|
bms_free(relation->rd_pkattr);
|
|
|
|
relation->rd_pkattr = NULL;
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
bms_free(relation->rd_idattr);
|
|
|
|
relation->rd_idattr = NULL;
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
bms_free(relation->rd_hotblockingattr);
|
|
|
|
relation->rd_hotblockingattr = NULL;
|
|
|
|
bms_free(relation->rd_summarizedattr);
|
|
|
|
relation->rd_summarizedattr = NULL;
|
Prevent memory leaks in RelationGetIndexList, RelationGetIndexAttrBitmap.
When replacing rd_indexlist, rd_indexattr, etc, we neglected to pfree any
old value of these fields. Under ordinary circumstances, the old value
would always be NULL, so this seemed reasonable enough. However, in cases
where we're rebuilding a system catalog's relcache entry and another cache
flush occurs on that same catalog meanwhile, it's possible for the field to
not be NULL when we return to the outer level, because we already refilled
it while recovering from the inner flush. This leads to a fairly small
session-lifespan leak in CacheMemoryContext. In real-world usage the leak
would be too small to notice; but in testing with CLOBBER_CACHE_RECURSIVELY
the leakage can add up to the point of causing OOM failures, as reported by
Tomas Vondra.
The issue has been there a long time, but it only seems worth fixing in
HEAD, like the previous fix in this area (commit 078b2ed291c758e7).
2014-08-13 17:27:28 +02:00
|
|
|
|
2014-05-14 20:55:48 +02:00
|
|
|
/*
|
|
|
|
* Now save copies of the bitmaps in the relcache entry. We intentionally
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
* set rd_attrsvalid last, because that's the one that signals validity of
|
2022-06-16 15:02:48 +02:00
|
|
|
* the values; if we run out of memory before making that copy, we won't
|
|
|
|
* leave the relcache entry looking like the other ones are valid but
|
|
|
|
* empty.
|
2014-05-14 20:55:48 +02:00
|
|
|
*/
|
2007-09-20 19:56:33 +02:00
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
relation->rd_keyattr = bms_copy(uindexattrs);
|
2017-01-19 18:00:00 +01:00
|
|
|
relation->rd_pkattr = bms_copy(pkindexattrs);
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
relation->rd_idattr = bms_copy(idindexattrs);
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
relation->rd_hotblockingattr = bms_copy(hotblockingattrs);
|
|
|
|
relation->rd_summarizedattr = bms_copy(summarizedattrs);
|
|
|
|
relation->rd_attrsvalid = true;
|
2007-09-20 19:56:33 +02:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
/* We return our original working copy for caller to play with */
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
switch (attrKind)
|
|
|
|
{
|
2014-05-14 20:55:48 +02:00
|
|
|
case INDEX_ATTR_BITMAP_KEY:
|
|
|
|
return uindexattrs;
|
2017-01-19 18:00:00 +01:00
|
|
|
case INDEX_ATTR_BITMAP_PRIMARY_KEY:
|
2019-01-22 00:33:32 +01:00
|
|
|
return pkindexattrs;
|
2014-05-14 20:55:48 +02:00
|
|
|
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
|
|
|
|
return idindexattrs;
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
case INDEX_ATTR_BITMAP_HOT_BLOCKING:
|
|
|
|
return hotblockingattrs;
|
|
|
|
case INDEX_ATTR_BITMAP_SUMMARIZED:
|
|
|
|
return summarizedattrs;
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
default:
|
|
|
|
elog(ERROR, "unknown attrKind %u", attrKind);
|
2014-01-07 20:47:54 +01:00
|
|
|
return NULL;
|
Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65. This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all. Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-11 00:33:45 +01:00
|
|
|
}
|
2007-09-20 19:56:33 +02:00
|
|
|
}
|
|
|
|
|
2021-04-27 04:58:26 +02:00
|
|
|
/*
|
|
|
|
* RelationGetIdentityKeyBitmap -- get a bitmap of replica identity attribute
|
|
|
|
* numbers
|
|
|
|
*
|
|
|
|
* A bitmap of index attribute numbers for the configured replica identity
|
|
|
|
* index is returned.
|
|
|
|
*
|
|
|
|
* See also comments of RelationGetIndexAttrBitmap().
|
|
|
|
*
|
|
|
|
* This is a special purpose function used during logical replication. Here,
|
|
|
|
* unlike RelationGetIndexAttrBitmap(), we don't acquire a lock on the required
|
|
|
|
* index as we build the cache entry using a historic snapshot and all the
|
|
|
|
* later changes are absorbed while decoding WAL. Due to this reason, we don't
|
|
|
|
* need to retry here in case of a change in the set of indexes.
|
|
|
|
*/
|
|
|
|
Bitmapset *
|
|
|
|
RelationGetIdentityKeyBitmap(Relation relation)
|
|
|
|
{
|
|
|
|
Bitmapset *idindexattrs = NULL; /* columns in the replica identity */
|
|
|
|
Relation indexDesc;
|
|
|
|
int i;
|
2021-06-28 07:26:53 +02:00
|
|
|
Oid replidindex;
|
2021-04-27 04:58:26 +02:00
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
/* Quick exit if we already computed the result */
|
|
|
|
if (relation->rd_idattr != NULL)
|
|
|
|
return bms_copy(relation->rd_idattr);
|
|
|
|
|
|
|
|
/* Fast path if definitely no indexes */
|
|
|
|
if (!RelationGetForm(relation)->relhasindex)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
/* Historic snapshot must be set. */
|
|
|
|
Assert(HistoricSnapshotActive());
|
|
|
|
|
2021-06-28 07:26:53 +02:00
|
|
|
replidindex = RelationGetReplicaIndex(relation);
|
2021-04-27 04:58:26 +02:00
|
|
|
|
2021-06-19 08:00:33 +02:00
|
|
|
/* Fall out if there is no replica identity index */
|
2021-06-28 07:26:53 +02:00
|
|
|
if (!OidIsValid(replidindex))
|
2021-06-19 08:00:33 +02:00
|
|
|
return NULL;
|
|
|
|
|
|
|
|
/* Look up the description for the replica identity index */
|
2021-06-28 07:26:53 +02:00
|
|
|
indexDesc = RelationIdGetRelation(replidindex);
|
2021-06-19 08:00:33 +02:00
|
|
|
|
|
|
|
if (!RelationIsValid(indexDesc))
|
|
|
|
elog(ERROR, "could not open relation with OID %u",
|
|
|
|
relation->rd_replidindex);
|
|
|
|
|
|
|
|
/* Add referenced attributes to idindexattrs */
|
2021-04-27 04:58:26 +02:00
|
|
|
for (i = 0; i < indexDesc->rd_index->indnatts; i++)
|
|
|
|
{
|
|
|
|
int attrnum = indexDesc->rd_index->indkey.values[i];
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't include non-key columns into idindexattrs bitmaps. See
|
|
|
|
* RelationGetIndexAttrBitmap.
|
|
|
|
*/
|
|
|
|
if (attrnum != 0)
|
|
|
|
{
|
|
|
|
if (i < indexDesc->rd_index->indnkeyatts)
|
|
|
|
idindexattrs = bms_add_member(idindexattrs,
|
|
|
|
attrnum - FirstLowInvalidHeapAttributeNumber);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
RelationClose(indexDesc);
|
|
|
|
|
|
|
|
/* Don't leak the old values of these bitmaps, if any */
|
|
|
|
bms_free(relation->rd_idattr);
|
|
|
|
relation->rd_idattr = NULL;
|
|
|
|
|
|
|
|
/* Now save copy of the bitmap in the relcache entry */
|
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
|
|
|
relation->rd_idattr = bms_copy(idindexattrs);
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
/* We return our original working copy for caller to play with */
|
|
|
|
return idindexattrs;
|
|
|
|
}
|
|
|
|
|
2009-12-07 06:22:23 +01:00
|
|
|
/*
|
|
|
|
* RelationGetExclusionInfo -- get info about index's exclusion constraint
|
|
|
|
*
|
|
|
|
* This should be called only for an index that is known to have an
|
|
|
|
* associated exclusion constraint. It returns arrays (palloc'd in caller's
|
|
|
|
* context) of the exclusion operator OIDs, their underlying functions'
|
|
|
|
* OIDs, and their strategy numbers in the index's opclasses. We cache
|
|
|
|
* all this information since it requires a fair amount of work to get.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
RelationGetExclusionInfo(Relation indexRelation,
|
|
|
|
Oid **operators,
|
|
|
|
Oid **procs,
|
|
|
|
uint16 **strategies)
|
|
|
|
{
|
2018-04-07 22:00:39 +02:00
|
|
|
int indnkeyatts;
|
2009-12-07 06:22:23 +01:00
|
|
|
Oid *ops;
|
|
|
|
Oid *funcs;
|
|
|
|
uint16 *strats;
|
|
|
|
Relation conrel;
|
|
|
|
SysScanDesc conscan;
|
|
|
|
ScanKeyData skey[1];
|
|
|
|
HeapTuple htup;
|
|
|
|
bool found;
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
int i;
|
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRelation);
|
|
|
|
|
2009-12-07 06:22:23 +01:00
|
|
|
/* Allocate result space in caller context */
|
2018-04-07 22:00:39 +02:00
|
|
|
*operators = ops = (Oid *) palloc(sizeof(Oid) * indnkeyatts);
|
|
|
|
*procs = funcs = (Oid *) palloc(sizeof(Oid) * indnkeyatts);
|
|
|
|
*strategies = strats = (uint16 *) palloc(sizeof(uint16) * indnkeyatts);
|
2009-12-07 06:22:23 +01:00
|
|
|
|
|
|
|
/* Quick exit if we have the data cached already */
|
|
|
|
if (indexRelation->rd_exclstrats != NULL)
|
|
|
|
{
|
2018-04-07 22:00:39 +02:00
|
|
|
memcpy(ops, indexRelation->rd_exclops, sizeof(Oid) * indnkeyatts);
|
|
|
|
memcpy(funcs, indexRelation->rd_exclprocs, sizeof(Oid) * indnkeyatts);
|
|
|
|
memcpy(strats, indexRelation->rd_exclstrats, sizeof(uint16) * indnkeyatts);
|
2009-12-07 06:22:23 +01:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Search pg_constraint for the constraint associated with the index. To
|
|
|
|
* make this not too painfully slow, we use the index on conrelid; that
|
|
|
|
* will hold the parent relation's OID not the index's own OID.
|
Fully enforce uniqueness of constraint names.
It's been true for a long time that we expect names of table and domain
constraints to be unique among the constraints of that table or domain.
However, the enforcement of that has been pretty haphazard, and it missed
some corner cases such as creating a CHECK constraint and then an index
constraint of the same name (as per recent report from André Hänsel).
Also, due to the lack of an actual unique index enforcing this, duplicates
could be created through race conditions.
Moreover, the code that searches pg_constraint has been quite inconsistent
about how to handle duplicate names if one did occur: some places checked
and threw errors if there was more than one match, while others just
processed the first match they came to.
To fix, create a unique index on (conrelid, contypid, conname). Since
either conrelid or contypid is zero, this will separately enforce
uniqueness of constraint names among constraints of any one table and any
one domain. (If we ever implement SQL assertions, and put them into this
catalog, more thought might be needed. But it'd be at least as reasonable
to put them into a new catalog; having overloaded this one catalog with
two kinds of constraints was a mistake already IMO.) This index can replace
the existing non-unique index on conrelid, though we need to keep the one
on contypid for query performance reasons.
Having done that, we can simplify the logic in various places that either
coped with duplicates or neglected to, as well as potentially improve
lookup performance when searching for a constraint by name.
Also, as per our usual practice, install a preliminary check so that you
get something more friendly than a unique-index violation report in the
case complained of by André. And teach ChooseIndexName to avoid choosing
autogenerated names that would draw such a failure.
While it's not possible to make such a change in the back branches,
it doesn't seem quite too late to put this into v11, so do so.
Discussion: https://postgr.es/m/0c1001d4428f$0942b430$1bc81c90$@webkr.de
2018-09-04 19:45:35 +02:00
|
|
|
*
|
|
|
|
* Note: if we wanted to rely on the constraint name matching the index's
|
|
|
|
* name, we could just do a direct lookup using pg_constraint's unique
|
|
|
|
* index. For the moment it doesn't seem worth requiring that.
|
2009-12-07 06:22:23 +01:00
|
|
|
*/
|
|
|
|
ScanKeyInit(&skey[0],
|
|
|
|
Anum_pg_constraint_conrelid,
|
|
|
|
BTEqualStrategyNumber, F_OIDEQ,
|
|
|
|
ObjectIdGetDatum(indexRelation->rd_index->indrelid));
|
|
|
|
|
2019-01-21 19:32:19 +01:00
|
|
|
conrel = table_open(ConstraintRelationId, AccessShareLock);
|
Fully enforce uniqueness of constraint names.
It's been true for a long time that we expect names of table and domain
constraints to be unique among the constraints of that table or domain.
However, the enforcement of that has been pretty haphazard, and it missed
some corner cases such as creating a CHECK constraint and then an index
constraint of the same name (as per recent report from André Hänsel).
Also, due to the lack of an actual unique index enforcing this, duplicates
could be created through race conditions.
Moreover, the code that searches pg_constraint has been quite inconsistent
about how to handle duplicate names if one did occur: some places checked
and threw errors if there was more than one match, while others just
processed the first match they came to.
To fix, create a unique index on (conrelid, contypid, conname). Since
either conrelid or contypid is zero, this will separately enforce
uniqueness of constraint names among constraints of any one table and any
one domain. (If we ever implement SQL assertions, and put them into this
catalog, more thought might be needed. But it'd be at least as reasonable
to put them into a new catalog; having overloaded this one catalog with
two kinds of constraints was a mistake already IMO.) This index can replace
the existing non-unique index on conrelid, though we need to keep the one
on contypid for query performance reasons.
Having done that, we can simplify the logic in various places that either
coped with duplicates or neglected to, as well as potentially improve
lookup performance when searching for a constraint by name.
Also, as per our usual practice, install a preliminary check so that you
get something more friendly than a unique-index violation report in the
case complained of by André. And teach ChooseIndexName to avoid choosing
autogenerated names that would draw such a failure.
While it's not possible to make such a change in the back branches,
it doesn't seem quite too late to put this into v11, so do so.
Discussion: https://postgr.es/m/0c1001d4428f$0942b430$1bc81c90$@webkr.de
2018-09-04 19:45:35 +02:00
|
|
|
conscan = systable_beginscan(conrel, ConstraintRelidTypidNameIndexId, true,
|
Use an MVCC snapshot, rather than SnapshotNow, for catalog scans.
SnapshotNow scans have the undesirable property that, in the face of
concurrent updates, the scan can fail to see either the old or the new
versions of the row. In many cases, we work around this by requiring
DDL operations to hold AccessExclusiveLock on the object being
modified; in some cases, the existing locking is inadequate and random
failures occur as a result. This commit doesn't change anything
related to locking, but will hopefully pave the way to allowing lock
strength reductions in the future.
The major issue has held us back from making this change in the past
is that taking an MVCC snapshot is significantly more expensive than
using a static special snapshot such as SnapshotNow. However, testing
of various worst-case scenarios reveals that this problem is not
severe except under fairly extreme workloads. To mitigate those
problems, we avoid retaking the MVCC snapshot for each new scan;
instead, we take a new snapshot only when invalidation messages have
been processed. The catcache machinery already requires that
invalidation messages be sent before releasing the related heavyweight
lock; else other backends might rely on locally-cached data rather
than scanning the catalog at all. Thus, making snapshot reuse
dependent on the same guarantees shouldn't break anything that wasn't
already subtly broken.
Patch by me. Review by Michael Paquier and Andres Freund.
2013-07-02 15:47:01 +02:00
|
|
|
NULL, 1, skey);
|
2009-12-07 06:22:23 +01:00
|
|
|
found = false;
|
|
|
|
|
|
|
|
while (HeapTupleIsValid(htup = systable_getnext(conscan)))
|
|
|
|
{
|
|
|
|
Form_pg_constraint conform = (Form_pg_constraint) GETSTRUCT(htup);
|
|
|
|
Datum val;
|
|
|
|
bool isnull;
|
|
|
|
ArrayType *arr;
|
|
|
|
int nelem;
|
|
|
|
|
|
|
|
/* We want the exclusion constraint owning the index */
|
|
|
|
if (conform->contype != CONSTRAINT_EXCLUSION ||
|
|
|
|
conform->conindid != RelationGetRelid(indexRelation))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* There should be only one */
|
|
|
|
if (found)
|
|
|
|
elog(ERROR, "unexpected exclusion constraint record found for rel %s",
|
|
|
|
RelationGetRelationName(indexRelation));
|
|
|
|
found = true;
|
|
|
|
|
|
|
|
/* Extract the operator OIDS from conexclop */
|
|
|
|
val = fastgetattr(htup,
|
|
|
|
Anum_pg_constraint_conexclop,
|
|
|
|
conrel->rd_att, &isnull);
|
|
|
|
if (isnull)
|
|
|
|
elog(ERROR, "null conexclop for rel %s",
|
|
|
|
RelationGetRelationName(indexRelation));
|
|
|
|
|
|
|
|
arr = DatumGetArrayTypeP(val); /* ensure not toasted */
|
|
|
|
nelem = ARR_DIMS(arr)[0];
|
|
|
|
if (ARR_NDIM(arr) != 1 ||
|
2018-04-07 22:00:39 +02:00
|
|
|
nelem != indnkeyatts ||
|
2009-12-07 06:22:23 +01:00
|
|
|
ARR_HASNULL(arr) ||
|
|
|
|
ARR_ELEMTYPE(arr) != OIDOID)
|
|
|
|
elog(ERROR, "conexclop is not a 1-D Oid array");
|
|
|
|
|
2018-04-07 22:00:39 +02:00
|
|
|
memcpy(ops, ARR_DATA_PTR(arr), sizeof(Oid) * indnkeyatts);
|
2009-12-07 06:22:23 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
systable_endscan(conscan);
|
2019-01-21 19:32:19 +01:00
|
|
|
table_close(conrel, AccessShareLock);
|
2009-12-07 06:22:23 +01:00
|
|
|
|
|
|
|
if (!found)
|
|
|
|
elog(ERROR, "exclusion constraint record missing for rel %s",
|
|
|
|
RelationGetRelationName(indexRelation));
|
|
|
|
|
|
|
|
/* We need the func OIDs and strategy numbers too */
|
2018-04-07 22:00:39 +02:00
|
|
|
for (i = 0; i < indnkeyatts; i++)
|
2009-12-07 06:22:23 +01:00
|
|
|
{
|
|
|
|
funcs[i] = get_opcode(ops[i]);
|
|
|
|
strats[i] = get_op_opfamily_strategy(ops[i],
|
|
|
|
indexRelation->rd_opfamily[i]);
|
|
|
|
/* shouldn't fail, since it was checked at index creation */
|
|
|
|
if (strats[i] == InvalidStrategy)
|
|
|
|
elog(ERROR, "could not find strategy for operator %u in family %u",
|
|
|
|
ops[i], indexRelation->rd_opfamily[i]);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Save a copy of the results in the relcache entry. */
|
|
|
|
oldcxt = MemoryContextSwitchTo(indexRelation->rd_indexcxt);
|
2018-04-07 22:00:39 +02:00
|
|
|
indexRelation->rd_exclops = (Oid *) palloc(sizeof(Oid) * indnkeyatts);
|
|
|
|
indexRelation->rd_exclprocs = (Oid *) palloc(sizeof(Oid) * indnkeyatts);
|
|
|
|
indexRelation->rd_exclstrats = (uint16 *) palloc(sizeof(uint16) * indnkeyatts);
|
|
|
|
memcpy(indexRelation->rd_exclops, ops, sizeof(Oid) * indnkeyatts);
|
|
|
|
memcpy(indexRelation->rd_exclprocs, funcs, sizeof(Oid) * indnkeyatts);
|
|
|
|
memcpy(indexRelation->rd_exclstrats, strats, sizeof(uint16) * indnkeyatts);
|
2009-12-07 06:22:23 +01:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
|
|
|
|
2017-01-19 18:00:00 +01:00
|
|
|
/*
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
* Get the publication information for the given relation.
|
|
|
|
*
|
|
|
|
* Traverse all the publications which the relation is in to get the
|
|
|
|
* publication actions and validate the row filter expressions for such
|
|
|
|
* publications if any. We consider the row filter expression as invalid if it
|
|
|
|
* references any column which is not part of REPLICA IDENTITY.
|
|
|
|
*
|
|
|
|
* To avoid fetching the publication information repeatedly, we cache the
|
|
|
|
* publication actions and row filter validation information.
|
2017-01-19 18:00:00 +01:00
|
|
|
*/
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
void
|
|
|
|
RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
|
2017-01-19 18:00:00 +01:00
|
|
|
{
|
|
|
|
List *puboids;
|
|
|
|
ListCell *lc;
|
|
|
|
MemoryContext oldcxt;
|
Allow publishing the tables of schema.
A new option "FOR ALL TABLES IN SCHEMA" in Create/Alter Publication allows
one or more schemas to be specified, whose tables are selected by the
publisher for sending the data to the subscriber.
The new syntax allows specifying both the tables and schemas. For example:
CREATE PUBLICATION pub1 FOR TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
OR
ALTER PUBLICATION pub1 ADD TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
A new system table "pg_publication_namespace" has been added, to maintain
the schemas that the user wants to publish through the publication.
Modified the output plugin (pgoutput) to publish the changes if the
relation is part of schema publication.
Updates pg_dump to identify and dump schema publications. Updates the \d
family of commands to display schema publications and \dRp+ variant will
now display associated schemas if any.
Author: Vignesh C, Hou Zhijie, Amit Kapila
Syntax-Suggested-by: Tom Lane, Alvaro Herrera
Reviewed-by: Greg Nancarrow, Masahiko Sawada, Hou Zhijie, Amit Kapila, Haiying Tang, Ajin Cherian, Rahila Syed, Bharath Rupireddy, Mark Dilger
Tested-by: Haiying Tang
Discussion: https://www.postgresql.org/message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ@mail.gmail.com
2021-10-27 04:14:52 +02:00
|
|
|
Oid schemaid;
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
List *ancestors = NIL;
|
|
|
|
Oid relid = RelationGetRelid(relation);
|
2017-01-19 18:00:00 +01:00
|
|
|
|
2019-04-16 10:37:44 +02:00
|
|
|
/*
|
|
|
|
* If not publishable, it publishes no actions. (pgoutput_change() will
|
|
|
|
* ignore it.)
|
|
|
|
*/
|
|
|
|
if (!is_publishable_relation(relation))
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
{
|
|
|
|
memset(pubdesc, 0, sizeof(PublicationDesc));
|
|
|
|
pubdesc->rf_valid_for_update = true;
|
|
|
|
pubdesc->rf_valid_for_delete = true;
|
Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. The column list may be specified after the table
name, enclosed in parentheses. Columns not included in this list are not
sent to the subscriber, allowing the schema on the subscriber to be a
subset of the publisher schema.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns included in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied.
This means all columns are replicated if the table has no column list
at all (which is treated as column list with all columns), or when of
the publications is defined as FOR ALL TABLES (possibly IN SCHEMA that
matches the schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or the leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Alvaro Herrera, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
2022-03-26 00:45:21 +01:00
|
|
|
pubdesc->cols_valid_for_update = true;
|
|
|
|
pubdesc->cols_valid_for_delete = true;
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (relation->rd_pubdesc)
|
|
|
|
{
|
|
|
|
memcpy(pubdesc, relation->rd_pubdesc, sizeof(PublicationDesc));
|
|
|
|
return;
|
|
|
|
}
|
2019-04-16 10:37:44 +02:00
|
|
|
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
memset(pubdesc, 0, sizeof(PublicationDesc));
|
|
|
|
pubdesc->rf_valid_for_update = true;
|
|
|
|
pubdesc->rf_valid_for_delete = true;
|
Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. The column list may be specified after the table
name, enclosed in parentheses. Columns not included in this list are not
sent to the subscriber, allowing the schema on the subscriber to be a
subset of the publisher schema.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns included in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied.
This means all columns are replicated if the table has no column list
at all (which is treated as column list with all columns), or when of
the publications is defined as FOR ALL TABLES (possibly IN SCHEMA that
matches the schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or the leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Alvaro Herrera, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
2022-03-26 00:45:21 +01:00
|
|
|
pubdesc->cols_valid_for_update = true;
|
|
|
|
pubdesc->cols_valid_for_delete = true;
|
2017-01-19 18:00:00 +01:00
|
|
|
|
|
|
|
/* Fetch the publication membership info. */
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
puboids = GetRelationPublications(relid);
|
Allow publishing the tables of schema.
A new option "FOR ALL TABLES IN SCHEMA" in Create/Alter Publication allows
one or more schemas to be specified, whose tables are selected by the
publisher for sending the data to the subscriber.
The new syntax allows specifying both the tables and schemas. For example:
CREATE PUBLICATION pub1 FOR TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
OR
ALTER PUBLICATION pub1 ADD TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
A new system table "pg_publication_namespace" has been added, to maintain
the schemas that the user wants to publish through the publication.
Modified the output plugin (pgoutput) to publish the changes if the
relation is part of schema publication.
Updates pg_dump to identify and dump schema publications. Updates the \d
family of commands to display schema publications and \dRp+ variant will
now display associated schemas if any.
Author: Vignesh C, Hou Zhijie, Amit Kapila
Syntax-Suggested-by: Tom Lane, Alvaro Herrera
Reviewed-by: Greg Nancarrow, Masahiko Sawada, Hou Zhijie, Amit Kapila, Haiying Tang, Ajin Cherian, Rahila Syed, Bharath Rupireddy, Mark Dilger
Tested-by: Haiying Tang
Discussion: https://www.postgresql.org/message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ@mail.gmail.com
2021-10-27 04:14:52 +02:00
|
|
|
schemaid = RelationGetNamespace(relation);
|
2022-04-07 18:13:13 +02:00
|
|
|
puboids = list_concat_unique_oid(puboids, GetSchemaPublications(schemaid));
|
Allow publishing the tables of schema.
A new option "FOR ALL TABLES IN SCHEMA" in Create/Alter Publication allows
one or more schemas to be specified, whose tables are selected by the
publisher for sending the data to the subscriber.
The new syntax allows specifying both the tables and schemas. For example:
CREATE PUBLICATION pub1 FOR TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
OR
ALTER PUBLICATION pub1 ADD TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
A new system table "pg_publication_namespace" has been added, to maintain
the schemas that the user wants to publish through the publication.
Modified the output plugin (pgoutput) to publish the changes if the
relation is part of schema publication.
Updates pg_dump to identify and dump schema publications. Updates the \d
family of commands to display schema publications and \dRp+ variant will
now display associated schemas if any.
Author: Vignesh C, Hou Zhijie, Amit Kapila
Syntax-Suggested-by: Tom Lane, Alvaro Herrera
Reviewed-by: Greg Nancarrow, Masahiko Sawada, Hou Zhijie, Amit Kapila, Haiying Tang, Ajin Cherian, Rahila Syed, Bharath Rupireddy, Mark Dilger
Tested-by: Haiying Tang
Discussion: https://www.postgresql.org/message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ@mail.gmail.com
2021-10-27 04:14:52 +02:00
|
|
|
|
2020-04-08 09:59:27 +02:00
|
|
|
if (relation->rd_rel->relispartition)
|
|
|
|
{
|
|
|
|
/* Add publications that the ancestors are in too. */
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
ancestors = get_partition_ancestors(relid);
|
2020-04-08 09:59:27 +02:00
|
|
|
|
|
|
|
foreach(lc, ancestors)
|
|
|
|
{
|
|
|
|
Oid ancestor = lfirst_oid(lc);
|
|
|
|
|
|
|
|
puboids = list_concat_unique_oid(puboids,
|
|
|
|
GetRelationPublications(ancestor));
|
Allow publishing the tables of schema.
A new option "FOR ALL TABLES IN SCHEMA" in Create/Alter Publication allows
one or more schemas to be specified, whose tables are selected by the
publisher for sending the data to the subscriber.
The new syntax allows specifying both the tables and schemas. For example:
CREATE PUBLICATION pub1 FOR TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
OR
ALTER PUBLICATION pub1 ADD TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2;
A new system table "pg_publication_namespace" has been added, to maintain
the schemas that the user wants to publish through the publication.
Modified the output plugin (pgoutput) to publish the changes if the
relation is part of schema publication.
Updates pg_dump to identify and dump schema publications. Updates the \d
family of commands to display schema publications and \dRp+ variant will
now display associated schemas if any.
Author: Vignesh C, Hou Zhijie, Amit Kapila
Syntax-Suggested-by: Tom Lane, Alvaro Herrera
Reviewed-by: Greg Nancarrow, Masahiko Sawada, Hou Zhijie, Amit Kapila, Haiying Tang, Ajin Cherian, Rahila Syed, Bharath Rupireddy, Mark Dilger
Tested-by: Haiying Tang
Discussion: https://www.postgresql.org/message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ@mail.gmail.com
2021-10-27 04:14:52 +02:00
|
|
|
schemaid = get_rel_namespace(ancestor);
|
|
|
|
puboids = list_concat_unique_oid(puboids,
|
2022-04-07 18:13:13 +02:00
|
|
|
GetSchemaPublications(schemaid));
|
2020-04-08 09:59:27 +02:00
|
|
|
}
|
|
|
|
}
|
2022-04-07 18:13:13 +02:00
|
|
|
puboids = list_concat_unique_oid(puboids, GetAllTablesPublications());
|
2017-01-19 18:00:00 +01:00
|
|
|
|
|
|
|
foreach(lc, puboids)
|
|
|
|
{
|
|
|
|
Oid pubid = lfirst_oid(lc);
|
|
|
|
HeapTuple tup;
|
|
|
|
Form_pg_publication pubform;
|
|
|
|
|
|
|
|
tup = SearchSysCache1(PUBLICATIONOID, ObjectIdGetDatum(pubid));
|
|
|
|
|
|
|
|
if (!HeapTupleIsValid(tup))
|
|
|
|
elog(ERROR, "cache lookup failed for publication %u", pubid);
|
|
|
|
|
|
|
|
pubform = (Form_pg_publication) GETSTRUCT(tup);
|
|
|
|
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
pubdesc->pubactions.pubinsert |= pubform->pubinsert;
|
|
|
|
pubdesc->pubactions.pubupdate |= pubform->pubupdate;
|
|
|
|
pubdesc->pubactions.pubdelete |= pubform->pubdelete;
|
|
|
|
pubdesc->pubactions.pubtruncate |= pubform->pubtruncate;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if all columns referenced in the filter expression are part
|
|
|
|
* of the REPLICA IDENTITY index or not.
|
|
|
|
*
|
|
|
|
* If the publication is FOR ALL TABLES then it means the table has no
|
|
|
|
* row filters and we can skip the validation.
|
|
|
|
*/
|
|
|
|
if (!pubform->puballtables &&
|
|
|
|
(pubform->pubupdate || pubform->pubdelete) &&
|
Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. The column list may be specified after the table
name, enclosed in parentheses. Columns not included in this list are not
sent to the subscriber, allowing the schema on the subscriber to be a
subset of the publisher schema.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns included in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied.
This means all columns are replicated if the table has no column list
at all (which is treated as column list with all columns), or when of
the publications is defined as FOR ALL TABLES (possibly IN SCHEMA that
matches the schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or the leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Alvaro Herrera, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
2022-03-26 00:45:21 +01:00
|
|
|
pub_rf_contains_invalid_column(pubid, relation, ancestors,
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
pubform->pubviaroot))
|
|
|
|
{
|
|
|
|
if (pubform->pubupdate)
|
|
|
|
pubdesc->rf_valid_for_update = false;
|
|
|
|
if (pubform->pubdelete)
|
|
|
|
pubdesc->rf_valid_for_delete = false;
|
|
|
|
}
|
2017-01-19 18:00:00 +01:00
|
|
|
|
Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. The column list may be specified after the table
name, enclosed in parentheses. Columns not included in this list are not
sent to the subscriber, allowing the schema on the subscriber to be a
subset of the publisher schema.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns included in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied.
This means all columns are replicated if the table has no column list
at all (which is treated as column list with all columns), or when of
the publications is defined as FOR ALL TABLES (possibly IN SCHEMA that
matches the schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or the leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Alvaro Herrera, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
2022-03-26 00:45:21 +01:00
|
|
|
/*
|
|
|
|
* Check if all columns are part of the REPLICA IDENTITY index or not.
|
|
|
|
*
|
|
|
|
* If the publication is FOR ALL TABLES then it means the table has no
|
|
|
|
* column list and we can skip the validation.
|
|
|
|
*/
|
|
|
|
if (!pubform->puballtables &&
|
|
|
|
(pubform->pubupdate || pubform->pubdelete) &&
|
|
|
|
pub_collist_contains_invalid_column(pubid, relation, ancestors,
|
|
|
|
pubform->pubviaroot))
|
|
|
|
{
|
|
|
|
if (pubform->pubupdate)
|
|
|
|
pubdesc->cols_valid_for_update = false;
|
|
|
|
if (pubform->pubdelete)
|
|
|
|
pubdesc->cols_valid_for_delete = false;
|
|
|
|
}
|
|
|
|
|
2017-01-19 18:00:00 +01:00
|
|
|
ReleaseSysCache(tup);
|
|
|
|
|
|
|
|
/*
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
* If we know everything is replicated and the row filter is invalid
|
|
|
|
* for update and delete, there is no point to check for other
|
|
|
|
* publications.
|
2017-01-19 18:00:00 +01:00
|
|
|
*/
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
|
|
|
|
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
|
|
|
|
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
|
2017-01-19 18:00:00 +01:00
|
|
|
break;
|
Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. The column list may be specified after the table
name, enclosed in parentheses. Columns not included in this list are not
sent to the subscriber, allowing the schema on the subscriber to be a
subset of the publisher schema.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns included in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied.
This means all columns are replicated if the table has no column list
at all (which is treated as column list with all columns), or when of
the publications is defined as FOR ALL TABLES (possibly IN SCHEMA that
matches the schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or the leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Alvaro Herrera, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
2022-03-26 00:45:21 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we know everything is replicated and the column list is invalid
|
|
|
|
* for update and delete, there is no point to check for other
|
|
|
|
* publications.
|
|
|
|
*/
|
|
|
|
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
|
|
|
|
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
|
|
|
|
!pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
|
|
|
|
break;
|
2017-01-19 18:00:00 +01:00
|
|
|
}
|
|
|
|
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
if (relation->rd_pubdesc)
|
2017-01-19 18:00:00 +01:00
|
|
|
{
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
pfree(relation->rd_pubdesc);
|
|
|
|
relation->rd_pubdesc = NULL;
|
2017-01-19 18:00:00 +01:00
|
|
|
}
|
|
|
|
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
/* Now save copy of the descriptor in the relcache entry. */
|
2017-01-19 18:00:00 +01:00
|
|
|
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
relation->rd_pubdesc = palloc(sizeof(PublicationDesc));
|
|
|
|
memcpy(relation->rd_pubdesc, pubdesc, sizeof(PublicationDesc));
|
2017-01-19 18:00:00 +01:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
2000-06-20 01:40:48 +02:00
|
|
|
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
static bytea **
|
|
|
|
CopyIndexAttOptions(bytea **srcopts, int natts)
|
|
|
|
{
|
|
|
|
bytea **opts = palloc(sizeof(*opts) * natts);
|
|
|
|
|
|
|
|
for (int i = 0; i < natts; i++)
|
|
|
|
{
|
|
|
|
bytea *opt = srcopts[i];
|
|
|
|
|
|
|
|
opts[i] = !opt ? NULL : (bytea *)
|
|
|
|
DatumGetPointer(datumCopy(PointerGetDatum(opt), false, -1));
|
|
|
|
}
|
|
|
|
|
|
|
|
return opts;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RelationGetIndexAttOptions
|
|
|
|
* get AM/opclass-specific options for an index parsed into a binary form
|
|
|
|
*/
|
|
|
|
bytea **
|
|
|
|
RelationGetIndexAttOptions(Relation relation, bool copy)
|
|
|
|
{
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
bytea **opts = relation->rd_opcoptions;
|
|
|
|
Oid relid = RelationGetRelid(relation);
|
|
|
|
int natts = RelationGetNumberOfAttributes(relation); /* XXX
|
|
|
|
* IndexRelationGetNumberOfKeyAttributes */
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Try to copy cached options. */
|
|
|
|
if (opts)
|
|
|
|
return copy ? CopyIndexAttOptions(opts, natts) : opts;
|
|
|
|
|
|
|
|
/* Get and parse opclass options. */
|
|
|
|
opts = palloc0(sizeof(*opts) * natts);
|
|
|
|
|
|
|
|
for (i = 0; i < natts; i++)
|
|
|
|
{
|
|
|
|
if (criticalRelcachesBuilt && relid != AttributeRelidNumIndexId)
|
|
|
|
{
|
|
|
|
Datum attoptions = get_attoptions(relid, i + 1);
|
|
|
|
|
|
|
|
opts[i] = index_opclass_options(relation, i + 1, attoptions, false);
|
|
|
|
|
|
|
|
if (attoptions != (Datum) 0)
|
|
|
|
pfree(DatumGetPointer(attoptions));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Copy parsed options to the cache. */
|
|
|
|
oldcxt = MemoryContextSwitchTo(relation->rd_indexcxt);
|
|
|
|
relation->rd_opcoptions = CopyIndexAttOptions(opts, natts);
|
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
|
|
|
|
if (copy)
|
|
|
|
return opts;
|
|
|
|
|
|
|
|
for (i = 0; i < natts; i++)
|
|
|
|
{
|
|
|
|
if (opts[i])
|
|
|
|
pfree(opts[i]);
|
|
|
|
}
|
|
|
|
|
|
|
|
pfree(opts);
|
|
|
|
|
|
|
|
return relation->rd_opcoptions;
|
|
|
|
}
|
|
|
|
|
Provide database object names as separate fields in error messages.
This patch addresses the problem that applications currently have to
extract object names from possibly-localized textual error messages,
if they want to know for example which index caused a UNIQUE_VIOLATION
failure. It adds new error message fields to the wire protocol, which
can carry the name of a table, table column, data type, or constraint
associated with the error. (Since the protocol spec has always instructed
clients to ignore unrecognized field types, this should not create any
compatibility problem.)
Support for providing these new fields has been added to just a limited set
of error reports (mainly, those in the "integrity constraint violation"
SQLSTATE class), but we will doubtless add them to more calls in future.
Pavel Stehule, reviewed and extensively revised by Peter Geoghegan, with
additional hacking by Tom Lane.
2013-01-29 23:06:26 +01:00
|
|
|
/*
|
|
|
|
* Routines to support ereport() reports of relation-related errors
|
|
|
|
*
|
|
|
|
* These could have been put into elog.c, but it seems like a module layering
|
|
|
|
* violation to have elog.c calling relcache or syscache stuff --- and we
|
|
|
|
* definitely don't want elog.h including rel.h. So we put them here.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* errtable --- stores schema_name and table_name of a table
|
|
|
|
* within the current errordata.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
errtable(Relation rel)
|
|
|
|
{
|
|
|
|
err_generic_string(PG_DIAG_SCHEMA_NAME,
|
|
|
|
get_namespace_name(RelationGetNamespace(rel)));
|
|
|
|
err_generic_string(PG_DIAG_TABLE_NAME, RelationGetRelationName(rel));
|
|
|
|
|
|
|
|
return 0; /* return value does not matter */
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* errtablecol --- stores schema_name, table_name and column_name
|
|
|
|
* of a table column within the current errordata.
|
|
|
|
*
|
|
|
|
* The column is specified by attribute number --- for most callers, this is
|
|
|
|
* easier and less error-prone than getting the column name for themselves.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
errtablecol(Relation rel, int attnum)
|
|
|
|
{
|
|
|
|
TupleDesc reldesc = RelationGetDescr(rel);
|
|
|
|
const char *colname;
|
|
|
|
|
|
|
|
/* Use reldesc if it's a user attribute, else consult the catalogs */
|
|
|
|
if (attnum > 0 && attnum <= reldesc->natts)
|
2017-08-20 20:19:07 +02:00
|
|
|
colname = NameStr(TupleDescAttr(reldesc, attnum - 1)->attname);
|
Provide database object names as separate fields in error messages.
This patch addresses the problem that applications currently have to
extract object names from possibly-localized textual error messages,
if they want to know for example which index caused a UNIQUE_VIOLATION
failure. It adds new error message fields to the wire protocol, which
can carry the name of a table, table column, data type, or constraint
associated with the error. (Since the protocol spec has always instructed
clients to ignore unrecognized field types, this should not create any
compatibility problem.)
Support for providing these new fields has been added to just a limited set
of error reports (mainly, those in the "integrity constraint violation"
SQLSTATE class), but we will doubtless add them to more calls in future.
Pavel Stehule, reviewed and extensively revised by Peter Geoghegan, with
additional hacking by Tom Lane.
2013-01-29 23:06:26 +01:00
|
|
|
else
|
2018-02-12 23:30:30 +01:00
|
|
|
colname = get_attname(RelationGetRelid(rel), attnum, false);
|
Provide database object names as separate fields in error messages.
This patch addresses the problem that applications currently have to
extract object names from possibly-localized textual error messages,
if they want to know for example which index caused a UNIQUE_VIOLATION
failure. It adds new error message fields to the wire protocol, which
can carry the name of a table, table column, data type, or constraint
associated with the error. (Since the protocol spec has always instructed
clients to ignore unrecognized field types, this should not create any
compatibility problem.)
Support for providing these new fields has been added to just a limited set
of error reports (mainly, those in the "integrity constraint violation"
SQLSTATE class), but we will doubtless add them to more calls in future.
Pavel Stehule, reviewed and extensively revised by Peter Geoghegan, with
additional hacking by Tom Lane.
2013-01-29 23:06:26 +01:00
|
|
|
|
|
|
|
return errtablecolname(rel, colname);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* errtablecolname --- stores schema_name, table_name and column_name
|
|
|
|
* of a table column within the current errordata, where the column name is
|
|
|
|
* given directly rather than extracted from the relation's catalog data.
|
|
|
|
*
|
|
|
|
* Don't use this directly unless errtablecol() is inconvenient for some
|
|
|
|
* reason. This might possibly be needed during intermediate states in ALTER
|
|
|
|
* TABLE, for instance.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
errtablecolname(Relation rel, const char *colname)
|
|
|
|
{
|
|
|
|
errtable(rel);
|
|
|
|
err_generic_string(PG_DIAG_COLUMN_NAME, colname);
|
|
|
|
|
|
|
|
return 0; /* return value does not matter */
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* errtableconstraint --- stores schema_name, table_name and constraint_name
|
|
|
|
* of a table-related constraint within the current errordata.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
errtableconstraint(Relation rel, const char *conname)
|
|
|
|
{
|
|
|
|
errtable(rel);
|
|
|
|
err_generic_string(PG_DIAG_CONSTRAINT_NAME, conname);
|
|
|
|
|
|
|
|
return 0; /* return value does not matter */
|
|
|
|
}
|
|
|
|
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* load_relcache_init_file, write_relcache_init_file
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
* In late 1992, we started regularly having databases with more than
|
|
|
|
* a thousand classes in them. With this number of classes, it became
|
|
|
|
* critical to do indexed lookups on the system catalogs.
|
|
|
|
*
|
|
|
|
* Bootstrapping these lookups is very hard. We want to be able to
|
|
|
|
* use an index on pg_attribute, for example, but in order to do so,
|
|
|
|
* we must have read pg_attribute for the attributes in the index,
|
|
|
|
* which implies that we need to use the index.
|
|
|
|
*
|
|
|
|
* In order to get around the problem, we do the following:
|
|
|
|
*
|
|
|
|
* + When the database system is initialized (at initdb time), we
|
2002-02-19 21:11:20 +01:00
|
|
|
* don't use indexes. We do sequential scans.
|
1996-07-09 08:22:35 +02:00
|
|
|
*
|
|
|
|
* + When the backend is started up in normal mode, we load an image
|
|
|
|
* of the appropriate relation descriptors, in internal format,
|
|
|
|
* from an initialization file in the data/base/... directory.
|
|
|
|
*
|
|
|
|
* + If the initialization file isn't there, then we create the
|
1999-05-01 21:09:46 +02:00
|
|
|
* relation descriptors using sequential scans and write 'em to
|
1996-07-09 08:22:35 +02:00
|
|
|
* the initialization file for use by subsequent backends.
|
|
|
|
*
|
2010-02-17 05:19:41 +01:00
|
|
|
* As of Postgres 9.0, there is one local initialization file in each
|
2009-08-12 22:53:31 +02:00
|
|
|
* database, plus one shared initialization file for shared catalogs.
|
|
|
|
*
|
|
|
|
* We could dispense with the initialization files and just build the
|
1999-05-01 21:09:46 +02:00
|
|
|
* critical reldescs the hard way on every backend startup, but that
|
2002-02-19 21:11:20 +01:00
|
|
|
* slows down backend startup noticeably.
|
|
|
|
*
|
|
|
|
* We can in fact go further, and save more relcache entries than
|
|
|
|
* just the ones that are absolutely critical; this allows us to speed
|
|
|
|
* up backend startup by not having to build such entries the hard way.
|
|
|
|
* Presently, all the catalog and index entries that are referred to
|
2009-08-12 22:53:31 +02:00
|
|
|
* by catcaches are stored in the initialization files.
|
1999-05-01 21:09:46 +02:00
|
|
|
*
|
2002-08-04 20:12:15 +02:00
|
|
|
* The same mechanism that detects when catcache and relcache entries
|
|
|
|
* need to be invalidated (due to catalog updates) also arranges to
|
2009-08-12 22:53:31 +02:00
|
|
|
* unlink the initialization files when the contents may be out of date.
|
|
|
|
* The files will then be rebuilt during the next backend startup.
|
1996-07-09 08:22:35 +02:00
|
|
|
*/
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
2009-08-12 22:53:31 +02:00
|
|
|
* load_relcache_init_file -- attempt to load cache from the shared
|
|
|
|
* or local cache init file
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2017-08-16 06:22:32 +02:00
|
|
|
* If successful, return true and set criticalRelcachesBuilt or
|
2009-08-12 22:53:31 +02:00
|
|
|
* criticalSharedRelcachesBuilt to true.
|
2017-08-16 06:22:32 +02:00
|
|
|
* If not successful, return false.
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
|
|
|
* NOTE: we assume we are already switched into CacheMemoryContext.
|
|
|
|
*/
|
|
|
|
static bool
|
2009-08-12 22:53:31 +02:00
|
|
|
load_relcache_init_file(bool shared)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
FILE *fp;
|
|
|
|
char initfilename[MAXPGPATH];
|
|
|
|
Relation *rels;
|
|
|
|
int relno,
|
|
|
|
num_rels,
|
|
|
|
max_rels,
|
|
|
|
nailed_rels,
|
2003-11-09 22:30:38 +01:00
|
|
|
nailed_indexes,
|
|
|
|
magic;
|
1996-07-09 08:22:35 +02:00
|
|
|
int i;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
if (shared)
|
|
|
|
snprintf(initfilename, sizeof(initfilename), "global/%s",
|
|
|
|
RELCACHE_INIT_FILENAME);
|
|
|
|
else
|
|
|
|
snprintf(initfilename, sizeof(initfilename), "%s/%s",
|
|
|
|
DatabasePath, RELCACHE_INIT_FILENAME);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
fp = AllocateFile(initfilename, PG_BINARY_R);
|
|
|
|
if (fp == NULL)
|
|
|
|
return false;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Read the index relcache entries from the file. Note we will not enter
|
|
|
|
* any of them into the cache if the read fails partway through; this
|
|
|
|
* helps to guard against broken init files.
|
|
|
|
*/
|
|
|
|
max_rels = 100;
|
|
|
|
rels = (Relation *) palloc(max_rels * sizeof(Relation));
|
|
|
|
num_rels = 0;
|
|
|
|
nailed_rels = nailed_indexes = 0;
|
|
|
|
|
2003-11-09 22:30:38 +01:00
|
|
|
/* check for correct magic number (compatible version) */
|
|
|
|
if (fread(&magic, 1, sizeof(magic), fp) != sizeof(magic))
|
|
|
|
goto read_failed;
|
|
|
|
if (magic != RELCACHE_INIT_FILEMAGIC)
|
|
|
|
goto read_failed;
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
for (relno = 0;; relno++)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
Size len;
|
|
|
|
size_t nread;
|
|
|
|
Relation rel;
|
|
|
|
Form_pg_class relform;
|
2002-11-15 18:18:49 +01:00
|
|
|
bool has_not_null;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* first read the relation descriptor length */
|
2009-08-30 19:18:52 +02:00
|
|
|
nread = fread(&len, 1, sizeof(len), fp);
|
|
|
|
if (nread != sizeof(len))
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
|
|
|
if (nread == 0)
|
|
|
|
break; /* end of file */
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2001-10-07 01:21:45 +02:00
|
|
|
/* safety check for incompatible relcache layout */
|
|
|
|
if (len != sizeof(RelationData))
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
2001-10-07 01:21:45 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* allocate another relcache header */
|
|
|
|
if (num_rels >= max_rels)
|
|
|
|
{
|
|
|
|
max_rels *= 2;
|
|
|
|
rels = (Relation *) repalloc(rels, max_rels * sizeof(Relation));
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
rel = rels[num_rels++] = (Relation) palloc(len);
|
2000-10-16 16:52:28 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/* then, read the Relation structure */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(rel, 1, len, fp) != len)
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* next read the relation tuple form */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
relform = (Form_pg_class) palloc(len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(relform, 1, len, fp) != len)
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
rel->rd_rel = relform;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* initialize attribute tuple forms */
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
rel->rd_att = CreateTemplateTupleDesc(relform->relnatts);
|
2006-06-16 20:42:24 +02:00
|
|
|
rel->rd_att->tdrefcount = 1; /* mark as refcounted */
|
|
|
|
|
Don't create pg_type entries for sequences or toast tables.
Commit f7f70d5e2 left one inconsistency behind: we're still creating
pg_type entries for the composite types of sequences and toast tables,
but not arrays over those composites. But there seems precious little
reason to have named composite types for toast tables, and not much more
to have them for sequences (especially given the thought that sequences
may someday not be standalone relations at all).
So, let's close that inconsistency by removing these composite types,
rather than adding arrays for them. This buys back a little bit of
the initial pg_type bloat added by the previous patch, and could be
a significant savings in a large database with many toast tables.
Aside from a small logic rearrangement in heap_create_with_catalog,
this patch mostly needs to clean up some places that were assuming that
pg_class.reltype always has a valid value. Those are really pre-existing
bugs, given that it's documented otherwise; notably, the plpgsql changes
fix code that gives "cache lookup failed for type 0" on indexes today.
But none of these seem interesting enough to back-patch.
Also, remove the pg_dump/pg_upgrade infrastructure for propagating
a toast table's pg_type OID into the new database, since we no longer
need that.
Discussion: https://postgr.es/m/761F1389-C6A8-4C15-80CE-950C961F5341@gmail.com
2020-07-07 21:43:22 +02:00
|
|
|
rel->rd_att->tdtypeid = relform->reltype ? relform->reltype : RECORDOID;
|
|
|
|
rel->rd_att->tdtypmod = -1; /* just to be sure */
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* next read all the attribute tuple form data entries */
|
2002-11-15 18:18:49 +01:00
|
|
|
has_not_null = false;
|
1996-07-09 08:22:35 +02:00
|
|
|
for (i = 0; i < relform->relnatts; i++)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2017-08-20 20:19:07 +02:00
|
|
|
Form_pg_attribute attr = TupleDescAttr(rel->rd_att, i);
|
|
|
|
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
2009-01-22 21:16:10 +01:00
|
|
|
if (len != ATTRIBUTE_FIXED_PART_SIZE)
|
2005-03-07 05:42:17 +01:00
|
|
|
goto read_failed;
|
2017-08-20 20:19:07 +02:00
|
|
|
if (fread(attr, 1, len, fp) != len)
|
2002-01-16 18:34:42 +01:00
|
|
|
goto read_failed;
|
2002-11-15 18:18:49 +01:00
|
|
|
|
2017-08-20 20:19:07 +02:00
|
|
|
has_not_null |= attr->attnotnull;
|
2002-11-15 18:18:49 +01:00
|
|
|
}
|
|
|
|
|
2006-07-02 04:23:23 +02:00
|
|
|
/* next read the access method specific field */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2006-07-02 04:23:23 +02:00
|
|
|
goto read_failed;
|
|
|
|
if (len > 0)
|
|
|
|
{
|
|
|
|
rel->rd_options = palloc(len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(rel->rd_options, 1, len, fp) != len)
|
2006-07-02 04:23:23 +02:00
|
|
|
goto read_failed;
|
2007-02-28 00:48:10 +01:00
|
|
|
if (len != VARSIZE(rel->rd_options))
|
2006-07-04 00:45:41 +02:00
|
|
|
goto read_failed; /* sanity check */
|
2006-07-02 04:23:23 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
rel->rd_options = NULL;
|
|
|
|
}
|
|
|
|
|
2002-11-15 18:18:49 +01:00
|
|
|
/* mark not-null status */
|
|
|
|
if (has_not_null)
|
|
|
|
{
|
|
|
|
TupleConstr *constr = (TupleConstr *) palloc0(sizeof(TupleConstr));
|
|
|
|
|
|
|
|
constr->has_not_null = true;
|
|
|
|
rel->rd_att->constr = constr;
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
/*
|
|
|
|
* If it's an index, there's more to do. Note we explicitly ignore
|
|
|
|
* partitioned indexes here.
|
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
if (rel->rd_rel->relkind == RELKIND_INDEX)
|
|
|
|
{
|
|
|
|
MemoryContext indexcxt;
|
2006-12-23 01:43:13 +01:00
|
|
|
Oid *opfamily;
|
|
|
|
Oid *opcintype;
|
2002-02-19 21:11:20 +01:00
|
|
|
RegProcedure *support;
|
2003-11-09 22:30:38 +01:00
|
|
|
int nsupport;
|
2007-01-09 03:14:16 +01:00
|
|
|
int16 *indoption;
|
2011-02-08 22:04:18 +01:00
|
|
|
Oid *indcollation;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/* Count nailed indexes to ensure we have 'em all */
|
|
|
|
if (rel->rd_isnailed)
|
|
|
|
nailed_indexes++;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the pg_index tuple */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2002-02-19 21:11:20 +01:00
|
|
|
goto read_failed;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
rel->rd_indextuple = (HeapTuple) palloc(len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(rel->rd_indextuple, 1, len, fp) != len)
|
2002-02-19 21:11:20 +01:00
|
|
|
goto read_failed;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-05-28 18:04:02 +02:00
|
|
|
/* Fix up internal pointers in the tuple -- see heap_copytuple */
|
|
|
|
rel->rd_indextuple->t_data = (HeapTupleHeader) ((char *) rel->rd_indextuple + HEAPTUPLESIZE);
|
|
|
|
rel->rd_index = (Form_pg_index) GETSTRUCT(rel->rd_indextuple);
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* prepare index info context --- parameters should match
|
|
|
|
* RelationInitIndexAccessInfo
|
|
|
|
*/
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
indexcxt = AllocSetContextCreate(CacheMemoryContext,
|
|
|
|
"index info",
|
|
|
|
ALLOCSET_SMALL_SIZES);
|
2002-02-19 21:11:20 +01:00
|
|
|
rel->rd_indexcxt = indexcxt;
|
2018-04-06 18:10:00 +02:00
|
|
|
MemoryContextCopyAndSetIdentifier(indexcxt,
|
Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in
all cases, and therefore always copied them into the context header.
Commit 9fa6f00b1 rethought this a little bit and invented a distinction
between fixed and variable names, skipping the copy step for the former.
But we can make things both simpler and more useful by instead allowing
there to be two parts to a context's identification, a fixed "name" and
an optional, variable "ident". The name supplied in the context create
call is now required to be a compile-time-constant string in all cases,
as it is never copied but just pointed to. The "ident" string, if
wanted, is supplied later. This is needed because typically we want
the ident to be stored inside the context so that it's cleaned up
automatically on context deletion; that means it has to be copied into
the context before we can set the pointer.
The cost of this approach is basically just an additional pointer field
in struct MemoryContextData, which isn't much overhead, and is bought
back entirely in the AllocSet case by not needing a headerSize field
anymore, since we no longer have to cope with variable header length.
In addition, we can simplify the internal interfaces for memory context
creation still further, saving a few cycles there. And it's no longer
true that a custom identifier disqualifies a context from participating
in aset.c's freelist scheme, so possibly there's some win on that end.
All the places that were using non-compile-time-constant context names
are adjusted to put the variable info into the "ident" instead. This
allows more effective identification of those contexts in many cases;
for example, subsidary contexts of relcache entries are now identified
by both type (e.g. "index info") and relname, where before you got only
one or the other. Contexts associated with PL function cache entries
are now identified more fully and uniformly, too.
I also arranged for plancache contexts to use the query source string
as their identifier. This is basically free for CachedPlanSources, as
they contained a copy of that string already. We pay an extra pstrdup
to do it for CachedPlans. That could perhaps be avoided, but it would
make things more fragile (since the CachedPlanSource is sometimes
destroyed first). I suspect future improvements in error reporting will
require CachedPlans to have a copy of that string anyway, so it's not
clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight
to stderr, nor do they know all details of the output format. This
is useful immediately to reduce code duplication, and it also allows
for external code to do something with stats output that's different
from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that
it rethinks some of the API changes made by commit 9fa6f00b1. Seems
better for extension authors to endure just one round of API changes
not two.
Discussion: https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
2018-03-27 22:46:47 +02:00
|
|
|
RelationGetRelationName(rel));
|
2002-02-19 21:11:20 +01:00
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
/*
|
|
|
|
* Now we can fetch the index AM's API struct. (We can't store
|
|
|
|
* that in the init file, since it contains function pointers that
|
|
|
|
* might vary across server executions. Fortunately, it should be
|
|
|
|
* safe to call the amhandler even while bootstrapping indexes.)
|
|
|
|
*/
|
|
|
|
InitIndexAmRoutine(rel);
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of opfamily OIDs */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2006-12-23 01:43:13 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
opfamily = (Oid *) MemoryContextAlloc(indexcxt, len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(opfamily, 1, len, fp) != len)
|
2006-12-23 01:43:13 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
rel->rd_opfamily = opfamily;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of opcintype OIDs */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2006-12-23 01:43:13 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
opcintype = (Oid *) MemoryContextAlloc(indexcxt, len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(opcintype, 1, len, fp) != len)
|
2006-12-23 01:43:13 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
rel->rd_opcintype = opcintype;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of support procedure OIDs */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2002-02-19 21:11:20 +01:00
|
|
|
goto read_failed;
|
|
|
|
support = (RegProcedure *) MemoryContextAlloc(indexcxt, len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(support, 1, len, fp) != len)
|
2002-02-19 21:11:20 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
rel->rd_support = support;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of collation OIDs */
|
2011-02-08 22:04:18 +01:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
indcollation = (Oid *) MemoryContextAlloc(indexcxt, len);
|
|
|
|
if (fread(indcollation, 1, len, fp) != len)
|
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
rel->rd_indcollation = indcollation;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of indoption values */
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
2007-01-09 03:14:16 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
indoption = (int16 *) MemoryContextAlloc(indexcxt, len);
|
2009-08-30 19:18:52 +02:00
|
|
|
if (fread(indoption, 1, len, fp) != len)
|
2007-01-09 03:14:16 +01:00
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
rel->rd_indoption = indoption;
|
|
|
|
|
2023-02-06 22:39:48 +01:00
|
|
|
/* read the vector of opcoptions values */
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
rel->rd_opcoptions = (bytea **)
|
|
|
|
MemoryContextAllocZero(indexcxt, sizeof(*rel->rd_opcoptions) * relform->relnatts);
|
|
|
|
|
|
|
|
for (i = 0; i < relform->relnatts; i++)
|
|
|
|
{
|
|
|
|
if (fread(&len, 1, sizeof(len), fp) != sizeof(len))
|
|
|
|
goto read_failed;
|
|
|
|
|
|
|
|
if (len > 0)
|
|
|
|
{
|
|
|
|
rel->rd_opcoptions[i] = (bytea *) MemoryContextAlloc(indexcxt, len);
|
|
|
|
if (fread(rel->rd_opcoptions[i], 1, len, fp) != len)
|
|
|
|
goto read_failed;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Restructure index access method API to hide most of it at the C level.
This patch reduces pg_am to just two columns, a name and a handler
function. All the data formerly obtained from pg_am is now provided
in a C struct returned by the handler function. This is similar to
the designs we've adopted for FDWs and tablesample methods. There
are multiple advantages. For one, the index AM's support functions
are now simple C functions, making them faster to call and much less
error-prone, since the C compiler can now check function signatures.
For another, this will make it far more practical to define index access
methods in installable extensions.
A disadvantage is that SQL-level code can no longer see attributes
of index AMs; in particular, some of the crosschecks in the opr_sanity
regression test are no longer possible from SQL. We've addressed that
by adding a facility for the index AM to perform such checks instead.
(Much more could be done in that line, but for now we're content if the
amvalidate functions more or less replace what opr_sanity used to do.)
We might also want to expose some sort of reporting functionality, but
this patch doesn't do that.
Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily
editorialized on by me.
2016-01-18 01:36:59 +01:00
|
|
|
/* set up zeroed fmgr-info vector */
|
2020-03-30 22:40:22 +02:00
|
|
|
nsupport = relform->relnatts * rel->rd_indam->amsupport;
|
2002-02-19 21:11:20 +01:00
|
|
|
rel->rd_supportinfo = (FmgrInfo *)
|
2003-11-09 22:30:38 +01:00
|
|
|
MemoryContextAllocZero(indexcxt, nsupport * sizeof(FmgrInfo));
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Count nailed rels to ensure we have 'em all */
|
|
|
|
if (rel->rd_isnailed)
|
|
|
|
nailed_rels++;
|
|
|
|
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
/* Load table AM data */
|
2021-12-03 13:38:26 +01:00
|
|
|
if (RELKIND_HAS_TABLE_AM(rel->rd_rel->relkind) || rel->rd_rel->relkind == RELKIND_SEQUENCE)
|
tableam: introduce table AM infrastructure.
This introduces the concept of table access methods, i.e. CREATE
ACCESS METHOD ... TYPE TABLE and
CREATE TABLE ... USING (storage-engine).
No table access functionality is delegated to table AMs as of this
commit, that'll be done in following commits.
Subsequent commits will incrementally abstract table access
functionality to be routed through table access methods. That change
is too large to be reviewed & committed at once, so it'll be done
incrementally.
Docs will be updated at the end, as adding them incrementally would
likely make them less coherent, and definitely is a lot more work,
without a lot of benefit.
Table access methods are specified similar to index access methods,
i.e. pg_am.amhandler returns, as INTERNAL, a pointer to a struct with
callbacks. In contrast to index AMs that struct needs to live as long
as a backend, typically that's achieved by just returning a pointer to
a constant struct.
Psql's \d+ now displays a table's access method. That can be disabled
with HIDE_TABLEAM=true, which is mainly useful so regression tests can
be run against different AMs. It's quite possible that this behaviour
still needs to be fine tuned.
For now it's not allowed to set a table AM for a partitioned table, as
we've not resolved how partitions would inherit that. Disallowing
allows us to introduce, if we decide that's the way forward, such a
behaviour without a compatibility break.
Catversion bumped, to add the heap table AM and references to it.
Author: Haribabu Kommi, Andres Freund, Alvaro Herrera, Dimitri Golgov and others
Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20160812231527.GA690404@alvherre.pgsql
https://postgr.es/m/20190107235616.6lur25ph22u5u5av@alap3.anarazel.de
https://postgr.es/m/20190304234700.w5tmhducs5wxgzls@alap3.anarazel.de
2019-03-06 18:54:38 +01:00
|
|
|
RelationInitTableAccessMethod(rel);
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
Assert(rel->rd_index == NULL);
|
2003-05-28 18:04:02 +02:00
|
|
|
Assert(rel->rd_indextuple == NULL);
|
2002-02-19 21:11:20 +01:00
|
|
|
Assert(rel->rd_indexcxt == NULL);
|
2019-01-22 02:36:55 +01:00
|
|
|
Assert(rel->rd_indam == NULL);
|
2006-12-23 01:43:13 +01:00
|
|
|
Assert(rel->rd_opfamily == NULL);
|
|
|
|
Assert(rel->rd_opcintype == NULL);
|
2002-02-19 21:11:20 +01:00
|
|
|
Assert(rel->rd_support == NULL);
|
|
|
|
Assert(rel->rd_supportinfo == NULL);
|
2007-01-09 03:14:16 +01:00
|
|
|
Assert(rel->rd_indoption == NULL);
|
2011-02-08 22:04:18 +01:00
|
|
|
Assert(rel->rd_indcollation == NULL);
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
Assert(rel->rd_opcoptions == NULL);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Rules and triggers are not saved (mainly because the internal
|
|
|
|
* format is complex and subject to change). They must be rebuilt if
|
2009-08-12 22:53:31 +02:00
|
|
|
* needed by RelationCacheInitializePhase3. This is not expected to
|
2002-02-19 21:11:20 +01:00
|
|
|
* be a big performance hit since few system catalogs have such. Ditto
|
2019-04-13 19:22:26 +02:00
|
|
|
* for RLS policy data, partition info, index expressions, predicates,
|
|
|
|
* exclusion info, and FDW info.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
rel->rd_rules = NULL;
|
|
|
|
rel->rd_rulescxt = NULL;
|
|
|
|
rel->trigdesc = NULL;
|
Clean up includes from RLS patch
The initial patch for RLS mistakenly included headers associated with
the executor and planner bits in rewrite/rowsecurity.h. Per policy and
general good sense, executor headers should not be included in planner
headers or vice versa.
The include of execnodes.h was a mistaken holdover from previous
versions, while the include of relation.h was used for Relation's
definition, which should have been coming from utils/relcache.h. This
patch cleans these issues up, adds comments to the RowSecurityPolicy
struct and the RowSecurityConfigType enum, and changes Relation->rsdesc
to Relation->rd_rsdesc to follow Relation field naming convention.
Additionally, utils/rel.h was including rewrite/rowsecurity.h, which
wasn't a great idea since that was pulling in things not really needed
in utils/rel.h (which gets included in quite a few places). Instead,
use 'struct RowSecurityDesc' for the rd_rsdesc field and add comments
explaining why.
Lastly, add an include into access/nbtree/nbtsort.c for
utils/sortsupport.h, which was evidently missed due to the above mess.
Pointed out by Tom in 16970.1415838651@sss.pgh.pa.us; note that the
concerns regarding a similar situation in the custom-path commit still
need to be addressed.
2014-11-14 22:53:51 +01:00
|
|
|
rel->rd_rsdesc = NULL;
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
rel->rd_partkey = NULL;
|
2019-04-13 19:22:26 +02:00
|
|
|
rel->rd_partkeycxt = NULL;
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
rel->rd_partdesc = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
rel->rd_partdesc_nodetached = NULL;
|
|
|
|
rel->rd_partdesc_nodetached_xmin = InvalidTransactionId;
|
2019-04-13 19:22:26 +02:00
|
|
|
rel->rd_pdcxt = NULL;
|
2021-04-28 21:44:35 +02:00
|
|
|
rel->rd_pddcxt = NULL;
|
Implement table partitioning.
Table partitioning is like table inheritance and reuses much of the
existing infrastructure, but there are some important differences.
The parent is called a partitioned table and is always empty; it may
not have indexes or non-inherited constraints, since those make no
sense for a relation with no data of its own. The children are called
partitions and contain all of the actual data. Each partition has an
implicit partitioning constraint. Multiple inheritance is not
allowed, and partitioning and inheritance can't be mixed. Partitions
can't have extra columns and may not allow nulls unless the parent
does. Tuples inserted into the parent are automatically routed to the
correct partition, so tuple-routing ON INSERT triggers are not needed.
Tuple routing isn't yet supported for partitions which are foreign
tables, and it doesn't handle updates that cross partition boundaries.
Currently, tables can be range-partitioned or list-partitioned. List
partitioning is limited to a single column, but range partitioning can
involve multiple columns. A partitioning "column" can be an
expression.
Because table partitioning is less general than table inheritance, it
is hoped that it will be easier to reason about properties of
partitions, and therefore that this will serve as a better foundation
for a variety of possible optimizations, including query planner
optimizations. The tuple routing based which this patch does based on
the implicit partitioning constraints is an example of this, but it
seems likely that many other useful optimizations are also possible.
Amit Langote, reviewed and tested by Robert Haas, Ashutosh Bapat,
Amit Kapila, Rajkumar Raghuwanshi, Corey Huinker, Jaime Casanova,
Rushabh Lathia, Erik Rijkers, among others. Minor revisions by me.
2016-12-07 19:17:43 +01:00
|
|
|
rel->rd_partcheck = NIL;
|
2019-04-13 19:22:26 +02:00
|
|
|
rel->rd_partcheckvalid = false;
|
|
|
|
rel->rd_partcheckcxt = NULL;
|
2003-05-28 18:04:02 +02:00
|
|
|
rel->rd_indexprs = NIL;
|
|
|
|
rel->rd_indpred = NIL;
|
2009-12-07 06:22:23 +01:00
|
|
|
rel->rd_exclops = NULL;
|
|
|
|
rel->rd_exclprocs = NULL;
|
|
|
|
rel->rd_exclstrats = NULL;
|
2013-03-07 05:47:38 +01:00
|
|
|
rel->rd_fdwroutine = NULL;
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Reset transient-state fields in the relcache entry
|
|
|
|
*/
|
2004-02-10 02:55:27 +01:00
|
|
|
rel->rd_smgr = NULL;
|
2002-02-19 21:11:20 +01:00
|
|
|
if (rel->rd_isnailed)
|
2004-07-17 05:32:14 +02:00
|
|
|
rel->rd_refcnt = 1;
|
2002-02-19 21:11:20 +01:00
|
|
|
else
|
2004-07-17 05:32:14 +02:00
|
|
|
rel->rd_refcnt = 0;
|
2019-05-03 16:26:14 +02:00
|
|
|
rel->rd_indexvalid = false;
|
2002-02-19 21:11:20 +01:00
|
|
|
rel->rd_indexlist = NIL;
|
2017-01-19 18:00:00 +01:00
|
|
|
rel->rd_pkindex = InvalidOid;
|
2014-05-14 20:55:48 +02:00
|
|
|
rel->rd_replidindex = InvalidOid;
|
Ignore BRIN indexes when checking for HOT updates
When determining whether an index update may be skipped by using HOT, we
can ignore attributes indexed by block summarizing indexes without
references to individual tuples that need to be cleaned up.
A new type TU_UpdateIndexes provides a signal to the executor to
determine which indexes to update - no indexes, all indexes, or only the
summarizing indexes.
This also removes rd_indexattr list, and replaces it with rd_attrsvalid
flag. The list was not used anywhere, and a simple flag is sufficient.
This was originally committed as 5753d4ee32, but then got reverted by
e3fcca0d0d because of correctness issues.
Original patch by Josef Simanek, various fixes and improvements by Tomas
Vondra and me.
Authors: Matthias van de Meent, Josef Simanek, Tomas Vondra
Reviewed-by: Tomas Vondra, Alvaro Herrera
Discussion: https://postgr.es/m/05ebcb44-f383-86e3-4f31-0a97a55634cf@enterprisedb.com
Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com
2023-03-20 10:34:07 +01:00
|
|
|
rel->rd_attrsvalid = false;
|
2014-05-14 20:55:48 +02:00
|
|
|
rel->rd_keyattr = NULL;
|
2017-01-19 18:00:00 +01:00
|
|
|
rel->rd_pkattr = NULL;
|
2014-05-14 20:55:48 +02:00
|
|
|
rel->rd_idattr = NULL;
|
Allow specifying row filters for logical replication of tables.
This feature adds row filtering for publication tables. When a publication
is defined or modified, an optional WHERE clause can be specified. Rows
that don't satisfy this WHERE clause will be filtered out. This allows a
set of tables to be partially replicated. The row filter is per table. A
new row filter can be added simply by specifying a WHERE clause after the
table name. The WHERE clause must be enclosed by parentheses.
The row filter WHERE clause for a table added to a publication that
publishes UPDATE and/or DELETE operations must contain only columns that
are covered by REPLICA IDENTITY. The row filter WHERE clause for a table
added to a publication that publishes INSERT can use any column. If the
row filter evaluates to NULL, it is regarded as "false". The WHERE clause
only allows simple expressions that don't have user-defined functions,
user-defined operators, user-defined types, user-defined collations,
non-immutable built-in functions, or references to system columns. These
restrictions could be addressed in the future.
If you choose to do the initial table synchronization, only data that
satisfies the row filters is copied to the subscriber. If the subscription
has several publications in which a table has been published with
different WHERE clauses, rows that satisfy ANY of the expressions will be
copied. If a subscriber is a pre-15 version, the initial table
synchronization won't use row filters even if they are defined in the
publisher.
The row filters are applied before publishing the changes. If the
subscription has several publications in which the same table has been
published with different filters (for the same publish operation), those
expressions get OR'ed together so that rows satisfying any of the
expressions will be replicated.
This means all the other filters become redundant if (a) one of the
publications have no filter at all, (b) one of the publications was
created using FOR ALL TABLES, (c) one of the publications was created
using FOR ALL TABLES IN SCHEMA and the table belongs to that same schema.
If your publication contains a partitioned table, the publication
parameter publish_via_partition_root determines if it uses the partition's
row filter (if the parameter is false, the default) or the root
partitioned table's row filter.
Psql commands \dRp+ and \d <table-name> will display any row filters.
Author: Hou Zhijie, Euler Taveira, Peter Smith, Ajin Cherian
Reviewed-by: Greg Nancarrow, Haiying Tang, Amit Kapila, Tomas Vondra, Dilip Kumar, Vignesh C, Alvaro Herrera, Andres Freund, Wei Wang
Discussion: https://www.postgresql.org/message-id/flat/CAHE3wggb715X%2BmK_DitLXF25B%3DjE6xyNCH4YOwM860JR7HarGQ%40mail.gmail.com
2022-02-22 03:24:12 +01:00
|
|
|
rel->rd_pubdesc = NULL;
|
Implement multivariate n-distinct coefficients
Add support for explicitly declared statistic objects (CREATE
STATISTICS), allowing collection of statistics on more complex
combinations that individual table columns. Companion commands DROP
STATISTICS and ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME are
added too. All this DDL has been designed so that more statistic types
can be added later on, such as multivariate most-common-values and
multivariate histograms between columns of a single table, leaving room
for permitting columns on multiple tables, too, as well as expressions.
This commit only adds support for collection of n-distinct coefficient
on user-specified sets of columns in a single table. This is useful to
estimate number of distinct groups in GROUP BY and DISTINCT clauses;
estimation errors there can cause over-allocation of memory in hashed
aggregates, for instance, so it's a worthwhile problem to solve. A new
special pseudo-type pg_ndistinct is used.
(num-distinct estimation was deemed sufficiently useful by itself that
this is worthwhile even if no further statistic types are added
immediately; so much so that another version of essentially the same
functionality was submitted by Kyotaro Horiguchi:
https://postgr.es/m/20150828.173334.114731693.horiguchi.kyotaro@lab.ntt.co.jp
though this commit does not use that code.)
Author: Tomas Vondra. Some code rework by Álvaro.
Reviewed-by: Dean Rasheed, David Rowley, Kyotaro Horiguchi, Jeff Janes,
Ideriha Takeshi
Discussion: https://postgr.es/m/543AFA15.4080608@fuzzy.cz
https://postgr.es/m/20170320190220.ixlaueanxegqd5gr@alvherre.pgsql
2017-03-24 18:06:10 +01:00
|
|
|
rel->rd_statvalid = false;
|
|
|
|
rel->rd_statlist = NIL;
|
2019-05-03 16:26:14 +02:00
|
|
|
rel->rd_fkeyvalid = false;
|
|
|
|
rel->rd_fkeylist = NIL;
|
2004-11-20 21:19:52 +01:00
|
|
|
rel->rd_createSubid = InvalidSubTransactionId;
|
Change internal RelFileNode references to RelFileNumber or RelFileLocator.
We have been using the term RelFileNode to refer to either (1) the
integer that is used to name the sequence of files for a certain relation
within the directory set aside for that tablespace/database combination;
or (2) that value plus the OIDs of the tablespace and database; or
occasionally (3) the whole series of files created for a relation
based on those values. Using the same name for more than one thing is
confusing.
Replace RelFileNode with RelFileNumber when we're talking about just the
single number, i.e. (1) from above, and with RelFileLocator when we're
talking about all the things that are needed to locate a relation's files
on disk, i.e. (2) from above. In the places where we refer to (3) as
a relfilenode, instead refer to "relation storage".
Since there is a ton of SQL code in the world that knows about
pg_class.relfilenode, don't change the name of that column, or of other
SQL-facing things that derive their name from it.
On the other hand, do adjust closely-related internal terminology. For
example, the structure member names dbNode and spcNode appear to be
derived from the fact that the structure itself was called RelFileNode,
so change those to dbOid and spcOid. Likewise, various variables with
names like rnode and relnode get renamed appropriately, according to
how they're being used in context.
Hopefully, this is clearer than before. It is also preparation for
future patches that intend to widen the relfilenumber fields from its
current width of 32 bits. Variables that store a relfilenumber are now
declared as type RelFileNumber rather than type Oid; right now, these
are the same, but that can now more easily be changed.
Dilip Kumar, per an idea from me. Reviewed also by Andres Freund.
I fixed some whitespace issues, changed a couple of words in a
comment, and made one other minor correction.
Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com
Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
2022-07-06 17:39:09 +02:00
|
|
|
rel->rd_newRelfilelocatorSubid = InvalidSubTransactionId;
|
|
|
|
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
|
Skip WAL for new relfilenodes, under wal_level=minimal.
Until now, only selected bulk operations (e.g. COPY) did this. If a
given relfilenode received both a WAL-skipping COPY and a WAL-logged
operation (e.g. INSERT), recovery could lose tuples from the COPY. See
src/backend/access/transam/README section "Skipping WAL for New
RelFileNode" for the new coding rules. Maintainers of table access
methods should examine that section.
To maintain data durability, just before commit, we choose between an
fsync of the relfilenode and copying its contents to WAL. A new GUC,
wal_skip_threshold, guides that choice. If this change slows a workload
that creates small, permanent relfilenodes under wal_level=minimal, try
adjusting wal_skip_threshold. Users setting a timeout on COMMIT may
need to adjust that timeout, and log_min_duration_statement analysis
will reflect time consumption moving to COMMIT from commands like COPY.
Internally, this requires a reliable determination of whether
RollbackAndReleaseCurrentSubTransaction() would unlink a relation's
current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the
specification of rd_createSubid such that the field is zero when a new
rel has an old rd_node. Make relcache.c retain entries for certain
dropped relations until end of transaction.
Bump XLOG_PAGE_MAGIC, since this introduces XLOG_GIST_ASSIGN_LSN.
Future servers accept older WAL, so this bump is discretionary.
Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert
Haas. Heikki Linnakangas and Michael Paquier implemented earlier
designs that materially clarified the problem. Reviewed, in earlier
designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane,
Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout.
Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org
2020-04-04 21:25:34 +02:00
|
|
|
rel->rd_droppedSubid = InvalidSubTransactionId;
|
2006-04-26 00:46:05 +02:00
|
|
|
rel->rd_amcache = NULL;
|
2022-06-30 20:10:13 +02:00
|
|
|
rel->pgstat_info = NULL;
|
1999-09-18 21:08:25 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
2004-06-18 08:14:31 +02:00
|
|
|
* Recompute lock and physical addressing info. This is needed in
|
|
|
|
* case the pg_internal.init file was copied from some other database
|
|
|
|
* by CREATE DATABASE.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
RelationInitLockInfo(rel);
|
2004-06-18 08:14:31 +02:00
|
|
|
RelationInitPhysicalAddr(rel);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
* We reached the end of the init file without apparent problem. Did we
|
|
|
|
* get the right number of nailed items? This is a useful crosscheck in
|
|
|
|
* case the set of critical rels or indexes changes. However, that should
|
|
|
|
* not happen in a normally-running system, so let's bleat if it does.
|
2015-11-11 19:39:21 +01:00
|
|
|
*
|
|
|
|
* For the shared init file, we're called before client authentication is
|
|
|
|
* done, which means that elog(WARNING) will go only to the postmaster
|
|
|
|
* log, where it's easily missed. To ensure that developers notice bad
|
|
|
|
* values of NUM_CRITICAL_SHARED_RELS/NUM_CRITICAL_SHARED_INDEXES, we put
|
|
|
|
* an Assert(false) there.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
2009-08-12 22:53:31 +02:00
|
|
|
if (shared)
|
|
|
|
{
|
|
|
|
if (nailed_rels != NUM_CRITICAL_SHARED_RELS ||
|
|
|
|
nailed_indexes != NUM_CRITICAL_SHARED_INDEXES)
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
{
|
|
|
|
elog(WARNING, "found %d nailed shared rels and %d nailed shared indexes in init file, but expected %d and %d respectively",
|
|
|
|
nailed_rels, nailed_indexes,
|
|
|
|
NUM_CRITICAL_SHARED_RELS, NUM_CRITICAL_SHARED_INDEXES);
|
2015-11-11 19:39:21 +01:00
|
|
|
/* Make sure we get developers' attention about this */
|
|
|
|
Assert(false);
|
|
|
|
/* In production builds, recover by bootstrapping the relcache */
|
2009-08-12 22:53:31 +02:00
|
|
|
goto read_failed;
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
}
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
if (nailed_rels != NUM_CRITICAL_LOCAL_RELS ||
|
|
|
|
nailed_indexes != NUM_CRITICAL_LOCAL_INDEXES)
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
{
|
|
|
|
elog(WARNING, "found %d nailed rels and %d nailed indexes in init file, but expected %d and %d respectively",
|
|
|
|
nailed_rels, nailed_indexes,
|
|
|
|
NUM_CRITICAL_LOCAL_RELS, NUM_CRITICAL_LOCAL_INDEXES);
|
2015-11-11 19:39:21 +01:00
|
|
|
/* We don't need an Assert() in this case */
|
2009-08-12 22:53:31 +02:00
|
|
|
goto read_failed;
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
}
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
2002-02-19 21:11:20 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* OK, all appears well.
|
|
|
|
*
|
|
|
|
* Now insert all the new relcache entries into the cache.
|
|
|
|
*/
|
|
|
|
for (relno = 0; relno < num_rels; relno++)
|
|
|
|
{
|
Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that hash_search(HASH_ENTER)
might find a hashtable entry already present for the same OID. However,
that can in fact occur during recursive relcache load scenarios. When it
did happen, we overwrote the pointer to the pre-existing Relation, causing
a session-lifespan leakage of that entire structure. As far as is known,
the pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes the
pre-existing Relation if so. If by some chance its refcount is positive,
elog a WARNING and allow the pre-existing Relation to be leaked as before.
Also, AttrDefaultFetch() was sloppy about leaking the cstring form of the
pg_attrdef.adbin value it's copying into the relcache structure. This is
only a query-lifespan leakage, and normally not very significant, but it
adds up during CLOBBER_CACHE testing.
These bugs are of very ancient vintage, but I'll refrain from back-patching
since there's no evidence that these leaks amount to anything in ordinary
usage.
2014-05-18 22:51:46 +02:00
|
|
|
RelationCacheInsert(rels[relno], false);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2002-01-16 18:34:42 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
pfree(rels);
|
|
|
|
FreeFile(fp);
|
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
if (shared)
|
|
|
|
criticalSharedRelcachesBuilt = true;
|
|
|
|
else
|
|
|
|
criticalRelcachesBuilt = true;
|
2002-02-19 21:11:20 +01:00
|
|
|
return true;
|
2002-01-16 18:34:42 +01:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* init file is broken, so do it the hard way. We don't bother trying to
|
|
|
|
* free the clutter we just allocated; it's not in the relcache so it
|
|
|
|
* won't hurt.
|
|
|
|
*/
|
2002-01-16 18:34:42 +01:00
|
|
|
read_failed:
|
2002-02-19 21:11:20 +01:00
|
|
|
pfree(rels);
|
|
|
|
FreeFile(fp);
|
|
|
|
|
|
|
|
return false;
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Write out a new initialization file with the current contents
|
2009-08-12 22:53:31 +02:00
|
|
|
* of the relcache (either shared rels or local rels, as indicated).
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
1997-08-19 23:40:56 +02:00
|
|
|
static void
|
2009-08-12 22:53:31 +02:00
|
|
|
write_relcache_init_file(bool shared)
|
1996-07-09 08:22:35 +02:00
|
|
|
{
|
2002-02-19 21:11:20 +01:00
|
|
|
FILE *fp;
|
2000-03-31 21:39:22 +02:00
|
|
|
char tempfilename[MAXPGPATH];
|
|
|
|
char finalfilename[MAXPGPATH];
|
2003-11-09 22:30:38 +01:00
|
|
|
int magic;
|
2002-02-19 21:11:20 +01:00
|
|
|
HASH_SEQ_STATUS status;
|
2002-03-26 20:17:02 +01:00
|
|
|
RelIdCacheEnt *idhentry;
|
2002-02-19 21:11:20 +01:00
|
|
|
int i;
|
2000-03-31 21:39:22 +02:00
|
|
|
|
Use a safer method for determining whether relcache init file is stale.
When we invalidate the relcache entry for a system catalog or index, we
must also delete the relcache "init file" if the init file contains a copy
of that rel's entry. The old way of doing this relied on a specially
maintained list of the OIDs of relations present in the init file: we made
the list either when reading the file in, or when writing the file out.
The problem is that when writing the file out, we included only rels
present in our local relcache, which might have already suffered some
deletions due to relcache inval events. In such cases we correctly decided
not to overwrite the real init file with incomplete data --- but we still
used the incomplete initFileRelationIds list for the rest of the current
session. This could result in wrong decisions about whether the session's
own actions require deletion of the init file, potentially allowing an init
file created by some other concurrent session to be left around even though
it's been made stale.
Since we don't support changing the schema of a system catalog at runtime,
the only likely scenario in which this would cause a problem in the field
involves a "vacuum full" on a catalog concurrently with other activity, and
even then it's far from easy to provoke. Remarkably, this has been broken
since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had
never seen a reproducible test case until recently. If it did happen in
the field, the symptoms would probably involve unexpected "cache lookup
failed" errors to begin with, then "could not open file" failures after the
next checkpoint, as all accesses to the affected catalog stopped working.
Recovery would require manually removing the stale "pg_internal.init" file.
To fix, get rid of the initFileRelationIds list, and instead consult
syscache.c's list of relations used in catalog caches to decide whether a
relation is included in the init file. This should be a tad more efficient
anyway, since we're replacing linear search of a list with ~100 entries
with a binary search. It's a bit ugly that the init file contents are now
so directly tied to the catalog caches, but in practice that won't make
much difference.
Back-patch to all supported branches.
2015-06-07 21:32:09 +02:00
|
|
|
/*
|
|
|
|
* If we have already received any relcache inval events, there's no
|
|
|
|
* chance of succeeding so we may as well skip the whole thing.
|
|
|
|
*/
|
|
|
|
if (relcacheInvalsReceived != 0L)
|
|
|
|
return;
|
|
|
|
|
2000-03-31 21:39:22 +02:00
|
|
|
/*
|
|
|
|
* We must write a temporary file and rename it into place. Otherwise,
|
|
|
|
* another backend starting at about the same time might crash trying to
|
|
|
|
* read the partially-complete file.
|
|
|
|
*/
|
2009-08-12 22:53:31 +02:00
|
|
|
if (shared)
|
|
|
|
{
|
|
|
|
snprintf(tempfilename, sizeof(tempfilename), "global/%s.%d",
|
|
|
|
RELCACHE_INIT_FILENAME, MyProcPid);
|
|
|
|
snprintf(finalfilename, sizeof(finalfilename), "global/%s",
|
|
|
|
RELCACHE_INIT_FILENAME);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
snprintf(tempfilename, sizeof(tempfilename), "%s/%s.%d",
|
|
|
|
DatabasePath, RELCACHE_INIT_FILENAME, MyProcPid);
|
|
|
|
snprintf(finalfilename, sizeof(finalfilename), "%s/%s",
|
|
|
|
DatabasePath, RELCACHE_INIT_FILENAME);
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
unlink(tempfilename); /* in case it exists w/wrong permissions */
|
|
|
|
|
|
|
|
fp = AllocateFile(tempfilename, PG_BINARY_W);
|
|
|
|
if (fp == NULL)
|
2000-06-20 01:40:48 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We used to consider this a fatal error, but we might as well
|
|
|
|
* continue with backend startup ...
|
|
|
|
*/
|
2003-07-25 22:18:01 +02:00
|
|
|
ereport(WARNING,
|
|
|
|
(errcode_for_file_access(),
|
2003-09-25 08:58:07 +02:00
|
|
|
errmsg("could not create relation-cache initialization file \"%s\": %m",
|
2003-07-25 22:18:01 +02:00
|
|
|
tempfilename),
|
|
|
|
errdetail("Continuing anyway, but there's something wrong.")));
|
2000-06-20 01:40:48 +02:00
|
|
|
return;
|
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2003-11-09 22:30:38 +01:00
|
|
|
/*
|
|
|
|
* Write a magic number to serve as a file version identifier. We can
|
|
|
|
* change the magic number whenever the relcache layout changes.
|
|
|
|
*/
|
|
|
|
magic = RELCACHE_INIT_FILEMAGIC;
|
|
|
|
if (fwrite(&magic, 1, sizeof(magic), fp) != sizeof(magic))
|
|
|
|
elog(FATAL, "could not write init file");
|
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/*
|
2009-08-12 22:53:31 +02:00
|
|
|
* Write all the appropriate reldescs (in no particular order).
|
2000-02-18 10:30:20 +01:00
|
|
|
*/
|
2002-03-26 20:17:02 +01:00
|
|
|
hash_seq_init(&status, RelationIdCache);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2002-03-26 20:17:02 +01:00
|
|
|
while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2002-03-26 20:17:02 +01:00
|
|
|
Relation rel = idhentry->reldesc;
|
2002-02-19 21:11:20 +01:00
|
|
|
Form_pg_class relform = rel->rd_rel;
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/* ignore if not correct group */
|
|
|
|
if (relform->relisshared != shared)
|
|
|
|
continue;
|
|
|
|
|
Use a safer method for determining whether relcache init file is stale.
When we invalidate the relcache entry for a system catalog or index, we
must also delete the relcache "init file" if the init file contains a copy
of that rel's entry. The old way of doing this relied on a specially
maintained list of the OIDs of relations present in the init file: we made
the list either when reading the file in, or when writing the file out.
The problem is that when writing the file out, we included only rels
present in our local relcache, which might have already suffered some
deletions due to relcache inval events. In such cases we correctly decided
not to overwrite the real init file with incomplete data --- but we still
used the incomplete initFileRelationIds list for the rest of the current
session. This could result in wrong decisions about whether the session's
own actions require deletion of the init file, potentially allowing an init
file created by some other concurrent session to be left around even though
it's been made stale.
Since we don't support changing the schema of a system catalog at runtime,
the only likely scenario in which this would cause a problem in the field
involves a "vacuum full" on a catalog concurrently with other activity, and
even then it's far from easy to provoke. Remarkably, this has been broken
since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had
never seen a reproducible test case until recently. If it did happen in
the field, the symptoms would probably involve unexpected "cache lookup
failed" errors to begin with, then "could not open file" failures after the
next checkpoint, as all accesses to the affected catalog stopped working.
Recovery would require manually removing the stale "pg_internal.init" file.
To fix, get rid of the initFileRelationIds list, and instead consult
syscache.c's list of relations used in catalog caches to decide whether a
relation is included in the init file. This should be a tad more efficient
anyway, since we're replacing linear search of a list with ~100 entries
with a binary search. It's a bit ugly that the init file contents are now
so directly tied to the catalog caches, but in practice that won't make
much difference.
Back-patch to all supported branches.
2015-06-07 21:32:09 +02:00
|
|
|
/*
|
|
|
|
* Ignore if not supposed to be in init file. We can allow any shared
|
|
|
|
* relation that's been loaded so far to be in the shared init file,
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
* but unshared relations must be ones that should be in the local
|
|
|
|
* file per RelationIdIsInInitFile. (Note: if you want to change the
|
|
|
|
* criterion for rels to be kept in the init file, see also inval.c.
|
|
|
|
* The reason for filtering here is to be sure that we don't put
|
|
|
|
* anything into the local init file for which a relcache inval would
|
|
|
|
* not cause invalidation of that init file.)
|
Use a safer method for determining whether relcache init file is stale.
When we invalidate the relcache entry for a system catalog or index, we
must also delete the relcache "init file" if the init file contains a copy
of that rel's entry. The old way of doing this relied on a specially
maintained list of the OIDs of relations present in the init file: we made
the list either when reading the file in, or when writing the file out.
The problem is that when writing the file out, we included only rels
present in our local relcache, which might have already suffered some
deletions due to relcache inval events. In such cases we correctly decided
not to overwrite the real init file with incomplete data --- but we still
used the incomplete initFileRelationIds list for the rest of the current
session. This could result in wrong decisions about whether the session's
own actions require deletion of the init file, potentially allowing an init
file created by some other concurrent session to be left around even though
it's been made stale.
Since we don't support changing the schema of a system catalog at runtime,
the only likely scenario in which this would cause a problem in the field
involves a "vacuum full" on a catalog concurrently with other activity, and
even then it's far from easy to provoke. Remarkably, this has been broken
since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had
never seen a reproducible test case until recently. If it did happen in
the field, the symptoms would probably involve unexpected "cache lookup
failed" errors to begin with, then "could not open file" failures after the
next checkpoint, as all accesses to the affected catalog stopped working.
Recovery would require manually removing the stale "pg_internal.init" file.
To fix, get rid of the initFileRelationIds list, and instead consult
syscache.c's list of relations used in catalog caches to decide whether a
relation is included in the init file. This should be a tad more efficient
anyway, since we're replacing linear search of a list with ~100 entries
with a binary search. It's a bit ugly that the init file contents are now
so directly tied to the catalog caches, but in practice that won't make
much difference.
Back-patch to all supported branches.
2015-06-07 21:32:09 +02:00
|
|
|
*/
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
if (!shared && !RelationIdIsInInitFile(RelationGetRelid(rel)))
|
|
|
|
{
|
|
|
|
/* Nailed rels had better get stored. */
|
|
|
|
Assert(!rel->rd_isnailed);
|
Use a safer method for determining whether relcache init file is stale.
When we invalidate the relcache entry for a system catalog or index, we
must also delete the relcache "init file" if the init file contains a copy
of that rel's entry. The old way of doing this relied on a specially
maintained list of the OIDs of relations present in the init file: we made
the list either when reading the file in, or when writing the file out.
The problem is that when writing the file out, we included only rels
present in our local relcache, which might have already suffered some
deletions due to relcache inval events. In such cases we correctly decided
not to overwrite the real init file with incomplete data --- but we still
used the incomplete initFileRelationIds list for the rest of the current
session. This could result in wrong decisions about whether the session's
own actions require deletion of the init file, potentially allowing an init
file created by some other concurrent session to be left around even though
it's been made stale.
Since we don't support changing the schema of a system catalog at runtime,
the only likely scenario in which this would cause a problem in the field
involves a "vacuum full" on a catalog concurrently with other activity, and
even then it's far from easy to provoke. Remarkably, this has been broken
since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had
never seen a reproducible test case until recently. If it did happen in
the field, the symptoms would probably involve unexpected "cache lookup
failed" errors to begin with, then "could not open file" failures after the
next checkpoint, as all accesses to the affected catalog stopped working.
Recovery would require manually removing the stale "pg_internal.init" file.
To fix, get rid of the initFileRelationIds list, and instead consult
syscache.c's list of relations used in catalog caches to decide whether a
relation is included in the init file. This should be a tad more efficient
anyway, since we're replacing linear search of a list with ~100 entries
with a binary search. It's a bit ugly that the init file contents are now
so directly tied to the catalog caches, but in practice that won't make
much difference.
Back-patch to all supported branches.
2015-06-07 21:32:09 +02:00
|
|
|
continue;
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
}
|
Use a safer method for determining whether relcache init file is stale.
When we invalidate the relcache entry for a system catalog or index, we
must also delete the relcache "init file" if the init file contains a copy
of that rel's entry. The old way of doing this relied on a specially
maintained list of the OIDs of relations present in the init file: we made
the list either when reading the file in, or when writing the file out.
The problem is that when writing the file out, we included only rels
present in our local relcache, which might have already suffered some
deletions due to relcache inval events. In such cases we correctly decided
not to overwrite the real init file with incomplete data --- but we still
used the incomplete initFileRelationIds list for the rest of the current
session. This could result in wrong decisions about whether the session's
own actions require deletion of the init file, potentially allowing an init
file created by some other concurrent session to be left around even though
it's been made stale.
Since we don't support changing the schema of a system catalog at runtime,
the only likely scenario in which this would cause a problem in the field
involves a "vacuum full" on a catalog concurrently with other activity, and
even then it's far from easy to provoke. Remarkably, this has been broken
since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had
never seen a reproducible test case until recently. If it did happen in
the field, the symptoms would probably involve unexpected "cache lookup
failed" errors to begin with, then "could not open file" failures after the
next checkpoint, as all accesses to the affected catalog stopped working.
Recovery would require manually removing the stale "pg_internal.init" file.
To fix, get rid of the initFileRelationIds list, and instead consult
syscache.c's list of relations used in catalog caches to decide whether a
relation is included in the init file. This should be a tad more efficient
anyway, since we're replacing linear search of a list with ~100 entries
with a binary search. It's a bit ugly that the init file contents are now
so directly tied to the catalog caches, but in practice that won't make
much difference.
Back-patch to all supported branches.
2015-06-07 21:32:09 +02:00
|
|
|
|
2006-07-02 04:23:23 +02:00
|
|
|
/* first write the relcache entry proper */
|
|
|
|
write_item(rel, sizeof(RelationData), fp);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* next write the relation tuple form */
|
2006-07-02 04:23:23 +02:00
|
|
|
write_item(relform, CLASS_TUPLE_SIZE, fp);
|
1997-09-07 07:04:48 +02:00
|
|
|
|
1996-07-09 08:22:35 +02:00
|
|
|
/* next, do all the attribute tuple form data entries */
|
|
|
|
for (i = 0; i < relform->relnatts; i++)
|
1997-09-07 07:04:48 +02:00
|
|
|
{
|
2017-08-20 20:19:07 +02:00
|
|
|
write_item(TupleDescAttr(rel->rd_att, i),
|
|
|
|
ATTRIBUTE_FIXED_PART_SIZE, fp);
|
1997-09-07 07:04:48 +02:00
|
|
|
}
|
|
|
|
|
2006-07-02 04:23:23 +02:00
|
|
|
/* next, do the access method specific field */
|
|
|
|
write_item(rel->rd_options,
|
2007-02-28 00:48:10 +01:00
|
|
|
(rel->rd_options ? VARSIZE(rel->rd_options) : 0),
|
2006-07-04 00:45:41 +02:00
|
|
|
fp);
|
2006-07-02 04:23:23 +02:00
|
|
|
|
Local partitioned indexes
When CREATE INDEX is run on a partitioned table, create catalog entries
for an index on the partitioned table (which is just a placeholder since
the table proper has no data of its own), and recurse to create actual
indexes on the existing partitions; create them in future partitions
also.
As a convenience gadget, if the new index definition matches some
existing index in partitions, these are picked up and used instead of
creating new ones. Whichever way these indexes come about, they become
attached to the index on the parent table and are dropped alongside it,
and cannot be dropped on isolation unless they are detached first.
To support pg_dump'ing these indexes, add commands
CREATE INDEX ON ONLY <table>
(which creates the index on the parent partitioned table, without
recursing) and
ALTER INDEX ATTACH PARTITION
(which is used after the indexes have been created individually on each
partition, to attach them to the parent index). These reconstruct prior
database state exactly.
Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
Langote, Jesper Pedersen, Simon Riggs, David Rowley
Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql
2018-01-19 15:49:22 +01:00
|
|
|
/*
|
|
|
|
* If it's an index, there's more to do. Note we explicitly ignore
|
|
|
|
* partitioned indexes here.
|
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
if (rel->rd_rel->relkind == RELKIND_INDEX)
|
|
|
|
{
|
2003-05-28 18:04:02 +02:00
|
|
|
/* write the pg_index tuple */
|
|
|
|
/* we assume this was created by heap_copytuple! */
|
2006-07-02 04:23:23 +02:00
|
|
|
write_item(rel->rd_indextuple,
|
2006-07-04 00:45:41 +02:00
|
|
|
HEAPTUPLESIZE + rel->rd_indextuple->t_len,
|
|
|
|
fp);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of opfamily OIDs */
|
2006-12-23 01:43:13 +01:00
|
|
|
write_item(rel->rd_opfamily,
|
|
|
|
relform->relnatts * sizeof(Oid),
|
|
|
|
fp);
|
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of opcintype OIDs */
|
2006-12-23 01:43:13 +01:00
|
|
|
write_item(rel->rd_opcintype,
|
|
|
|
relform->relnatts * sizeof(Oid),
|
|
|
|
fp);
|
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of support procedure OIDs */
|
2006-07-04 00:45:41 +02:00
|
|
|
write_item(rel->rd_support,
|
2020-03-30 22:40:22 +02:00
|
|
|
relform->relnatts * (rel->rd_indam->amsupport * sizeof(RegProcedure)),
|
2006-07-04 00:45:41 +02:00
|
|
|
fp);
|
2007-01-09 03:14:16 +01:00
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of collation OIDs */
|
2011-02-08 22:04:18 +01:00
|
|
|
write_item(rel->rd_indcollation,
|
|
|
|
relform->relnatts * sizeof(Oid),
|
|
|
|
fp);
|
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of indoption values */
|
2007-01-09 03:14:16 +01:00
|
|
|
write_item(rel->rd_indoption,
|
|
|
|
relform->relnatts * sizeof(int16),
|
|
|
|
fp);
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
|
|
|
|
Assert(rel->rd_opcoptions);
|
|
|
|
|
2023-02-06 22:58:44 +01:00
|
|
|
/* write the vector of opcoptions values */
|
Implement operator class parameters
PostgreSQL provides set of template index access methods, where opclasses have
much freedom in the semantics of indexing. These index AMs are GiST, GIN,
SP-GiST and BRIN. There opclasses define representation of keys, operations on
them and supported search strategies. So, it's natural that opclasses may be
faced some tradeoffs, which require user-side decision. This commit implements
opclass parameters allowing users to set some values, which tell opclass how to
index the particular dataset.
This commit doesn't introduce new storage in system catalog. Instead it uses
pg_attribute.attoptions, which is used for table column storage options but
unused for index attributes.
In order to evade changing signature of each opclass support function, we
implement unified way to pass options to opclass support functions. Options
are set to fn_expr as the constant bytea expression. It's possible due to the
fact that opclass support functions are executed outside of expressions, so
fn_expr is unused for them.
This commit comes with some examples of opclass options usage. We parametrize
signature length in GiST. That applies to multiple opclasses: tsvector_ops,
gist__intbig_ops, gist_ltree_ops, gist__ltree_ops, gist_trgm_ops and
gist_hstore_ops. Also we parametrize maximum number of integer ranges for
gist__int_ops. However, the main future usage of this feature is expected
to be json, where users would be able to specify which way to index particular
json parts.
Catversion is bumped.
Discussion: https://postgr.es/m/d22c3a18-31c7-1879-fc11-4c1ce2f5e5af%40postgrespro.ru
Author: Nikita Glukhov, revised by me
Reviwed-by: Nikolay Shaplov, Robert Haas, Tom Lane, Tomas Vondra, Alvaro Herrera
2020-03-30 18:17:11 +02:00
|
|
|
for (i = 0; i < relform->relnatts; i++)
|
|
|
|
{
|
|
|
|
bytea *opt = rel->rd_opcoptions[i];
|
|
|
|
|
|
|
|
write_item(opt, opt ? VARSIZE(opt) : 0, fp);
|
|
|
|
}
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
1997-09-07 07:04:48 +02:00
|
|
|
|
2004-01-26 23:35:32 +01:00
|
|
|
if (FreeFile(fp))
|
|
|
|
elog(FATAL, "could not write init file");
|
2000-03-31 21:39:22 +02:00
|
|
|
|
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* Now we have to check whether the data we've so painstakingly
|
|
|
|
* accumulated is already obsolete due to someone else's just-committed
|
|
|
|
* catalog changes. If so, we just delete the temp file and leave it to
|
|
|
|
* the next backend to try again. (Our own relcache entries will be
|
|
|
|
* updated by SI message processing, but we can't be sure whether what we
|
|
|
|
* wrote out was up-to-date.)
|
2002-01-16 18:34:42 +01:00
|
|
|
*
|
2011-08-16 19:11:54 +02:00
|
|
|
* This mustn't run concurrently with the code that unlinks an init file
|
|
|
|
* and sends SI messages, so grab a serialization lock for the duration.
|
2000-03-31 21:39:22 +02:00
|
|
|
*/
|
2002-02-19 21:11:20 +01:00
|
|
|
LWLockAcquire(RelCacheInitLock, LW_EXCLUSIVE);
|
|
|
|
|
|
|
|
/* Make sure we have seen all incoming SI messages */
|
|
|
|
AcceptInvalidationMessages();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we have received any SI relcache invals since backend start, assume
|
|
|
|
* we may have written out-of-date data.
|
|
|
|
*/
|
|
|
|
if (relcacheInvalsReceived == 0L)
|
2002-01-15 23:33:20 +01:00
|
|
|
{
|
|
|
|
/*
|
2002-02-19 21:11:20 +01:00
|
|
|
* OK, rename the temp file to its final name, deleting any
|
|
|
|
* previously-existing init file.
|
2004-12-12 06:07:50 +01:00
|
|
|
*
|
|
|
|
* Note: a failure here is possible under Cygwin, if some other
|
|
|
|
* backend is holding open an unlinked-but-not-yet-gone init file. So
|
|
|
|
* treat this as a noncritical failure; just remove the useless temp
|
|
|
|
* file on failure.
|
2002-01-15 23:33:20 +01:00
|
|
|
*/
|
2004-12-12 06:07:50 +01:00
|
|
|
if (rename(tempfilename, finalfilename) < 0)
|
|
|
|
unlink(tempfilename);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Delete the already-obsolete temp file */
|
2002-01-15 23:33:20 +01:00
|
|
|
unlink(tempfilename);
|
|
|
|
}
|
2004-12-12 06:07:50 +01:00
|
|
|
|
|
|
|
LWLockRelease(RelCacheInitLock);
|
2002-02-19 21:11:20 +01:00
|
|
|
}
|
|
|
|
|
2006-07-04 00:45:41 +02:00
|
|
|
/* write a chunk of data preceded by its length */
|
|
|
|
static void
|
|
|
|
write_item(const void *data, Size len, FILE *fp)
|
|
|
|
{
|
|
|
|
if (fwrite(&len, 1, sizeof(len), fp) != sizeof(len))
|
|
|
|
elog(FATAL, "could not write init file");
|
2022-03-23 21:05:25 +01:00
|
|
|
if (len > 0 && fwrite(data, 1, len, fp) != len)
|
2006-07-04 00:45:41 +02:00
|
|
|
elog(FATAL, "could not write init file");
|
|
|
|
}
|
|
|
|
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
/*
|
|
|
|
* Determine whether a given relation (identified by OID) is one of the ones
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
* we should store in a relcache init file.
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
*
|
|
|
|
* We must cache all nailed rels, and for efficiency we should cache every rel
|
|
|
|
* that supports a syscache. The former set is almost but not quite a subset
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
* of the latter. The special cases are relations where
|
|
|
|
* RelationCacheInitializePhase2/3 chooses to nail for efficiency reasons, but
|
|
|
|
* which do not support any syscache.
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
*/
|
|
|
|
bool
|
|
|
|
RelationIdIsInInitFile(Oid relationId)
|
|
|
|
{
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
if (relationId == SharedSecLabelRelationId ||
|
|
|
|
relationId == TriggerRelidNameIndexId ||
|
|
|
|
relationId == DatabaseNameIndexId ||
|
|
|
|
relationId == SharedSecLabelObjectIndexId)
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
{
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
/*
|
|
|
|
* If this Assert fails, we don't need the applicable special case
|
|
|
|
* anymore.
|
|
|
|
*/
|
Fix the logic for putting relations into the relcache init file.
Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy
of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index
into the relcache init file, because that index is not used by any
syscache. However, we have historically nailed that index into cache for
performance reasons. The upshot was that load_relcache_init_file always
decided that the init file was busted and silently ignored it, resulting
in a significant hit to backend startup speed.
To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around
RelationSupportsSysCache(), which can know about additional relations
that should be in the init file despite being unknown to syscache.c.
Also install some guards against future mistakes of this type: make
write_relcache_init_file Assert that all nailed relations get written to
the init file, and make load_relcache_init_file emit a WARNING if it takes
the "wrong number of nailed relations" exit path. Now that we remove the
init files during postmaster startup, that case should never occur in the
field, even if we are starting a minor-version update that added or removed
rels from the nailed set. So the warning shouldn't ever be seen by end
users, but it will show up in the regression tests if somebody breaks this
logic.
Back-patch to all supported branches, like the previous commit.
2015-06-25 20:39:05 +02:00
|
|
|
Assert(!RelationSupportsSysCache(relationId));
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return RelationSupportsSysCache(relationId);
|
|
|
|
}
|
|
|
|
|
2002-02-19 21:11:20 +01:00
|
|
|
/*
|
|
|
|
* Invalidate (remove) the init file during commit of a transaction that
|
|
|
|
* changed one or more of the relation cache entries that are kept in the
|
2009-08-12 22:53:31 +02:00
|
|
|
* local init file.
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2011-08-16 19:11:54 +02:00
|
|
|
* To be safe against concurrent inspection or rewriting of the init file,
|
|
|
|
* we must take RelCacheInitLock, then remove the old init file, then send
|
|
|
|
* the SI messages that include relcache inval for such relations, and then
|
|
|
|
* release RelCacheInitLock. This serializes the whole affair against
|
|
|
|
* write_relcache_init_file, so that we can be sure that any other process
|
|
|
|
* that's concurrently trying to create a new init file won't move an
|
|
|
|
* already-stale version into place after we unlink. Also, because we unlink
|
|
|
|
* before sending the SI messages, a backend that's currently starting cannot
|
|
|
|
* read the now-obsolete init file and then miss the SI messages that will
|
|
|
|
* force it to update its relcache entries. (This works because the backend
|
|
|
|
* startup sequence gets into the sinval array before trying to load the init
|
|
|
|
* file.)
|
2002-02-19 21:11:20 +01:00
|
|
|
*
|
2011-08-16 19:11:54 +02:00
|
|
|
* We take the lock and do the unlink in RelationCacheInitFilePreInvalidate,
|
|
|
|
* then release the lock in RelationCacheInitFilePostInvalidate. Caller must
|
|
|
|
* send any pending SI messages between those calls.
|
2002-02-19 21:11:20 +01:00
|
|
|
*/
|
|
|
|
void
|
2011-08-16 19:11:54 +02:00
|
|
|
RelationCacheInitFilePreInvalidate(void)
|
2002-02-19 21:11:20 +01:00
|
|
|
{
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
char localinitfname[MAXPGPATH];
|
|
|
|
char sharedinitfname[MAXPGPATH];
|
2002-02-19 21:11:20 +01:00
|
|
|
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
if (DatabasePath)
|
|
|
|
snprintf(localinitfname, sizeof(localinitfname), "%s/%s",
|
|
|
|
DatabasePath, RELCACHE_INIT_FILENAME);
|
|
|
|
snprintf(sharedinitfname, sizeof(sharedinitfname), "global/%s",
|
|
|
|
RELCACHE_INIT_FILENAME);
|
2002-02-19 21:11:20 +01:00
|
|
|
|
2011-08-16 19:11:54 +02:00
|
|
|
LWLockAcquire(RelCacheInitLock, LW_EXCLUSIVE);
|
|
|
|
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
/*
|
|
|
|
* The files might not be there if no backend has been started since the
|
|
|
|
* last removal. But complain about failures other than ENOENT with
|
|
|
|
* ERROR. Fortunately, it's not too late to abort the transaction if we
|
|
|
|
* can't get rid of the would-be-obsolete init file.
|
|
|
|
*/
|
|
|
|
if (DatabasePath)
|
|
|
|
unlink_initfile(localinitfname, ERROR);
|
|
|
|
unlink_initfile(sharedinitfname, ERROR);
|
1996-07-09 08:22:35 +02:00
|
|
|
}
|
2006-11-06 00:40:31 +01:00
|
|
|
|
2011-08-16 19:11:54 +02:00
|
|
|
void
|
|
|
|
RelationCacheInitFilePostInvalidate(void)
|
|
|
|
{
|
|
|
|
LWLockRelease(RelCacheInitLock);
|
|
|
|
}
|
|
|
|
|
2006-11-06 00:40:31 +01:00
|
|
|
/*
|
2009-08-12 22:53:31 +02:00
|
|
|
* Remove the init files during postmaster startup.
|
2006-11-06 00:40:31 +01:00
|
|
|
*
|
2009-08-12 22:53:31 +02:00
|
|
|
* We used to keep the init files across restarts, but that is unsafe in PITR
|
2006-11-06 00:40:31 +01:00
|
|
|
* scenarios, and even in simple crash-recovery cases there are windows for
|
2009-08-12 22:53:31 +02:00
|
|
|
* the init files to become out-of-sync with the database. So now we just
|
|
|
|
* remove them during startup and expect the first backend launch to rebuild
|
|
|
|
* them. Of course, this has to happen in each database of the cluster.
|
2006-11-06 00:40:31 +01:00
|
|
|
*/
|
|
|
|
void
|
2009-08-12 22:53:31 +02:00
|
|
|
RelationCacheInitFileRemove(void)
|
|
|
|
{
|
|
|
|
const char *tblspcdir = "pg_tblspc";
|
|
|
|
DIR *dir;
|
|
|
|
struct dirent *de;
|
2017-04-11 20:13:31 +02:00
|
|
|
char path[MAXPGPATH + 10 + sizeof(TABLESPACE_VERSION_DIRECTORY)];
|
2009-08-12 22:53:31 +02:00
|
|
|
|
|
|
|
snprintf(path, sizeof(path), "global/%s",
|
|
|
|
RELCACHE_INIT_FILENAME);
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
unlink_initfile(path, LOG);
|
2009-08-12 22:53:31 +02:00
|
|
|
|
|
|
|
/* Scan everything in the default tablespace */
|
|
|
|
RelationCacheInitFileRemoveInDir("base");
|
|
|
|
|
|
|
|
/* Scan the tablespace link directory to find non-default tablespaces */
|
|
|
|
dir = AllocateDir(tblspcdir);
|
|
|
|
|
Clean up assorted messiness around AllocateDir() usage.
This patch fixes a couple of low-probability bugs that could lead to
reporting an irrelevant errno value (and hence possibly a wrong SQLSTATE)
concerning directory-open or file-open failures. It also fixes places
where we took shortcuts in reporting such errors, either by using elog
instead of ereport or by using ereport but forgetting to specify an
errcode. And it eliminates a lot of just plain redundant error-handling
code.
In service of all this, export fd.c's formerly-static function
ReadDirExtended, so that external callers can make use of the coding
pattern
dir = AllocateDir(path);
while ((de = ReadDirExtended(dir, path, LOG)) != NULL)
if they'd like to treat directory-open failures as mere LOG conditions
rather than errors. Also fix FreeDir to be a no-op if we reach it
with dir == NULL, as such a coding pattern would cause.
Then, remove code at many call sites that was throwing an error or log
message for AllocateDir failure, as ReadDir or ReadDirExtended can handle
that job just fine. Aside from being a net code savings, this gets rid of
a lot of not-quite-up-to-snuff reports, as mentioned above. (In some
places these changes result in replacing a custom error message such as
"could not open tablespace directory" with more generic wording "could not
open directory", but it was agreed that the custom wording buys little as
long as we report the directory name.) In some other call sites where we
can't just remove code, change the error reports to be fully
project-style-compliant.
Also reorder code in restoreTwoPhaseData that was acquiring a lock
between AllocateDir and ReadDir; in the unlikely but surely not
impossible case that LWLockAcquire changes errno, AllocateDir failures
would be misreported. There is no great value in opening the directory
before acquiring TwoPhaseStateLock, so just do it in the other order.
Also fix CheckXLogRemoved to guarantee that it preserves errno,
as quite a number of call sites are implicitly assuming. (Again,
it's unlikely but I think not impossible that errno could change
during a SpinLockAcquire. If so, this function was broken for its
own purposes as well as breaking callers.)
And change a few places that were using not-per-project-style messages,
such as "could not read directory" when "could not open directory" is
more correct.
Back-patch the exporting of ReadDirExtended, in case we have occasion
to back-patch some fix that makes use of it; it's not needed right now
but surely making it global is pretty harmless. Also back-patch the
restoreTwoPhaseData and CheckXLogRemoved fixes. The rest of this is
essentially cosmetic and need not get back-patched.
Michael Paquier, with a bit of additional work by me
Discussion: https://postgr.es/m/CAB7nPqRpOCxjiirHmebEFhXVTK7V5Jvw4bz82p7Oimtsm3TyZA@mail.gmail.com
2017-12-04 23:02:52 +01:00
|
|
|
while ((de = ReadDirExtended(dir, tblspcdir, LOG)) != NULL)
|
2009-08-12 22:53:31 +02:00
|
|
|
{
|
|
|
|
if (strspn(de->d_name, "0123456789") == strlen(de->d_name))
|
|
|
|
{
|
|
|
|
/* Scan the tablespace dir for per-database dirs */
|
2010-01-12 03:42:52 +01:00
|
|
|
snprintf(path, sizeof(path), "%s/%s/%s",
|
|
|
|
tblspcdir, de->d_name, TABLESPACE_VERSION_DIRECTORY);
|
2009-08-12 22:53:31 +02:00
|
|
|
RelationCacheInitFileRemoveInDir(path);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
FreeDir(dir);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Process one per-tablespace directory for RelationCacheInitFileRemove */
|
|
|
|
static void
|
|
|
|
RelationCacheInitFileRemoveInDir(const char *tblspcpath)
|
2006-11-06 00:40:31 +01:00
|
|
|
{
|
2009-08-12 22:53:31 +02:00
|
|
|
DIR *dir;
|
|
|
|
struct dirent *de;
|
2017-04-11 20:13:31 +02:00
|
|
|
char initfilename[MAXPGPATH * 2];
|
2006-11-06 00:40:31 +01:00
|
|
|
|
2009-08-12 22:53:31 +02:00
|
|
|
/* Scan the tablespace directory to find per-database directories */
|
|
|
|
dir = AllocateDir(tblspcpath);
|
|
|
|
|
Clean up assorted messiness around AllocateDir() usage.
This patch fixes a couple of low-probability bugs that could lead to
reporting an irrelevant errno value (and hence possibly a wrong SQLSTATE)
concerning directory-open or file-open failures. It also fixes places
where we took shortcuts in reporting such errors, either by using elog
instead of ereport or by using ereport but forgetting to specify an
errcode. And it eliminates a lot of just plain redundant error-handling
code.
In service of all this, export fd.c's formerly-static function
ReadDirExtended, so that external callers can make use of the coding
pattern
dir = AllocateDir(path);
while ((de = ReadDirExtended(dir, path, LOG)) != NULL)
if they'd like to treat directory-open failures as mere LOG conditions
rather than errors. Also fix FreeDir to be a no-op if we reach it
with dir == NULL, as such a coding pattern would cause.
Then, remove code at many call sites that was throwing an error or log
message for AllocateDir failure, as ReadDir or ReadDirExtended can handle
that job just fine. Aside from being a net code savings, this gets rid of
a lot of not-quite-up-to-snuff reports, as mentioned above. (In some
places these changes result in replacing a custom error message such as
"could not open tablespace directory" with more generic wording "could not
open directory", but it was agreed that the custom wording buys little as
long as we report the directory name.) In some other call sites where we
can't just remove code, change the error reports to be fully
project-style-compliant.
Also reorder code in restoreTwoPhaseData that was acquiring a lock
between AllocateDir and ReadDir; in the unlikely but surely not
impossible case that LWLockAcquire changes errno, AllocateDir failures
would be misreported. There is no great value in opening the directory
before acquiring TwoPhaseStateLock, so just do it in the other order.
Also fix CheckXLogRemoved to guarantee that it preserves errno,
as quite a number of call sites are implicitly assuming. (Again,
it's unlikely but I think not impossible that errno could change
during a SpinLockAcquire. If so, this function was broken for its
own purposes as well as breaking callers.)
And change a few places that were using not-per-project-style messages,
such as "could not read directory" when "could not open directory" is
more correct.
Back-patch the exporting of ReadDirExtended, in case we have occasion
to back-patch some fix that makes use of it; it's not needed right now
but surely making it global is pretty harmless. Also back-patch the
restoreTwoPhaseData and CheckXLogRemoved fixes. The rest of this is
essentially cosmetic and need not get back-patched.
Michael Paquier, with a bit of additional work by me
Discussion: https://postgr.es/m/CAB7nPqRpOCxjiirHmebEFhXVTK7V5Jvw4bz82p7Oimtsm3TyZA@mail.gmail.com
2017-12-04 23:02:52 +01:00
|
|
|
while ((de = ReadDirExtended(dir, tblspcpath, LOG)) != NULL)
|
2009-08-12 22:53:31 +02:00
|
|
|
{
|
|
|
|
if (strspn(de->d_name, "0123456789") == strlen(de->d_name))
|
|
|
|
{
|
|
|
|
/* Try to remove the init file in each database */
|
|
|
|
snprintf(initfilename, sizeof(initfilename), "%s/%s/%s",
|
|
|
|
tblspcpath, de->d_name, RELCACHE_INIT_FILENAME);
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
unlink_initfile(initfilename, LOG);
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
FreeDir(dir);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
unlink_initfile(const char *initfilename, int elevel)
|
2009-08-12 22:53:31 +02:00
|
|
|
{
|
|
|
|
if (unlink(initfilename) < 0)
|
|
|
|
{
|
|
|
|
/* It might not be there, but log any error other than ENOENT */
|
|
|
|
if (errno != ENOENT)
|
Fix bugs in vacuum of shared rels, by keeping their relcache entries current.
When vacuum processes a relation it uses the corresponding relcache
entry's relfrozenxid / relminmxid as a cutoff for when to remove
tuples etc. Unfortunately for nailed relations (i.e. critical system
catalogs) bugs could frequently lead to the corresponding relcache
entry being stale.
This set of bugs could cause actual data corruption as vacuum would
potentially not remove the correct row versions, potentially reviving
them at a later point. After 699bf7d05c some corruptions in this vein
were prevented, but the additional error checks could also trigger
spuriously. Examples of such errors are:
ERROR: found xmin ... from before relfrozenxid ...
and
ERROR: found multixact ... from before relminmxid ...
To be caused by this bug the errors have to occur on system catalog
tables.
The two bugs are:
1) Invalidations for nailed relations were ignored, based on the
theory that the relcache entry for such tables doesn't
change. Which is largely true, except for fields like relfrozenxid
etc. This means that changes to relations vacuumed in other
sessions weren't picked up by already existing sessions. Luckily
autovacuum doesn't have particularly longrunning sessions.
2) For shared *and* nailed relations, the shared relcache init file
was never invalidated while running. That means that for such
tables (e.g. pg_authid, pg_database) it's not just already existing
sessions that are affected, but even new connections are as well.
That explains why the reports usually were about pg_authid et. al.
To fix 1), revalidate the rd_rel portion of a relcache entry when
invalid. This implies a bit of extra complexity to deal with
bootstrapping, but it's not too bad. The fix for 2) is simpler,
simply always remove both the shared and local init files.
Author: Andres Freund
Reviewed-By: Alvaro Herrera
Discussion:
https://postgr.es/m/20180525203736.crkbg36muzxrjj5e@alap3.anarazel.de
https://postgr.es/m/CAMa1XUhKSJd98JW4o9StWPrfS=11bPgG+_GDMxe25TvUY4Sugg@mail.gmail.com
https://postgr.es/m/CAKMFJucqbuoDRfxPDX39WhA3vJyxweRg_zDVXzncr6+5wOguWA@mail.gmail.com
https://postgr.es/m/CAGewt-ujGpMLQ09gXcUFMZaZsGJC98VXHEFbF-tpPB0fB13K+A@mail.gmail.com
Backpatch: 9.3-
2018-06-12 20:13:21 +02:00
|
|
|
ereport(elevel,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not remove cache file \"%s\": %m",
|
|
|
|
initfilename)));
|
2009-08-12 22:53:31 +02:00
|
|
|
}
|
2006-11-06 00:40:31 +01:00
|
|
|
}
|
Make ResourceOwners more easily extensible.
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify
the ResourceOwnerData struct. In particular, this makes it possible
for extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind,
and if it fills up, switch to a hash table. The new approach also uses
an array and a hash, but now the array and the hash are used at the
same time. The array is used to hold the recently added resources, and
when it fills up, they are moved to the hash. This keeps the access to
recent entries fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(),
ResourceOwnerRemember*(), and ResourceOwnerForget*() functions have
been replaced with three generic functions that take resource kind as
argument. For convenience, we still define resource-specific wrapper
macros around the generic functions with the old names, but they are
now defined in the source files that use those resource kinds.
The release callback no longer needs to call ResourceOwnerForget on
the resource being released. ResourceOwnerRelease unregisters the
resource from the owner before calling the callback. That needed some
changes in bufmgr.c and some other files, where releasing the
resources previously always called ResourceOwnerForget.
Each resource kind specifies a release priority, and
ResourceOwnerReleaseAll releases the resources in priority order. To
make that possible, we have to restrict what you can do between
phases. After calling ResourceOwnerRelease(), you are no longer
allowed to remember any more resources in it or to forget any
previously remembered resources by calling ResourceOwnerForget. There
was one case where that was done previously. At subtransaction commit,
AtEOSubXact_Inval() would handle the invalidation messages and call
RelationFlushRelation(), which temporarily increased the reference
count on the relation being flushed. We now switch to the parent
subtransaction's resource owner before calling AtEOSubXact_Inval(), so
that there is a valid ResourceOwner to temporarily hold that relcache
reference.
Other end-of-xact routines make similar calls to AtEOXact_Inval()
between release phases, but I didn't see any regression test failures
from those, so I'm not sure if they could reach a codepath that needs
remembering extra resources.
There were two exceptions to how the resource leak WARNINGs on commit
were printed previously: llvmjit silently released the context without
printing the warning, and a leaked buffer io triggered a PANIC. Now
everything prints a WARNING, including those cases.
Add tests in src/test/modules/test_resowner.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud
Reviewed-by: Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
2023-11-08 12:30:50 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* ResourceOwner callbacks
|
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
ResOwnerPrintRelCache(Datum res)
|
|
|
|
{
|
|
|
|
Relation rel = (Relation) DatumGetPointer(res);
|
|
|
|
|
|
|
|
return psprintf("relation \"%s\"", RelationGetRelationName(rel));
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
ResOwnerReleaseRelation(Datum res)
|
|
|
|
{
|
|
|
|
Relation rel = (Relation) DatumGetPointer(res);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This reference has already been removed from the resource owner, so
|
|
|
|
* just decrement reference count without calling
|
|
|
|
* ResourceOwnerForgetRelationRef.
|
|
|
|
*/
|
|
|
|
Assert(rel->rd_refcnt > 0);
|
|
|
|
rel->rd_refcnt -= 1;
|
|
|
|
|
|
|
|
RelationCloseCleanup((Relation) res);
|
|
|
|
}
|