2001-07-14 00:55:59 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* vacuumlazy.c
|
|
|
|
* Concurrent ("lazy") vacuuming.
|
|
|
|
*
|
|
|
|
*
|
|
|
|
* The major space usage for LAZY VACUUM is storage for the array of dead
|
|
|
|
* tuple TIDs, with the next biggest need being storage for per-disk-page
|
|
|
|
* free space info. We want to ensure we can vacuum even the very largest
|
|
|
|
* relations with finite memory space usage. To do that, we set upper bounds
|
|
|
|
* on the number of tuples and pages we will keep track of at once.
|
|
|
|
*
|
2013-12-12 12:42:39 +01:00
|
|
|
* We are willing to use at most maintenance_work_mem (or perhaps
|
|
|
|
* autovacuum_work_mem) memory space to keep track of dead tuples. We
|
|
|
|
* initially allocate an array of TIDs of that size, with an upper limit that
|
|
|
|
* depends on table size (this limit ensures we don't allocate a huge area
|
|
|
|
* uselessly for vacuuming small tables). If the array threatens to overflow,
|
|
|
|
* we suspend the heap scan phase and perform a pass of index cleanup and page
|
|
|
|
* compaction, then resume the heap scan with an empty TID array.
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
2006-09-13 19:47:08 +02:00
|
|
|
* If we're processing a table with no indexes, we can just vacuum each page
|
|
|
|
* as we go; there's no need to save up multiple tuples to minimize the number
|
|
|
|
* of index scans performed. So we don't use maintenance_work_mem memory for
|
|
|
|
* the TID array, just enough to hold as many heap tuples as fit on one page.
|
|
|
|
*
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
2014-01-07 22:05:30 +01:00
|
|
|
* Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
|
2001-07-14 00:55:59 +02:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
|
|
|
*
|
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/backend/commands/vacuumlazy.c
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
#include "postgres.h"
|
|
|
|
|
2004-12-01 20:00:56 +01:00
|
|
|
#include <math.h>
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
#include "access/genam.h"
|
|
|
|
#include "access/heapam.h"
|
2012-08-29 01:02:00 +02:00
|
|
|
#include "access/heapam_xlog.h"
|
2012-08-30 22:15:44 +02:00
|
|
|
#include "access/htup_details.h"
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
#include "access/multixact.h"
|
2006-07-13 18:49:20 +02:00
|
|
|
#include "access/transam.h"
|
2008-12-03 14:05:22 +01:00
|
|
|
#include "access/visibilitymap.h"
|
2014-11-06 12:52:08 +01:00
|
|
|
#include "access/xlog.h"
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
#include "catalog/catalog.h"
|
2008-11-19 11:34:52 +01:00
|
|
|
#include "catalog/storage.h"
|
2007-04-18 18:44:18 +02:00
|
|
|
#include "commands/dbcommands.h"
|
2001-07-14 00:55:59 +02:00
|
|
|
#include "commands/vacuum.h"
|
|
|
|
#include "miscadmin.h"
|
2005-07-14 07:13:45 +02:00
|
|
|
#include "pgstat.h"
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
#include "portability/instr_time.h"
|
2007-04-18 18:44:18 +02:00
|
|
|
#include "postmaster/autovacuum.h"
|
2008-05-12 02:00:54 +02:00
|
|
|
#include "storage/bufmgr.h"
|
2001-07-14 00:55:59 +02:00
|
|
|
#include "storage/freespace.h"
|
2008-05-12 02:00:54 +02:00
|
|
|
#include "storage/lmgr.h"
|
2002-04-02 03:03:07 +02:00
|
|
|
#include "utils/lsyscache.h"
|
2006-03-04 20:09:09 +01:00
|
|
|
#include "utils/memutils.h"
|
2005-10-04 00:52:26 +02:00
|
|
|
#include "utils/pg_rusage.h"
|
2011-09-09 19:23:41 +02:00
|
|
|
#include "utils/timestamp.h"
|
2008-03-26 22:10:39 +01:00
|
|
|
#include "utils/tqual.h"
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Space/time tradeoff parameters: do these need to be user-tunable?
|
|
|
|
*
|
|
|
|
* To consider truncating the relation, we want there to be at least
|
2003-03-04 22:51:22 +01:00
|
|
|
* REL_TRUNCATE_MINIMUM or (relsize / REL_TRUNCATE_FRACTION) (whichever
|
|
|
|
* is less) potentially-freeable pages.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
2003-03-04 22:51:22 +01:00
|
|
|
#define REL_TRUNCATE_MINIMUM 1000
|
2001-07-14 00:55:59 +02:00
|
|
|
#define REL_TRUNCATE_FRACTION 16
|
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
/*
|
|
|
|
* Timing parameters for truncate locking heuristics.
|
|
|
|
*
|
|
|
|
* These were not exposed as user tunable GUC values because it didn't seem
|
|
|
|
* that the potential for improvement was great enough to merit the cost of
|
|
|
|
* supporting them.
|
|
|
|
*/
|
2013-05-29 22:58:43 +02:00
|
|
|
#define VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL 20 /* ms */
|
|
|
|
#define VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL 50 /* ms */
|
|
|
|
#define VACUUM_TRUNCATE_LOCK_TIMEOUT 5000 /* ms */
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
|
2007-09-24 05:52:55 +02:00
|
|
|
/*
|
|
|
|
* Guesstimation of number of dead tuples per page. This is used to
|
|
|
|
* provide an upper limit to memory allocated when vacuuming small
|
|
|
|
* tables.
|
|
|
|
*/
|
2007-09-26 22:16:28 +02:00
|
|
|
#define LAZY_ALLOC_TUPLES MaxHeapTuplesPerPage
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2009-01-22 20:25:00 +01:00
|
|
|
/*
|
|
|
|
* Before we consider skipping a page that's marked as clean in
|
|
|
|
* visibility map, we must've seen at least this many clean pages.
|
|
|
|
*/
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
#define SKIP_PAGES_THRESHOLD ((BlockNumber) 32)
|
2009-01-22 20:25:00 +01:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
typedef struct LVRelStats
|
|
|
|
{
|
2006-09-13 19:47:08 +02:00
|
|
|
/* hasindex = true means two-pass strategy; false means one-pass */
|
|
|
|
bool hasindex;
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Overall statistics about rel */
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
BlockNumber old_rel_pages; /* previous value of pg_class.relpages */
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
BlockNumber rel_pages; /* total number of pages */
|
|
|
|
BlockNumber scanned_pages; /* number of pages we examined */
|
2011-06-09 20:32:50 +02:00
|
|
|
double scanned_tuples; /* counts only tuples on scanned pages */
|
2009-06-11 16:49:15 +02:00
|
|
|
double old_rel_tuples; /* previous value of pg_class.reltuples */
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
double new_rel_tuples; /* new estimated total # of tuples */
|
2014-01-19 01:24:20 +01:00
|
|
|
double new_dead_tuples; /* new estimated total # of dead tuples */
|
2005-10-15 04:49:52 +02:00
|
|
|
BlockNumber pages_removed;
|
2004-12-01 20:00:56 +01:00
|
|
|
double tuples_deleted;
|
2001-10-25 07:50:21 +02:00
|
|
|
BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
|
2001-07-14 00:55:59 +02:00
|
|
|
/* List of TIDs of tuples we intend to delete */
|
|
|
|
/* NB: this list is ordered by TID address */
|
2001-10-28 07:26:15 +01:00
|
|
|
int num_dead_tuples; /* current # of entries */
|
|
|
|
int max_dead_tuples; /* # slots allocated in array */
|
2001-10-25 07:50:21 +02:00
|
|
|
ItemPointer dead_tuples; /* array of ItemPointerData */
|
2007-04-18 18:44:18 +02:00
|
|
|
int num_index_scans;
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
TransactionId latestRemovedXid;
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
bool lock_waiter_detected;
|
2001-07-14 00:55:59 +02:00
|
|
|
} LVRelStats;
|
|
|
|
|
|
|
|
|
2007-05-30 22:12:03 +02:00
|
|
|
/* A few variables that don't seem worth passing around as parameters */
|
2002-09-04 22:31:48 +02:00
|
|
|
static int elevel = -1;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
static TransactionId OldestXmin;
|
|
|
|
static TransactionId FreezeLimit;
|
2013-09-16 20:45:00 +02:00
|
|
|
static MultiXactId MultiXactCutoff;
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
|
2007-05-30 22:12:03 +02:00
|
|
|
static BufferAccessStrategy vac_strategy;
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/* non-export function prototypes */
|
|
|
|
static void lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
|
2008-12-03 14:05:22 +01:00
|
|
|
Relation *Irel, int nindexes, bool scan_all);
|
2001-07-14 00:55:59 +02:00
|
|
|
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
|
2011-11-08 03:39:40 +01:00
|
|
|
static bool lazy_check_needs_freeze(Buffer buf);
|
2004-12-01 20:00:56 +01:00
|
|
|
static void lazy_vacuum_index(Relation indrel,
|
2006-10-04 02:30:14 +02:00
|
|
|
IndexBulkDeleteResult **stats,
|
|
|
|
LVRelStats *vacrelstats);
|
2006-05-03 00:25:10 +02:00
|
|
|
static void lazy_cleanup_index(Relation indrel,
|
2006-10-04 02:30:14 +02:00
|
|
|
IndexBulkDeleteResult *stats,
|
|
|
|
LVRelStats *vacrelstats);
|
2001-10-25 07:50:21 +02:00
|
|
|
static int lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
|
2013-02-13 16:46:23 +01:00
|
|
|
int tupindex, LVRelStats *vacrelstats, Buffer *vmbuffer);
|
2010-02-09 22:43:30 +01:00
|
|
|
static void lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats);
|
2001-07-14 00:55:59 +02:00
|
|
|
static BlockNumber count_nondeletable_pages(Relation onerel,
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
LVRelStats *vacrelstats);
|
2006-09-13 19:47:08 +02:00
|
|
|
static void lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks);
|
2001-07-14 00:55:59 +02:00
|
|
|
static void lazy_record_dead_tuple(LVRelStats *vacrelstats,
|
2001-10-25 07:50:21 +02:00
|
|
|
ItemPointer itemptr);
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
static bool lazy_tid_reaped(ItemPointer itemptr, void *state);
|
2001-07-14 00:55:59 +02:00
|
|
|
static int vac_cmp_itemptr(const void *left, const void *right);
|
2013-07-22 19:26:33 +02:00
|
|
|
static bool heap_page_is_all_visible(Relation rel, Buffer buf,
|
2013-02-13 16:46:23 +01:00
|
|
|
TransactionId *visibility_cutoff_xid);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_vacuum_rel() -- perform LAZY VACUUM for one heap relation
|
|
|
|
*
|
|
|
|
* This routine vacuums a single heap, cleans out its indexes, and
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
* updates its relpages and reltuples statistics.
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
|
|
|
* At entry, we have already established a transaction and opened
|
|
|
|
* and locked the relation.
|
|
|
|
*/
|
2010-02-09 22:43:30 +01:00
|
|
|
void
|
2007-05-30 22:12:03 +02:00
|
|
|
lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
BufferAccessStrategy bstrategy)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
LVRelStats *vacrelstats;
|
|
|
|
Relation *Irel;
|
|
|
|
int nindexes;
|
2001-10-25 07:50:21 +02:00
|
|
|
BlockNumber possibly_freeable;
|
2007-04-18 18:44:18 +02:00
|
|
|
PGRUsage ru0;
|
2007-11-15 22:14:46 +01:00
|
|
|
TimestampTz starttime = 0;
|
2012-06-10 21:20:04 +02:00
|
|
|
long secs;
|
|
|
|
int usecs;
|
|
|
|
double read_rate,
|
2011-11-25 16:10:46 +01:00
|
|
|
write_rate;
|
2013-11-27 12:10:16 +01:00
|
|
|
bool scan_all; /* should we scan all pages? */
|
|
|
|
bool scanned_all; /* did we actually scan all pages? */
|
2013-11-28 20:52:54 +01:00
|
|
|
TransactionId xidFullScanLimit;
|
|
|
|
MultiXactId mxactFullScanLimit;
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
BlockNumber new_rel_pages;
|
|
|
|
double new_rel_tuples;
|
2011-10-14 23:23:01 +02:00
|
|
|
BlockNumber new_rel_allvisible;
|
2014-01-19 01:24:20 +01:00
|
|
|
double new_live_tuples;
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
TransactionId new_frozen_xid;
|
2013-05-29 22:58:43 +02:00
|
|
|
MultiXactId new_min_multi;
|
2007-04-18 18:44:18 +02:00
|
|
|
|
|
|
|
/* measure elapsed time iff autovacuum logging requires it */
|
2011-08-18 15:55:04 +02:00
|
|
|
if (IsAutoVacuumWorkerProcess() && Log_autovacuum_min_duration >= 0)
|
|
|
|
{
|
|
|
|
pg_rusage_init(&ru0);
|
2011-11-25 16:10:46 +01:00
|
|
|
starttime = GetCurrentTimestamp();
|
2011-08-18 15:55:04 +02:00
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2009-11-16 22:32:07 +01:00
|
|
|
if (vacstmt->options & VACOPT_VERBOSE)
|
Commit to match discussed elog() changes. Only update is that LOG is
now just below FATAL in server_min_messages. Added more text to
highlight ordering difference between it and client_min_messages.
---------------------------------------------------------------------------
REALLYFATAL => PANIC
STOP => PANIC
New INFO level the prints to client by default
New LOG level the prints to server log by default
Cause VACUUM information to print only to the client
NOTICE => INFO where purely information messages are sent
DEBUG => LOG for purely server status messages
DEBUG removed, kept as backward compatible
DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1 added
DebugLvl removed in favor of new DEBUG[1-5] symbols
New server_min_messages GUC parameter with values:
DEBUG[5-1], INFO, NOTICE, ERROR, LOG, FATAL, PANIC
New client_min_messages GUC parameter with values:
DEBUG[5-1], LOG, INFO, NOTICE, ERROR, FATAL, PANIC
Server startup now logged with LOG instead of DEBUG
Remove debug_level GUC parameter
elog() numbers now start at 10
Add test to print error message if older elog() values are passed to elog()
Bootstrap mode now has a -d that requires an argument, like postmaster
2002-03-02 22:39:36 +01:00
|
|
|
elevel = INFO;
|
2001-07-14 00:55:59 +02:00
|
|
|
else
|
2003-05-27 19:49:47 +02:00
|
|
|
elevel = DEBUG2;
|
2002-03-06 07:10:59 +01:00
|
|
|
|
2007-05-30 22:12:03 +02:00
|
|
|
vac_strategy = bstrategy;
|
|
|
|
|
Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables. The output format is controlled by a
so-called "output plugin"; an example is included. To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 22:32:18 +01:00
|
|
|
vacuum_set_xid_limits(onerel,
|
|
|
|
vacstmt->freeze_min_age, vacstmt->freeze_table_age,
|
Separate multixact freezing parameters from xid's
Previously we were piggybacking on transaction ID parameters to freeze
multixacts; but since there isn't necessarily any relationship between
rates of Xid and multixact consumption, this turns out not to be a good
idea.
Therefore, we now have multixact-specific freezing parameters:
vacuum_multixact_freeze_min_age: when to remove multis as we come across
them in vacuum (default to 5 million, i.e. early in comparison to Xid's
default of 50 million)
vacuum_multixact_freeze_table_age: when to force whole-table scans
instead of scanning only the pages marked as not all visible in
visibility map (default to 150 million, same as for Xids). Whichever of
both which reaches the 150 million mark earlier will cause a whole-table
scan.
autovacuum_multixact_freeze_max_age: when for cause emergency,
uninterruptible whole-table scans (default to 400 million, double as
that for Xids). This means there shouldn't be more frequent emergency
vacuuming than previously, unless multixacts are being used very
rapidly.
Backpatch to 9.3 where multixacts were made to persist enough to require
freezing. To avoid an ABI break in 9.3, VacuumStmt has a couple of
fields in an unnatural place, and StdRdOptions is split in two so that
the newly added fields can go at the end.
Patch by me, reviewed by Robert Haas, with additional input from Andres
Freund and Tom Lane.
2014-02-13 23:30:30 +01:00
|
|
|
vacstmt->multixact_freeze_min_age,
|
|
|
|
vacstmt->multixact_freeze_table_age,
|
2013-11-28 20:52:54 +01:00
|
|
|
&OldestXmin, &FreezeLimit, &xidFullScanLimit,
|
|
|
|
&MultiXactCutoff, &mxactFullScanLimit);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We request a full scan if either the table's frozen Xid is now older
|
|
|
|
* than or equal to the requested Xid full-table scan limit; or if the
|
Separate multixact freezing parameters from xid's
Previously we were piggybacking on transaction ID parameters to freeze
multixacts; but since there isn't necessarily any relationship between
rates of Xid and multixact consumption, this turns out not to be a good
idea.
Therefore, we now have multixact-specific freezing parameters:
vacuum_multixact_freeze_min_age: when to remove multis as we come across
them in vacuum (default to 5 million, i.e. early in comparison to Xid's
default of 50 million)
vacuum_multixact_freeze_table_age: when to force whole-table scans
instead of scanning only the pages marked as not all visible in
visibility map (default to 150 million, same as for Xids). Whichever of
both which reaches the 150 million mark earlier will cause a whole-table
scan.
autovacuum_multixact_freeze_max_age: when for cause emergency,
uninterruptible whole-table scans (default to 400 million, double as
that for Xids). This means there shouldn't be more frequent emergency
vacuuming than previously, unless multixacts are being used very
rapidly.
Backpatch to 9.3 where multixacts were made to persist enough to require
freezing. To avoid an ABI break in 9.3, VacuumStmt has a couple of
fields in an unnatural place, and StdRdOptions is split in two so that
the newly added fields can go at the end.
Patch by me, reviewed by Robert Haas, with additional input from Andres
Freund and Tom Lane.
2014-02-13 23:30:30 +01:00
|
|
|
* table's minimum MultiXactId is older than or equal to the requested
|
|
|
|
* mxid full-table scan limit.
|
2013-11-28 20:52:54 +01:00
|
|
|
*/
|
2009-01-16 14:27:24 +01:00
|
|
|
scan_all = TransactionIdPrecedesOrEquals(onerel->rd_rel->relfrozenxid,
|
2013-11-28 20:52:54 +01:00
|
|
|
xidFullScanLimit);
|
|
|
|
scan_all |= MultiXactIdPrecedesOrEquals(onerel->rd_rel->relminmxid,
|
|
|
|
mxactFullScanLimit);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2002-11-13 01:39:48 +01:00
|
|
|
vacrelstats = (LVRelStats *) palloc0(sizeof(LVRelStats));
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
vacrelstats->old_rel_pages = onerel->rd_rel->relpages;
|
2009-06-07 00:13:52 +02:00
|
|
|
vacrelstats->old_rel_tuples = onerel->rd_rel->reltuples;
|
|
|
|
vacrelstats->num_index_scans = 0;
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
vacrelstats->pages_removed = 0;
|
|
|
|
vacrelstats->lock_waiter_detected = false;
|
2007-04-18 18:44:18 +02:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Open all indexes of the relation */
|
2006-07-31 22:09:10 +02:00
|
|
|
vac_open_indexes(onerel, RowExclusiveLock, &nindexes, &Irel);
|
2006-09-13 19:47:08 +02:00
|
|
|
vacrelstats->hasindex = (nindexes > 0);
|
2009-06-11 16:49:15 +02:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Do the vacuuming */
|
2009-01-16 14:27:24 +01:00
|
|
|
lazy_scan_heap(onerel, vacrelstats, Irel, nindexes, scan_all);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/* Done with indexes */
|
2004-10-01 01:21:26 +02:00
|
|
|
vac_close_indexes(nindexes, Irel, NoLock);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2013-11-27 12:10:16 +01:00
|
|
|
/*
|
|
|
|
* Compute whether we actually scanned the whole relation. If we did, we
|
|
|
|
* can adjust relfrozenxid and relminmxid.
|
|
|
|
*
|
|
|
|
* NB: We need to check this before truncating the relation, because that
|
|
|
|
* will change ->rel_pages.
|
|
|
|
*/
|
|
|
|
if (vacrelstats->scanned_pages < vacrelstats->rel_pages)
|
|
|
|
{
|
|
|
|
Assert(!scan_all);
|
|
|
|
scanned_all = false;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
scanned_all = true;
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/*
|
|
|
|
* Optionally truncate the relation.
|
|
|
|
*
|
2001-10-25 07:50:21 +02:00
|
|
|
* Don't even think about it unless we have a shot at releasing a goodly
|
|
|
|
* number of pages. Otherwise, the time taken isn't worth it.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
possibly_freeable = vacrelstats->rel_pages - vacrelstats->nonempty_pages;
|
2009-01-06 15:55:37 +01:00
|
|
|
if (possibly_freeable > 0 &&
|
|
|
|
(possibly_freeable >= REL_TRUNCATE_MINIMUM ||
|
|
|
|
possibly_freeable >= vacrelstats->rel_pages / REL_TRUNCATE_FRACTION))
|
2010-02-09 22:43:30 +01:00
|
|
|
lazy_truncate_heap(onerel, vacrelstats);
|
|
|
|
|
|
|
|
/* Vacuum the Free Space Map */
|
|
|
|
FreeSpaceMapVacuum(onerel);
|
2007-02-21 23:15:21 +01:00
|
|
|
|
2008-12-03 14:05:22 +01:00
|
|
|
/*
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
* Update statistics in pg_class.
|
|
|
|
*
|
|
|
|
* A corner case here is that if we scanned no pages at all because every
|
|
|
|
* page is all-visible, we should not update relpages/reltuples, because
|
2012-06-10 21:20:04 +02:00
|
|
|
* we have no new information to contribute. In particular this keeps us
|
|
|
|
* from replacing relpages=reltuples=0 (which means "unknown tuple
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
* density") with nonzero relpages and reltuples=0 (which means "zero
|
|
|
|
* tuple density") unless there's some actual evidence for the latter.
|
|
|
|
*
|
2012-06-10 21:20:04 +02:00
|
|
|
* We do update relallvisible even in the corner case, since if the table
|
|
|
|
* is all-visible we'd definitely like to know that. But clamp the value
|
|
|
|
* to be not more than what we're setting relpages to.
|
2011-10-14 23:23:01 +02:00
|
|
|
*
|
2013-11-27 12:10:16 +01:00
|
|
|
* Also, don't change relfrozenxid/relminmxid if we skipped any pages,
|
|
|
|
* since then we don't know for certain that all tuples have a newer xmin.
|
2008-12-03 14:05:22 +01:00
|
|
|
*/
|
Fix a missed case in code for "moving average" estimate of reltuples.
It is possible for VACUUM to scan no pages at all, if the visibility map
shows that all pages are all-visible. In this situation VACUUM has no new
information to report about the relation's tuple density, so it wasn't
changing pg_class.reltuples ... but it updated pg_class.relpages anyway.
That's wrong in general, since there is no evidence to justify changing the
density ratio reltuples/relpages, but it's particularly bad if the previous
state was relpages=reltuples=0, which means "unknown tuple density".
We just replaced "unknown" with "zero". ANALYZE would eventually recover
from this, but it could take a lot of repetitions of ANALYZE to do so if
the relation size is much larger than the maximum number of pages ANALYZE
will scan, because of the moving-average behavior introduced by commit
b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8.
The only known situation where we could have relpages=reltuples=0 and yet
the visibility map asserts everything's visible is immediately following
a pg_upgrade. It might be advisable for pg_upgrade to try to preserve the
relpages/reltuples statistics; but in any case this code is wrong on its
own terms, so fix it. Per report from Sergey Koposov.
Back-patch to 8.4, where the visibility map was introduced, same as the
previous change.
2011-08-30 20:49:45 +02:00
|
|
|
new_rel_pages = vacrelstats->rel_pages;
|
|
|
|
new_rel_tuples = vacrelstats->new_rel_tuples;
|
|
|
|
if (vacrelstats->scanned_pages == 0 && new_rel_pages > 0)
|
|
|
|
{
|
|
|
|
new_rel_pages = vacrelstats->old_rel_pages;
|
|
|
|
new_rel_tuples = vacrelstats->old_rel_tuples;
|
|
|
|
}
|
|
|
|
|
2011-10-14 23:23:01 +02:00
|
|
|
new_rel_allvisible = visibilitymap_count(onerel);
|
|
|
|
if (new_rel_allvisible > new_rel_pages)
|
|
|
|
new_rel_allvisible = new_rel_pages;
|
|
|
|
|
2013-11-27 12:10:16 +01:00
|
|
|
new_frozen_xid = scanned_all ? FreezeLimit : InvalidTransactionId;
|
|
|
|
new_min_multi = scanned_all ? MultiXactCutoff : InvalidMultiXactId;
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
vac_update_relstats(onerel,
|
2011-10-14 23:23:01 +02:00
|
|
|
new_rel_pages,
|
|
|
|
new_rel_tuples,
|
|
|
|
new_rel_allvisible,
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
vacrelstats->hasindex,
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
new_frozen_xid,
|
2014-10-30 18:03:22 +01:00
|
|
|
new_min_multi,
|
|
|
|
false);
|
2005-07-14 07:13:45 +02:00
|
|
|
|
2013-04-29 20:05:26 +02:00
|
|
|
/* report results to the stats collector, too */
|
2014-01-19 01:24:20 +01:00
|
|
|
new_live_tuples = new_rel_tuples - vacrelstats->new_dead_tuples;
|
|
|
|
if (new_live_tuples < 0)
|
|
|
|
new_live_tuples = 0; /* just in case */
|
|
|
|
|
2013-04-29 20:05:26 +02:00
|
|
|
pgstat_report_vacuum(RelationGetRelid(onerel),
|
2013-05-29 22:58:43 +02:00
|
|
|
onerel->rd_rel->relisshared,
|
2014-01-19 01:24:20 +01:00
|
|
|
new_live_tuples,
|
|
|
|
vacrelstats->new_dead_tuples);
|
2007-04-18 18:44:18 +02:00
|
|
|
|
|
|
|
/* and log the action if appropriate */
|
2007-09-24 05:12:23 +02:00
|
|
|
if (IsAutoVacuumWorkerProcess() && Log_autovacuum_min_duration >= 0)
|
2007-04-18 18:44:18 +02:00
|
|
|
{
|
2012-06-10 21:20:04 +02:00
|
|
|
TimestampTz endtime = GetCurrentTimestamp();
|
2011-11-25 16:10:46 +01:00
|
|
|
|
2007-09-24 05:12:23 +02:00
|
|
|
if (Log_autovacuum_min_duration == 0 ||
|
2011-11-25 16:10:46 +01:00
|
|
|
TimestampDifferenceExceeds(starttime, endtime,
|
2007-09-24 05:12:23 +02:00
|
|
|
Log_autovacuum_min_duration))
|
2011-11-25 16:10:46 +01:00
|
|
|
{
|
|
|
|
TimestampDifference(starttime, endtime, &secs, &usecs);
|
|
|
|
|
|
|
|
read_rate = 0;
|
|
|
|
write_rate = 0;
|
|
|
|
if ((secs > 0) || (usecs > 0))
|
|
|
|
{
|
2012-06-10 21:20:04 +02:00
|
|
|
read_rate = (double) BLCKSZ *VacuumPageMiss / (1024 * 1024) /
|
|
|
|
(secs + usecs / 1000000.0);
|
|
|
|
write_rate = (double) BLCKSZ *VacuumPageDirty / (1024 * 1024) /
|
|
|
|
(secs + usecs / 1000000.0);
|
2011-11-25 16:10:46 +01:00
|
|
|
}
|
2007-04-18 18:44:18 +02:00
|
|
|
ereport(LOG,
|
|
|
|
(errmsg("automatic vacuum of table \"%s.%s.%s\": index scans: %d\n"
|
|
|
|
"pages: %d removed, %d remain\n"
|
2014-01-19 01:24:20 +01:00
|
|
|
"tuples: %.0f removed, %.0f remain, %.0f are dead but not yet removable\n"
|
2011-11-25 16:10:46 +01:00
|
|
|
"buffer usage: %d hits, %d misses, %d dirtied\n"
|
2013-05-29 22:58:43 +02:00
|
|
|
"avg read rate: %.3f MB/s, avg write rate: %.3f MB/s\n"
|
2007-04-18 18:44:18 +02:00
|
|
|
"system usage: %s",
|
|
|
|
get_database_name(MyDatabaseId),
|
|
|
|
get_namespace_name(RelationGetNamespace(onerel)),
|
|
|
|
RelationGetRelationName(onerel),
|
|
|
|
vacrelstats->num_index_scans,
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
vacrelstats->pages_removed,
|
|
|
|
vacrelstats->rel_pages,
|
|
|
|
vacrelstats->tuples_deleted,
|
2011-11-25 16:10:46 +01:00
|
|
|
vacrelstats->new_rel_tuples,
|
2014-01-19 01:24:20 +01:00
|
|
|
vacrelstats->new_dead_tuples,
|
2011-11-25 16:10:46 +01:00
|
|
|
VacuumPageHit,
|
|
|
|
VacuumPageMiss,
|
|
|
|
VacuumPageDirty,
|
2012-06-10 21:20:04 +02:00
|
|
|
read_rate, write_rate,
|
2007-04-18 18:44:18 +02:00
|
|
|
pg_rusage_show(&ru0))));
|
2011-11-25 16:10:46 +01:00
|
|
|
}
|
2007-04-18 18:44:18 +02:00
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
/*
|
|
|
|
* For Hot Standby we need to know the highest transaction id that will
|
|
|
|
* be removed by any change. VACUUM proceeds in a number of passes so
|
|
|
|
* we need to consider how each pass operates. The first phase runs
|
|
|
|
* heap_page_prune(), which can issue XLOG_HEAP2_CLEAN records as it
|
|
|
|
* progresses - these will have a latestRemovedXid on each record.
|
|
|
|
* In some cases this removes all of the tuples to be removed, though
|
|
|
|
* often we have dead tuples with index pointers so we must remember them
|
|
|
|
* for removal in phase 3. Index records for those rows are removed
|
|
|
|
* in phase 2 and index blocks do not have MVCC information attached.
|
|
|
|
* So before we can allow removal of any index tuples we need to issue
|
|
|
|
* a WAL record containing the latestRemovedXid of rows that will be
|
|
|
|
* removed in phase three. This allows recovery queries to block at the
|
|
|
|
* correct place, i.e. before phase two, rather than during phase three
|
|
|
|
* which would be after the rows have become inaccessible.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
vacuum_log_cleanup_info(Relation rel, LVRelStats *vacrelstats)
|
|
|
|
{
|
|
|
|
/*
|
2010-12-13 18:34:26 +01:00
|
|
|
* Skip this for relations for which no WAL is to be written, or if we're
|
|
|
|
* not trying to support archive recovery.
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
*/
|
2010-12-13 18:34:26 +01:00
|
|
|
if (!RelationNeedsWAL(rel) || !XLogIsNeeded())
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
return;
|
|
|
|
|
2010-04-22 04:15:45 +02:00
|
|
|
/*
|
|
|
|
* No need to write the record at all unless it contains a valid value
|
|
|
|
*/
|
|
|
|
if (TransactionIdIsValid(vacrelstats->latestRemovedXid))
|
2010-04-21 21:53:24 +02:00
|
|
|
(void) log_heap_cleanup_info(rel->rd_node, vacrelstats->latestRemovedXid);
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_scan_heap() -- scan an open heap relation
|
|
|
|
*
|
2012-04-13 14:49:59 +02:00
|
|
|
* This routine prunes each page in the heap, which will among other
|
|
|
|
* things truncate dead tuples to dead line pointers, defragment the
|
|
|
|
* page, and set commit status bits (see heap_page_prune). It also builds
|
|
|
|
* lists of dead tuples and pages with free space, calculates statistics
|
|
|
|
* on the number of live tuples in the heap, and marks pages as
|
|
|
|
* all-visible if appropriate. When done, or when we run low on space for
|
|
|
|
* dead-tuple TIDs, invoke vacuuming of indexes and call lazy_vacuum_heap
|
|
|
|
* to reclaim dead line pointers.
|
2006-07-10 18:20:52 +02:00
|
|
|
*
|
2012-04-13 14:49:59 +02:00
|
|
|
* If there are no indexes then we can reclaim line pointers on the fly;
|
|
|
|
* dead line pointers need only be retained until all index pointers that
|
|
|
|
* reference them have been killed.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
|
2008-12-03 14:05:22 +01:00
|
|
|
Relation *Irel, int nindexes, bool scan_all)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
BlockNumber nblocks,
|
|
|
|
blkno;
|
|
|
|
HeapTupleData tuple;
|
|
|
|
char *relname;
|
2006-09-13 19:47:08 +02:00
|
|
|
BlockNumber empty_pages,
|
|
|
|
vacuumed_pages;
|
2001-07-14 00:55:59 +02:00
|
|
|
double num_tuples,
|
|
|
|
tups_vacuumed,
|
|
|
|
nkeep,
|
|
|
|
nunused;
|
2006-05-03 00:25:10 +02:00
|
|
|
IndexBulkDeleteResult **indstats;
|
2001-07-14 00:55:59 +02:00
|
|
|
int i;
|
2005-10-04 00:52:26 +02:00
|
|
|
PGRUsage ru0;
|
2008-12-03 14:05:22 +01:00
|
|
|
Buffer vmbuffer = InvalidBuffer;
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
BlockNumber next_not_all_visible_block;
|
|
|
|
bool skipping_all_visible_blocks;
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
xl_heap_freeze_tuple *frozen;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_init(&ru0);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
relname = RelationGetRelationName(onerel);
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(elevel,
|
|
|
|
(errmsg("vacuuming \"%s.%s\"",
|
|
|
|
get_namespace_name(RelationGetNamespace(onerel)),
|
|
|
|
relname)));
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
empty_pages = vacuumed_pages = 0;
|
2001-07-14 00:55:59 +02:00
|
|
|
num_tuples = tups_vacuumed = nkeep = nunused = 0;
|
|
|
|
|
2006-05-03 00:25:10 +02:00
|
|
|
indstats = (IndexBulkDeleteResult **)
|
|
|
|
palloc0(nindexes * sizeof(IndexBulkDeleteResult *));
|
2004-12-01 20:00:56 +01:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
nblocks = RelationGetNumberOfBlocks(onerel);
|
|
|
|
vacrelstats->rel_pages = nblocks;
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
vacrelstats->scanned_pages = 0;
|
2001-07-14 00:55:59 +02:00
|
|
|
vacrelstats->nonempty_pages = 0;
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
vacrelstats->latestRemovedXid = InvalidTransactionId;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2006-09-13 19:47:08 +02:00
|
|
|
lazy_space_alloc(vacrelstats, nblocks);
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
frozen = palloc(sizeof(xl_heap_freeze_tuple) * MaxHeapTuplesPerPage);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
/*
|
|
|
|
* We want to skip pages that don't require vacuuming according to the
|
|
|
|
* visibility map, but only when we can skip at least SKIP_PAGES_THRESHOLD
|
|
|
|
* consecutive pages. Since we're reading sequentially, the OS should be
|
|
|
|
* doing readahead for us, so there's no gain in skipping a page now and
|
|
|
|
* then; that's likely to disable readahead and so be counterproductive.
|
|
|
|
* Also, skipping even a single page means that we can't update
|
|
|
|
* relfrozenxid, so we only want to do it if we can skip a goodly number
|
|
|
|
* of pages.
|
|
|
|
*
|
|
|
|
* Before entering the main loop, establish the invariant that
|
2011-06-09 20:32:50 +02:00
|
|
|
* next_not_all_visible_block is the next block number >= blkno that's not
|
|
|
|
* all-visible according to the visibility map, or nblocks if there's no
|
2014-05-06 18:12:18 +02:00
|
|
|
* such block. Also, we set up the skipping_all_visible_blocks flag,
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
* which is needed because we need hysteresis in the decision: once we've
|
|
|
|
* started skipping blocks, we may as well skip everything up to the next
|
|
|
|
* not-all-visible block.
|
|
|
|
*
|
|
|
|
* Note: if scan_all is true, we won't actually skip any pages; but we
|
|
|
|
* maintain next_not_all_visible_block anyway, so as to set up the
|
|
|
|
* all_visible_according_to_vm flag correctly for each page.
|
Fix more crash-safe visibility map bugs, and improve comments.
In lazy_scan_heap, we could issue bogus warnings about incorrect
information in the visibility map, because we checked the visibility
map bit before locking the heap page, creating a race condition. Fix
by rechecking the visibility map bit before we complain. Rejigger
some related logic so that we rely on the possibly-outdated
all_visible_according_to_vm value as little as possible.
In heap_multi_insert, it's not safe to clear the visibility map bit
before beginning the critical section. The visibility map is not
crash-safe unless we treat clearing the bit as a critical operation.
Specifically, if the transaction were to error out after we set the
bit and before entering the critical section, we could end up writing
the heap page to disk (with the bit cleared) and crashing before the
visibility map page made it to disk. That would be bad. heap_insert
has this correct, but somehow the order of operations got rearranged
when heap_multi_insert was added.
Also, add some more comments to visibilitymap_test, lazy_scan_heap,
and IndexOnlyNext, expounding on concurrency issues.
Per extensive code review by Andres Freund, and further review by Tom
Lane, who also made the original report about the bogus warnings.
2012-06-07 18:25:41 +02:00
|
|
|
*
|
|
|
|
* Note: The value returned by visibilitymap_test could be slightly
|
|
|
|
* out-of-date, since we make this test before reading the corresponding
|
|
|
|
* heap page or locking the buffer. This is OK. If we mistakenly think
|
|
|
|
* that the page is all-visible when in fact the flag's just been cleared,
|
|
|
|
* we might fail to vacuum the page. But it's OK to skip pages when
|
|
|
|
* scan_all is not set, so no great harm done; the next vacuum will find
|
|
|
|
* them. If we make the reverse mistake and vacuum a page unnecessarily,
|
|
|
|
* it'll just be a no-op.
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
*/
|
|
|
|
for (next_not_all_visible_block = 0;
|
|
|
|
next_not_all_visible_block < nblocks;
|
|
|
|
next_not_all_visible_block++)
|
|
|
|
{
|
|
|
|
if (!visibilitymap_test(onerel, next_not_all_visible_block, &vmbuffer))
|
|
|
|
break;
|
|
|
|
vacuum_delay_point();
|
|
|
|
}
|
|
|
|
if (next_not_all_visible_block >= SKIP_PAGES_THRESHOLD)
|
|
|
|
skipping_all_visible_blocks = true;
|
|
|
|
else
|
|
|
|
skipping_all_visible_blocks = false;
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
for (blkno = 0; blkno < nblocks; blkno++)
|
|
|
|
{
|
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
|
|
|
OffsetNumber offnum,
|
|
|
|
maxoff;
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
bool tupgone,
|
2001-07-14 00:55:59 +02:00
|
|
|
hastup;
|
|
|
|
int prev_dead_count;
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
int nfrozen;
|
2008-09-30 12:52:14 +02:00
|
|
|
Size freespace;
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
bool all_visible_according_to_vm;
|
2008-12-03 14:05:22 +01:00
|
|
|
bool all_visible;
|
2011-03-08 19:13:52 +01:00
|
|
|
bool has_dead_tuples;
|
2012-04-27 02:00:21 +02:00
|
|
|
TransactionId visibility_cutoff_xid = InvalidTransactionId;
|
2008-12-03 14:05:22 +01:00
|
|
|
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
if (blkno == next_not_all_visible_block)
|
2008-12-03 14:05:22 +01:00
|
|
|
{
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
/* Time to advance next_not_all_visible_block */
|
|
|
|
for (next_not_all_visible_block++;
|
|
|
|
next_not_all_visible_block < nblocks;
|
|
|
|
next_not_all_visible_block++)
|
2008-12-03 14:05:22 +01:00
|
|
|
{
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
if (!visibilitymap_test(onerel, next_not_all_visible_block,
|
|
|
|
&vmbuffer))
|
|
|
|
break;
|
|
|
|
vacuum_delay_point();
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We know we can't skip the current block. But set up
|
|
|
|
* skipping_all_visible_blocks to do the right thing at the
|
|
|
|
* following blocks.
|
|
|
|
*/
|
|
|
|
if (next_not_all_visible_block - blkno > SKIP_PAGES_THRESHOLD)
|
|
|
|
skipping_all_visible_blocks = true;
|
2009-01-22 20:25:00 +01:00
|
|
|
else
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
skipping_all_visible_blocks = false;
|
|
|
|
all_visible_according_to_vm = false;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Current block is all-visible */
|
|
|
|
if (skipping_all_visible_blocks && !scan_all)
|
|
|
|
continue;
|
|
|
|
all_visible_according_to_vm = true;
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2004-02-10 04:42:45 +01:00
|
|
|
vacuum_delay_point();
|
2004-02-06 20:36:18 +01:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* If we are close to overrunning the available space for dead-tuple
|
|
|
|
* TIDs, pause and do a cycle of vacuuming before we tackle this page.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
2005-09-02 21:02:20 +02:00
|
|
|
if ((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
|
2001-07-14 00:55:59 +02:00
|
|
|
vacrelstats->num_dead_tuples > 0)
|
|
|
|
{
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
/*
|
2012-06-10 21:20:04 +02:00
|
|
|
* Before beginning index vacuuming, we release any pin we may
|
|
|
|
* hold on the visibility map page. This isn't necessary for
|
|
|
|
* correctness, but we do it anyway to avoid holding the pin
|
|
|
|
* across a lengthy, unrelated operation.
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
*/
|
|
|
|
if (BufferIsValid(vmbuffer))
|
|
|
|
{
|
|
|
|
ReleaseBuffer(vmbuffer);
|
|
|
|
vmbuffer = InvalidBuffer;
|
|
|
|
}
|
|
|
|
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
/* Log cleanup info before we touch indexes */
|
|
|
|
vacuum_log_cleanup_info(onerel, vacrelstats);
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Remove index entries */
|
|
|
|
for (i = 0; i < nindexes; i++)
|
2004-12-01 20:00:56 +01:00
|
|
|
lazy_vacuum_index(Irel[i],
|
2006-05-03 00:25:10 +02:00
|
|
|
&indstats[i],
|
2004-12-01 20:00:56 +01:00
|
|
|
vacrelstats);
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Remove tuples from heap */
|
|
|
|
lazy_vacuum_heap(onerel, vacrelstats);
|
2010-07-06 21:19:02 +02:00
|
|
|
|
2010-04-21 19:20:56 +02:00
|
|
|
/*
|
|
|
|
* Forget the now-vacuumed tuples, and press on, but be careful
|
2010-07-06 21:19:02 +02:00
|
|
|
* not to reset latestRemovedXid since we want that value to be
|
|
|
|
* valid.
|
2010-04-21 19:20:56 +02:00
|
|
|
*/
|
2001-07-14 00:55:59 +02:00
|
|
|
vacrelstats->num_dead_tuples = 0;
|
2007-04-18 18:44:18 +02:00
|
|
|
vacrelstats->num_index_scans++;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
/*
|
|
|
|
* Pin the visibility map page in case we need to mark the page
|
|
|
|
* all-visible. In most cases this will be very cheap, because we'll
|
2012-06-10 21:20:04 +02:00
|
|
|
* already have the correct page pinned anyway. However, it's
|
|
|
|
* possible that (a) next_not_all_visible_block is covered by a
|
|
|
|
* different VM page than the current block or (b) we released our pin
|
|
|
|
* and did a cycle of index vacuuming.
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
*/
|
|
|
|
visibilitymap_pin(onerel, blkno, &vmbuffer);
|
|
|
|
|
Unite ReadBufferWithFork, ReadBufferWithStrategy, and ZeroOrReadBuffer
functions into one ReadBufferExtended function, that takes the strategy
and mode as argument. There's three modes, RBM_NORMAL which is the default
used by plain ReadBuffer(), RBM_ZERO, which replaces ZeroOrReadBuffer, and
a new mode RBM_ZERO_ON_ERROR, which allows callers to read corrupt pages
without throwing an error. The FSM needs the new mode to recover from
corrupt pages, which could happend if we crash after extending an FSM file,
and the new page is "torn".
Add fork number to some error messages in bufmgr.c, that still lacked it.
2008-10-31 16:05:00 +01:00
|
|
|
buf = ReadBufferExtended(onerel, MAIN_FORKNUM, blkno,
|
|
|
|
RBM_NORMAL, vac_strategy);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/* We need buffer cleanup lock so that we can prune HOT chains. */
|
2011-11-08 03:39:40 +01:00
|
|
|
if (!ConditionalLockBufferForCleanup(buf))
|
|
|
|
{
|
|
|
|
/*
|
2011-11-08 14:11:25 +01:00
|
|
|
* If we're not scanning the whole relation to guard against XID
|
|
|
|
* wraparound, it's OK to skip vacuuming a page. The next vacuum
|
|
|
|
* will clean it up.
|
2011-11-08 03:39:40 +01:00
|
|
|
*/
|
|
|
|
if (!scan_all)
|
|
|
|
{
|
|
|
|
ReleaseBuffer(buf);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If this is a wraparound checking vacuum, then we read the page
|
|
|
|
* with share lock to see if any xids need to be frozen. If the
|
|
|
|
* page doesn't need attention we just skip and continue. If it
|
|
|
|
* does, we wait for cleanup lock.
|
|
|
|
*
|
|
|
|
* We could defer the lock request further by remembering the page
|
2012-04-13 14:54:13 +02:00
|
|
|
* and coming back to it later, or we could even register
|
2011-11-08 03:39:40 +01:00
|
|
|
* ourselves for multiple buffers and then service whichever one
|
|
|
|
* is received first. For now, this seems good enough.
|
|
|
|
*/
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_SHARE);
|
|
|
|
if (!lazy_check_needs_freeze(buf))
|
|
|
|
{
|
|
|
|
UnlockReleaseBuffer(buf);
|
2013-11-18 08:51:09 +01:00
|
|
|
vacrelstats->scanned_pages++;
|
2011-11-08 03:39:40 +01:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
|
|
|
|
LockBufferForCleanup(buf);
|
|
|
|
/* drop through to normal processing */
|
|
|
|
}
|
|
|
|
|
|
|
|
vacrelstats->scanned_pages++;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
page = BufferGetPage(buf);
|
|
|
|
|
|
|
|
if (PageIsNew(page))
|
|
|
|
{
|
2005-05-07 23:32:24 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* An all-zeroes page could be left over if a backend extends the
|
|
|
|
* relation but crashes before initializing the page. Reclaim such
|
|
|
|
* pages for use.
|
2005-05-07 23:32:24 +02:00
|
|
|
*
|
2005-11-22 19:17:34 +01:00
|
|
|
* We have to be careful here because we could be looking at a
|
|
|
|
* page that someone has just added to the relation and not yet
|
|
|
|
* been able to initialize (see RelationGetBufferForTuple). To
|
2007-09-20 19:56:33 +02:00
|
|
|
* protect against that, release the buffer lock, grab the
|
2007-11-15 22:14:46 +01:00
|
|
|
* relation extension lock momentarily, and re-lock the buffer. If
|
|
|
|
* the page is still uninitialized by then, it must be left over
|
|
|
|
* from a crashed backend, and we can initialize it.
|
2005-05-07 23:32:24 +02:00
|
|
|
*
|
2005-11-22 19:17:34 +01:00
|
|
|
* We don't really need the relation lock when this is a new or
|
|
|
|
* temp relation, but it's probably not worth the code space to
|
|
|
|
* check that, since this surely isn't a critical path.
|
2005-05-07 23:32:24 +02:00
|
|
|
*
|
2005-11-22 19:17:34 +01:00
|
|
|
* Note: the comparable code in vacuum.c need not worry because
|
|
|
|
* it's got exclusive lock on the whole relation.
|
2005-05-07 23:32:24 +02:00
|
|
|
*/
|
2001-07-14 00:55:59 +02:00
|
|
|
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
|
2005-05-07 23:32:24 +02:00
|
|
|
LockRelationForExtension(onerel, ExclusiveLock);
|
|
|
|
UnlockRelationForExtension(onerel, ExclusiveLock);
|
2007-09-20 19:56:33 +02:00
|
|
|
LockBufferForCleanup(buf);
|
2001-07-14 00:55:59 +02:00
|
|
|
if (PageIsNew(page))
|
|
|
|
{
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(WARNING,
|
2005-10-15 04:49:52 +02:00
|
|
|
(errmsg("relation \"%s\" page %u is uninitialized --- fixing",
|
|
|
|
relname, blkno)));
|
2001-07-14 00:55:59 +02:00
|
|
|
PageInit(page, BufferGetPageSize(buf), 0);
|
2003-07-20 23:56:35 +02:00
|
|
|
empty_pages++;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
2008-09-30 12:52:14 +02:00
|
|
|
freespace = PageGetHeapFreeSpace(page);
|
2006-04-01 01:32:07 +02:00
|
|
|
MarkBufferDirty(buf);
|
|
|
|
UnlockReleaseBuffer(buf);
|
2008-09-30 12:52:14 +02:00
|
|
|
|
|
|
|
RecordPageWithFreeSpace(onerel, blkno, freespace);
|
2001-07-14 00:55:59 +02:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (PageIsEmpty(page))
|
|
|
|
{
|
|
|
|
empty_pages++;
|
2008-09-30 12:52:14 +02:00
|
|
|
freespace = PageGetHeapFreeSpace(page);
|
2008-12-03 14:05:22 +01:00
|
|
|
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
/* empty pages are always all-visible */
|
2008-12-03 14:05:22 +01:00
|
|
|
if (!PageIsAllVisible(page))
|
|
|
|
{
|
2013-10-23 13:03:54 +02:00
|
|
|
START_CRIT_SECTION();
|
|
|
|
|
|
|
|
/* mark buffer dirty before writing a WAL record */
|
|
|
|
MarkBufferDirty(buf);
|
|
|
|
|
2013-06-06 16:03:37 +02:00
|
|
|
/*
|
|
|
|
* It's possible that another backend has extended the heap,
|
|
|
|
* initialized the page, and then failed to WAL-log the page
|
|
|
|
* due to an ERROR. Since heap extension is not WAL-logged,
|
2014-05-06 18:12:18 +02:00
|
|
|
* recovery might try to replay our record setting the page
|
|
|
|
* all-visible and find that the page isn't initialized, which
|
|
|
|
* will cause a PANIC. To prevent that, check whether the
|
|
|
|
* page has been previously WAL-logged, and if not, do that
|
2013-06-06 16:03:37 +02:00
|
|
|
* now.
|
|
|
|
*/
|
|
|
|
if (RelationNeedsWAL(onerel) &&
|
|
|
|
PageGetLSN(page) == InvalidXLogRecPtr)
|
2013-12-03 23:10:47 +01:00
|
|
|
log_newpage_buffer(buf, true);
|
2013-06-06 16:03:37 +02:00
|
|
|
|
2008-12-03 14:05:22 +01:00
|
|
|
PageSetAllVisible(page);
|
2013-03-22 14:54:07 +01:00
|
|
|
visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr,
|
|
|
|
vmbuffer, InvalidTransactionId);
|
2013-10-23 13:03:54 +02:00
|
|
|
END_CRIT_SECTION();
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
|
|
|
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
UnlockReleaseBuffer(buf);
|
2008-09-30 12:52:14 +02:00
|
|
|
RecordPageWithFreeSpace(onerel, blkno, freespace);
|
2001-07-14 00:55:59 +02:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2007-11-15 22:14:46 +01:00
|
|
|
/*
|
2007-09-20 19:56:33 +02:00
|
|
|
* Prune all HOT-update chains in this page.
|
|
|
|
*
|
|
|
|
* We count tuples removed by the pruning step as removed by VACUUM.
|
|
|
|
*/
|
2010-04-21 19:20:56 +02:00
|
|
|
tups_vacuumed += heap_page_prune(onerel, buf, OldestXmin, false,
|
2010-07-06 21:19:02 +02:00
|
|
|
&vacrelstats->latestRemovedXid);
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/*
|
2007-11-15 22:14:46 +01:00
|
|
|
* Now scan the page to collect vacuumable items and check for tuples
|
|
|
|
* requiring freezing.
|
2007-09-20 19:56:33 +02:00
|
|
|
*/
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = true;
|
2011-03-08 19:13:52 +01:00
|
|
|
has_dead_tuples = false;
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
nfrozen = 0;
|
2001-07-14 00:55:59 +02:00
|
|
|
hastup = false;
|
|
|
|
prev_dead_count = vacrelstats->num_dead_tuples;
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
2013-02-13 16:46:23 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Note: If you change anything in the loop below, also look at
|
|
|
|
* heap_page_is_all_visible to see if that needs to be changed.
|
|
|
|
*/
|
2001-07-14 00:55:59 +02:00
|
|
|
for (offnum = FirstOffsetNumber;
|
|
|
|
offnum <= maxoff;
|
|
|
|
offnum = OffsetNumberNext(offnum))
|
|
|
|
{
|
|
|
|
ItemId itemid;
|
|
|
|
|
|
|
|
itemid = PageGetItemId(page, offnum);
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/* Unused items require no processing, but we count 'em */
|
2001-07-14 00:55:59 +02:00
|
|
|
if (!ItemIdIsUsed(itemid))
|
|
|
|
{
|
|
|
|
nunused += 1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/* Redirect items mustn't be touched */
|
2007-11-15 22:14:46 +01:00
|
|
|
if (ItemIdIsRedirected(itemid))
|
|
|
|
{
|
2007-09-20 19:56:33 +02:00
|
|
|
hastup = true; /* this page won't be truncatable */
|
2007-11-15 22:14:46 +01:00
|
|
|
continue;
|
|
|
|
}
|
2007-09-20 19:56:33 +02:00
|
|
|
|
2007-11-15 22:14:46 +01:00
|
|
|
ItemPointerSet(&(tuple.t_self), blkno, offnum);
|
2007-09-20 19:56:33 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* DEAD item pointers are to be vacuumed normally; but we don't
|
2007-11-15 22:14:46 +01:00
|
|
|
* count them in tups_vacuumed, else we'd be double-counting (at
|
|
|
|
* least in the common case where heap_page_prune() just freed up
|
|
|
|
* a non-HOT tuple).
|
2007-09-20 19:56:33 +02:00
|
|
|
*/
|
|
|
|
if (ItemIdIsDead(itemid))
|
|
|
|
{
|
|
|
|
lazy_record_dead_tuple(vacrelstats, &(tuple.t_self));
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = false;
|
2007-09-20 19:56:33 +02:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
Assert(ItemIdIsNormal(itemid));
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
|
|
|
|
tuple.t_len = ItemIdGetLength(itemid);
|
2013-07-22 19:26:33 +02:00
|
|
|
tuple.t_tableOid = RelationGetRelid(onerel);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
tupgone = false;
|
|
|
|
|
2013-07-22 19:26:33 +02:00
|
|
|
switch (HeapTupleSatisfiesVacuum(&tuple, OldestXmin, buf))
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
case HEAPTUPLE_DEAD:
|
2007-11-15 22:14:46 +01:00
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
/*
|
|
|
|
* Ordinarily, DEAD tuples would have been removed by
|
|
|
|
* heap_page_prune(), but it's possible that the tuple
|
|
|
|
* state changed since heap_page_prune() looked. In
|
|
|
|
* particular an INSERT_IN_PROGRESS tuple could have
|
|
|
|
* changed to DEAD if the inserter aborted. So this
|
|
|
|
* cannot be considered an error condition.
|
|
|
|
*
|
|
|
|
* If the tuple is HOT-updated then it must only be
|
2007-11-15 22:14:46 +01:00
|
|
|
* removed by a prune operation; so we keep it just as if
|
|
|
|
* it were RECENTLY_DEAD. Also, if it's a heap-only
|
|
|
|
* tuple, we choose to keep it, because it'll be a lot
|
|
|
|
* cheaper to get rid of it in the next pruning pass than
|
|
|
|
* to treat it like an indexed tuple.
|
2007-09-20 19:56:33 +02:00
|
|
|
*/
|
|
|
|
if (HeapTupleIsHotUpdated(&tuple) ||
|
|
|
|
HeapTupleIsHeapOnly(&tuple))
|
|
|
|
nkeep += 1;
|
|
|
|
else
|
2007-11-15 22:14:46 +01:00
|
|
|
tupgone = true; /* we can delete the tuple */
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = false;
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
case HEAPTUPLE_LIVE:
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
/* Tuple is good --- but let's do some validity checks */
|
2004-02-12 06:39:55 +01:00
|
|
|
if (onerel->rd_rel->relhasoids &&
|
|
|
|
!OidIsValid(HeapTupleGetOid(&tuple)))
|
|
|
|
elog(WARNING, "relation \"%s\" TID %u/%u: OID is invalid",
|
|
|
|
relname, blkno, offnum);
|
2008-12-03 14:05:22 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Is the tuple definitely visible to all transactions?
|
|
|
|
*
|
|
|
|
* NB: Like with per-tuple hint bits, we can't set the
|
|
|
|
* PD_ALL_VISIBLE flag if the inserter committed
|
|
|
|
* asynchronously. See SetHintBits for more info. Check
|
2014-05-06 18:12:18 +02:00
|
|
|
* that the tuple is hinted xmin-committed because of
|
|
|
|
* that.
|
2008-12-03 14:05:22 +01:00
|
|
|
*/
|
|
|
|
if (all_visible)
|
|
|
|
{
|
|
|
|
TransactionId xmin;
|
|
|
|
|
2013-12-22 21:49:09 +01:00
|
|
|
if (!HeapTupleHeaderXminCommitted(tuple.t_data))
|
2008-12-03 14:05:22 +01:00
|
|
|
{
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
}
|
2009-06-11 16:49:15 +02:00
|
|
|
|
2008-12-03 14:05:22 +01:00
|
|
|
/*
|
2009-06-11 16:49:15 +02:00
|
|
|
* The inserter definitely committed. But is it old
|
|
|
|
* enough that everyone sees it as committed?
|
2008-12-03 14:05:22 +01:00
|
|
|
*/
|
|
|
|
xmin = HeapTupleHeaderGetXmin(tuple.t_data);
|
|
|
|
if (!TransactionIdPrecedes(xmin, OldestXmin))
|
|
|
|
{
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
}
|
2012-04-27 02:00:21 +02:00
|
|
|
|
|
|
|
/* Track newest xmin on page. */
|
|
|
|
if (TransactionIdFollows(xmin, visibility_cutoff_xid))
|
|
|
|
visibility_cutoff_xid = xmin;
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
case HEAPTUPLE_RECENTLY_DEAD:
|
2001-10-25 07:50:21 +02:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* If tuple is recently deleted then we must not remove it
|
|
|
|
* from relation.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
nkeep += 1;
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = false;
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
case HEAPTUPLE_INSERT_IN_PROGRESS:
|
|
|
|
/* This is an expected case during concurrent vacuum */
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = false;
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
case HEAPTUPLE_DELETE_IN_PROGRESS:
|
|
|
|
/* This is an expected case during concurrent vacuum */
|
2008-12-03 14:05:22 +01:00
|
|
|
all_visible = false;
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
default:
|
2003-07-20 23:56:35 +02:00
|
|
|
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
|
2001-07-14 00:55:59 +02:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tupgone)
|
|
|
|
{
|
|
|
|
lazy_record_dead_tuple(vacrelstats, &(tuple.t_self));
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
HeapTupleHeaderAdvanceLatestRemovedXid(tuple.t_data,
|
2010-02-26 03:01:40 +01:00
|
|
|
&vacrelstats->latestRemovedXid);
|
2001-07-14 00:55:59 +02:00
|
|
|
tups_vacuumed += 1;
|
2011-03-08 19:13:52 +01:00
|
|
|
has_dead_tuples = true;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
num_tuples += 1;
|
|
|
|
hastup = true;
|
2006-07-10 18:20:52 +02:00
|
|
|
|
2006-10-04 02:30:14 +02:00
|
|
|
/*
|
2007-11-15 22:14:46 +01:00
|
|
|
* Each non-removable tuple must be checked to see if it needs
|
|
|
|
* freezing. Note we already have exclusive buffer lock.
|
2006-07-10 18:20:52 +02:00
|
|
|
*/
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
if (heap_prepare_freeze_tuple(tuple.t_data, FreezeLimit,
|
|
|
|
MultiXactCutoff, &frozen[nfrozen]))
|
|
|
|
frozen[nfrozen++].offset = offnum;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
2001-10-25 07:50:21 +02:00
|
|
|
} /* scan along page */
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
/*
|
|
|
|
* If we froze any tuples, mark the buffer dirty, and write a WAL
|
|
|
|
* record recording the changes. We must log the changes to be
|
|
|
|
* crash-safe against future truncation of CLOG.
|
|
|
|
*/
|
|
|
|
if (nfrozen > 0)
|
|
|
|
{
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
START_CRIT_SECTION();
|
|
|
|
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
MarkBufferDirty(buf);
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
|
|
|
|
/* execute collected freezes */
|
|
|
|
for (i = 0; i < nfrozen; i++)
|
|
|
|
{
|
|
|
|
ItemId itemid;
|
|
|
|
HeapTupleHeader htup;
|
|
|
|
|
|
|
|
itemid = PageGetItemId(page, frozen[i].offset);
|
|
|
|
htup = (HeapTupleHeader) PageGetItem(page, itemid);
|
|
|
|
|
|
|
|
heap_execute_freeze_tuple(htup, &frozen[i]);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Now WAL-log freezing if neccessary */
|
2010-12-13 18:34:26 +01:00
|
|
|
if (RelationNeedsWAL(onerel))
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
{
|
|
|
|
XLogRecPtr recptr;
|
|
|
|
|
|
|
|
recptr = log_heap_freeze(onerel, buf, FreezeLimit,
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
frozen, nfrozen);
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
PageSetLSN(page, recptr);
|
|
|
|
}
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
|
|
|
|
END_CRIT_SECTION();
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
}
|
|
|
|
|
2006-09-13 19:47:08 +02:00
|
|
|
/*
|
|
|
|
* If there are no indexes then we can vacuum the page right now
|
|
|
|
* instead of doing a second scan.
|
|
|
|
*/
|
|
|
|
if (nindexes == 0 &&
|
|
|
|
vacrelstats->num_dead_tuples > 0)
|
|
|
|
{
|
|
|
|
/* Remove tuples from heap */
|
2013-02-13 16:46:23 +01:00
|
|
|
lazy_vacuum_page(onerel, blkno, buf, 0, vacrelstats, &vmbuffer);
|
Fix spurious warning after vacuuming a page on a table with no indexes.
There is a rare race condition, when a transaction that inserted a tuple
aborts while vacuum is processing the page containing the inserted tuple.
Vacuum prunes the page first, which normally removes any dead tuples, but
if the inserting transaction aborts right after that, the loop after
pruning will see a dead tuple and remove it instead. That's OK, but if the
page is on a table with no indexes, and the page becomes completely empty
after removing the dead tuple (or tuples) on it, it will be immediately
marked as all-visible. That's OK, but the sanity check in vacuum would
throw a warning because it thinks that the page contains dead tuples and
was nevertheless marked as all-visible, even though it just vacuumed away
the dead tuples and so it doesn't actually contain any.
Spotted this while reading the code. It's difficult to hit the race
condition otherwise, but can be done by putting a breakpoint after the
heap_page_prune() call.
Backpatch all the way to 8.4, where this code first appeared.
2013-09-26 10:24:40 +02:00
|
|
|
has_dead_tuples = false;
|
2010-07-06 21:19:02 +02:00
|
|
|
|
2010-04-21 19:20:56 +02:00
|
|
|
/*
|
|
|
|
* Forget the now-vacuumed tuples, and press on, but be careful
|
2010-07-06 21:19:02 +02:00
|
|
|
* not to reset latestRemovedXid since we want that value to be
|
|
|
|
* valid.
|
2010-04-21 19:20:56 +02:00
|
|
|
*/
|
2006-09-13 19:47:08 +02:00
|
|
|
vacrelstats->num_dead_tuples = 0;
|
|
|
|
vacuumed_pages++;
|
|
|
|
}
|
|
|
|
|
2008-09-30 12:52:14 +02:00
|
|
|
freespace = PageGetHeapFreeSpace(page);
|
|
|
|
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
/* mark page all-visible, if appropriate */
|
2013-04-30 09:15:49 +02:00
|
|
|
if (all_visible && !all_visible_according_to_vm)
|
2008-12-03 14:05:22 +01:00
|
|
|
{
|
2013-04-30 09:15:49 +02:00
|
|
|
/*
|
|
|
|
* It should never be the case that the visibility map page is set
|
|
|
|
* while the page-level bit is clear, but the reverse is allowed
|
2014-05-06 18:12:18 +02:00
|
|
|
* (if checksums are not enabled). Regardless, set the both bits
|
2013-04-30 09:15:49 +02:00
|
|
|
* so that we get back in sync.
|
|
|
|
*
|
|
|
|
* NB: If the heap page is all-visible but the VM bit is not set,
|
2013-05-29 22:58:43 +02:00
|
|
|
* we don't need to dirty the heap page. However, if checksums
|
|
|
|
* are enabled, we do need to make sure that the heap page is
|
|
|
|
* dirtied before passing it to visibilitymap_set(), because it
|
|
|
|
* may be logged. Given that this situation should only happen in
|
|
|
|
* rare cases after a crash, it is not worth optimizing.
|
2013-04-30 09:15:49 +02:00
|
|
|
*/
|
|
|
|
PageSetAllVisible(page);
|
|
|
|
MarkBufferDirty(buf);
|
|
|
|
visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr,
|
|
|
|
vmbuffer, visibility_cutoff_xid);
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* As of PostgreSQL 9.2, the visibility map bit should never be set if
|
Fix more crash-safe visibility map bugs, and improve comments.
In lazy_scan_heap, we could issue bogus warnings about incorrect
information in the visibility map, because we checked the visibility
map bit before locking the heap page, creating a race condition. Fix
by rechecking the visibility map bit before we complain. Rejigger
some related logic so that we rely on the possibly-outdated
all_visible_according_to_vm value as little as possible.
In heap_multi_insert, it's not safe to clear the visibility map bit
before beginning the critical section. The visibility map is not
crash-safe unless we treat clearing the bit as a critical operation.
Specifically, if the transaction were to error out after we set the
bit and before entering the critical section, we could end up writing
the heap page to disk (with the bit cleared) and crashing before the
visibility map page made it to disk. That would be bad. heap_insert
has this correct, but somehow the order of operations got rearranged
when heap_multi_insert was added.
Also, add some more comments to visibilitymap_test, lazy_scan_heap,
and IndexOnlyNext, expounding on concurrency issues.
Per extensive code review by Andres Freund, and further review by Tom
Lane, who also made the original report about the bogus warnings.
2012-06-07 18:25:41 +02:00
|
|
|
* the page-level bit is clear. However, it's possible that the bit
|
|
|
|
* got cleared after we checked it and before we took the buffer
|
|
|
|
* content lock, so we must recheck before jumping to the conclusion
|
|
|
|
* that something bad has happened.
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
*/
|
Fix more crash-safe visibility map bugs, and improve comments.
In lazy_scan_heap, we could issue bogus warnings about incorrect
information in the visibility map, because we checked the visibility
map bit before locking the heap page, creating a race condition. Fix
by rechecking the visibility map bit before we complain. Rejigger
some related logic so that we rely on the possibly-outdated
all_visible_according_to_vm value as little as possible.
In heap_multi_insert, it's not safe to clear the visibility map bit
before beginning the critical section. The visibility map is not
crash-safe unless we treat clearing the bit as a critical operation.
Specifically, if the transaction were to error out after we set the
bit and before entering the critical section, we could end up writing
the heap page to disk (with the bit cleared) and crashing before the
visibility map page made it to disk. That would be bad. heap_insert
has this correct, but somehow the order of operations got rearranged
when heap_multi_insert was added.
Also, add some more comments to visibilitymap_test, lazy_scan_heap,
and IndexOnlyNext, expounding on concurrency issues.
Per extensive code review by Andres Freund, and further review by Tom
Lane, who also made the original report about the bogus warnings.
2012-06-07 18:25:41 +02:00
|
|
|
else if (all_visible_according_to_vm && !PageIsAllVisible(page)
|
|
|
|
&& visibilitymap_test(onerel, blkno, &vmbuffer))
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
{
|
|
|
|
elog(WARNING, "page is not marked all-visible but visibility map bit is set in relation \"%s\" page %u",
|
|
|
|
relname, blkno);
|
|
|
|
visibilitymap_clear(onerel, blkno, vmbuffer);
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
2011-04-10 17:42:00 +02:00
|
|
|
|
2011-03-08 19:13:52 +01:00
|
|
|
/*
|
|
|
|
* It's possible for the value returned by GetOldestXmin() to move
|
|
|
|
* backwards, so it's not wrong for us to see tuples that appear to
|
|
|
|
* not be visible to everyone yet, while PD_ALL_VISIBLE is already
|
|
|
|
* set. The real safe xmin value never moves backwards, but
|
|
|
|
* GetOldestXmin() is conservative and sometimes returns a value
|
2011-04-10 17:42:00 +02:00
|
|
|
* that's unnecessarily small, so if we see that contradiction it just
|
|
|
|
* means that the tuples that we think are not visible to everyone yet
|
|
|
|
* actually are, and the PD_ALL_VISIBLE flag is correct.
|
2011-03-08 19:13:52 +01:00
|
|
|
*
|
|
|
|
* There should never be dead tuples on a page with PD_ALL_VISIBLE
|
|
|
|
* set, however.
|
|
|
|
*/
|
|
|
|
else if (PageIsAllVisible(page) && has_dead_tuples)
|
2008-12-03 14:05:22 +01:00
|
|
|
{
|
2011-03-08 19:13:52 +01:00
|
|
|
elog(WARNING, "page containing dead tuples is marked as all-visible in relation \"%s\" page %u",
|
2009-08-24 04:18:32 +02:00
|
|
|
relname, blkno);
|
2008-12-03 14:05:22 +01:00
|
|
|
PageClearAllVisible(page);
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
MarkBufferDirty(buf);
|
Make the visibility map crash-safe.
This involves two main changes from the previous behavior. First,
when we set a bit in the visibility map, emit a new WAL record of type
XLOG_HEAP2_VISIBLE. Replay sets the page-level PD_ALL_VISIBLE bit and
the visibility map bit. Second, when inserting, updating, or deleting
a tuple, we can no longer get away with clearing the visibility map
bit after releasing the lock on the corresponding heap page, because
an intervening crash might leave the visibility map bit set and the
page-level bit clear. Making this work requires a bit of interface
refactoring.
In passing, a few minor but related cleanups: change the test in
visibilitymap_set and visibilitymap_clear to throw an error if the
wrong page (or no page) is pinned, rather than silently doing nothing;
this case should never occur. Also, remove duplicate definitions of
InvalidXLogRecPtr.
Patch by me, review by Noah Misch.
2011-06-22 05:04:40 +02:00
|
|
|
visibilitymap_clear(onerel, blkno, vmbuffer);
|
2008-12-03 14:05:22 +01:00
|
|
|
}
|
|
|
|
|
Rearrange lazy_scan_heap to avoid visibility map race conditions.
We must set the visibility map bit before releasing our exclusive lock
on the heap page; otherwise, someone might clear the heap page bit
before we set the visibility map bit, leading to a situation where the
visibility map thinks the page is all-visible but it's really not.
This problem has existed since 8.4, but it wasn't critical before we
had index-only scans, since the worst case scenario was that the page
wouldn't get vacuumed until the next scan_all vacuum.
Along the way, a couple of minor, related improvements: (1) if we
pause the heap scan to do an index vac cycle, release any visibility
map page we're holding, since really long-running pins are not good
for a variety of reasons; and (2) warn if we see a page that's marked
all-visible in the visibility map but not on the page level, since
that should never happen any more (it was allowed in previous
releases, but not in 9.2).
2012-04-24 04:08:06 +02:00
|
|
|
UnlockReleaseBuffer(buf);
|
2008-12-03 14:05:22 +01:00
|
|
|
|
2008-09-30 12:52:14 +02:00
|
|
|
/* Remember the location of the last page with nonremovable tuples */
|
|
|
|
if (hastup)
|
|
|
|
vacrelstats->nonempty_pages = blkno + 1;
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/*
|
2001-10-25 07:50:21 +02:00
|
|
|
* If we remembered any tuples for deletion, then the page will be
|
2005-10-15 04:49:52 +02:00
|
|
|
* visited again by lazy_vacuum_heap, which will compute and record
|
2014-05-06 18:12:18 +02:00
|
|
|
* its post-compaction free space. If not, then we're done with this
|
|
|
|
* page, so remember its free space as-is. (This path will always be
|
2006-10-04 02:30:14 +02:00
|
|
|
* taken if there are no indexes.)
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
if (vacrelstats->num_dead_tuples == prev_dead_count)
|
2008-09-30 12:52:14 +02:00
|
|
|
RecordPageWithFreeSpace(onerel, blkno, freespace);
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
Rework tuple freezing protocol
Tuple freezing was broken in connection to MultiXactIds; commit
8e53ae025de9 tried to fix it, but didn't go far enough. As noted by
Noah Misch, freezing a tuple whose Xmax is a multi containing an aborted
update might cause locks in the multi to go ignored by later
transactions. This is because the code depended on a multixact above
their cutoff point not having any lock-only member older than the cutoff
point for Xids, which is easily defeated in READ COMMITTED transactions.
The fix for this involves creating a new MultiXactId when necessary.
But this cannot be done during WAL replay, and moreover multixact
examination requires using CLOG access routines which are not supposed
to be used during WAL replay either; so tuple freezing cannot be done
with the old freeze WAL record. Therefore, separate the freezing
computation from its execution, and change the WAL record to carry all
necessary information. At WAL replay time, it's easy to re-execute
freezing because we don't need to re-compute the new infomask/Xmax
values but just take them from the WAL record.
While at it, restructure the coding to ensure all page changes occur in
a single critical section without much room for failures. The previous
coding wasn't using a critical section, without any explanation as to
why this was acceptable.
In replication scenarios using the 9.3 branch, standby servers must be
upgraded before their master, so that they are prepared to deal with the
new WAL record once the master is upgraded; failure to do so will cause
WAL replay to die with a PANIC message. Later upgrade of the standby
will allow the process to continue where it left off, so there's no
disruption of the data in the standby in any case. Standbys know how to
deal with the old WAL record, so it's okay to keep the master running
the old code for a while.
In master, the old freeze WAL record is gone, for cleanliness' sake;
there's no compatibility concern there.
Backpatch to 9.3, where the original bug was introduced and where the
previous fix was backpatched.
Álvaro Herrera and Andres Freund
2013-12-16 15:29:50 +01:00
|
|
|
pfree(frozen);
|
|
|
|
|
2001-07-18 02:46:25 +02:00
|
|
|
/* save stats for use later */
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
vacrelstats->scanned_tuples = num_tuples;
|
2004-12-01 20:00:56 +01:00
|
|
|
vacrelstats->tuples_deleted = tups_vacuumed;
|
2014-01-19 01:24:20 +01:00
|
|
|
vacrelstats->new_dead_tuples = nkeep;
|
2001-07-18 02:46:25 +02:00
|
|
|
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
/* now we can compute the new value for pg_class.reltuples */
|
|
|
|
vacrelstats->new_rel_tuples = vac_estimate_reltuples(onerel, false,
|
|
|
|
nblocks,
|
2011-06-09 20:32:50 +02:00
|
|
|
vacrelstats->scanned_pages,
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
num_tuples);
|
|
|
|
|
2012-12-03 19:53:31 +01:00
|
|
|
/*
|
|
|
|
* Release any remaining pin on visibility map page.
|
|
|
|
*/
|
|
|
|
if (BufferIsValid(vmbuffer))
|
|
|
|
{
|
|
|
|
ReleaseBuffer(vmbuffer);
|
|
|
|
vmbuffer = InvalidBuffer;
|
|
|
|
}
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* If any tuples need to be deleted, perform final vacuum cycle */
|
2003-02-22 01:45:05 +01:00
|
|
|
/* XXX put a threshold on min number of tuples here? */
|
2001-07-14 00:55:59 +02:00
|
|
|
if (vacrelstats->num_dead_tuples > 0)
|
|
|
|
{
|
Allow read only connections during recovery, known as Hot Standby.
Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.
New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.
This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.
Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.
Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.
2009-12-19 02:32:45 +01:00
|
|
|
/* Log cleanup info before we touch indexes */
|
|
|
|
vacuum_log_cleanup_info(onerel, vacrelstats);
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Remove index entries */
|
|
|
|
for (i = 0; i < nindexes; i++)
|
2004-12-01 20:00:56 +01:00
|
|
|
lazy_vacuum_index(Irel[i],
|
2006-05-03 00:25:10 +02:00
|
|
|
&indstats[i],
|
2004-12-01 20:00:56 +01:00
|
|
|
vacrelstats);
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Remove tuples from heap */
|
|
|
|
lazy_vacuum_heap(onerel, vacrelstats);
|
2007-04-18 18:44:18 +02:00
|
|
|
vacrelstats->num_index_scans++;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
2006-05-03 00:25:10 +02:00
|
|
|
|
|
|
|
/* Do post-vacuum cleanup and statistics update for each index */
|
|
|
|
for (i = 0; i < nindexes; i++)
|
|
|
|
lazy_cleanup_index(Irel[i], indstats[i], vacrelstats);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2006-09-13 19:47:08 +02:00
|
|
|
/* If no indexes, make log report that lazy_vacuum_heap would've made */
|
|
|
|
if (vacuumed_pages)
|
|
|
|
ereport(elevel,
|
|
|
|
(errmsg("\"%s\": removed %.0f row versions in %u pages",
|
|
|
|
RelationGetRelationName(onerel),
|
|
|
|
tups_vacuumed, vacuumed_pages)));
|
|
|
|
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(elevel,
|
2008-12-03 14:05:22 +01:00
|
|
|
(errmsg("\"%s\": found %.0f removable, %.0f nonremovable row versions in %u out of %u pages",
|
2003-07-20 23:56:35 +02:00
|
|
|
RelationGetRelationName(onerel),
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
tups_vacuumed, num_tuples,
|
|
|
|
vacrelstats->scanned_pages, nblocks),
|
2003-09-25 08:58:07 +02:00
|
|
|
errdetail("%.0f dead row versions cannot be removed yet.\n"
|
2003-07-20 23:56:35 +02:00
|
|
|
"There were %.0f unused item pointers.\n"
|
|
|
|
"%u pages are entirely empty.\n"
|
2005-10-04 00:52:26 +02:00
|
|
|
"%s.",
|
2003-07-20 23:56:35 +02:00
|
|
|
nkeep,
|
|
|
|
nunused,
|
|
|
|
empty_pages,
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_show(&ru0))));
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_vacuum_heap() -- second pass over the heap
|
|
|
|
*
|
|
|
|
* This routine marks dead tuples as unused and compacts out free
|
|
|
|
* space on their pages. Pages not having dead tuples recorded from
|
|
|
|
* lazy_scan_heap are not visited at all.
|
|
|
|
*
|
|
|
|
* Note: the reason for doing this as a second pass is we cannot remove
|
|
|
|
* the tuples until we've removed their index entries, and we want to
|
|
|
|
* process index entry removal in batches as large as possible.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
|
|
|
|
{
|
|
|
|
int tupindex;
|
|
|
|
int npages;
|
2005-10-04 00:52:26 +02:00
|
|
|
PGRUsage ru0;
|
2013-02-13 16:46:23 +01:00
|
|
|
Buffer vmbuffer = InvalidBuffer;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_init(&ru0);
|
2001-07-14 00:55:59 +02:00
|
|
|
npages = 0;
|
|
|
|
|
|
|
|
tupindex = 0;
|
|
|
|
while (tupindex < vacrelstats->num_dead_tuples)
|
|
|
|
{
|
2001-10-25 07:50:21 +02:00
|
|
|
BlockNumber tblk;
|
2001-07-14 00:55:59 +02:00
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
2008-09-30 12:52:14 +02:00
|
|
|
Size freespace;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2004-02-10 04:42:45 +01:00
|
|
|
vacuum_delay_point();
|
2004-02-06 20:36:18 +01:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
|
Unite ReadBufferWithFork, ReadBufferWithStrategy, and ZeroOrReadBuffer
functions into one ReadBufferExtended function, that takes the strategy
and mode as argument. There's three modes, RBM_NORMAL which is the default
used by plain ReadBuffer(), RBM_ZERO, which replaces ZeroOrReadBuffer, and
a new mode RBM_ZERO_ON_ERROR, which allows callers to read corrupt pages
without throwing an error. The FSM needs the new mode to recover from
corrupt pages, which could happend if we crash after extending an FSM file,
and the new page is "torn".
Add fork number to some error messages in bufmgr.c, that still lacked it.
2008-10-31 16:05:00 +01:00
|
|
|
buf = ReadBufferExtended(onerel, MAIN_FORKNUM, tblk, RBM_NORMAL,
|
|
|
|
vac_strategy);
|
2011-11-08 03:39:40 +01:00
|
|
|
if (!ConditionalLockBufferForCleanup(buf))
|
2012-01-13 14:22:31 +01:00
|
|
|
{
|
|
|
|
ReleaseBuffer(buf);
|
|
|
|
++tupindex;
|
2011-11-08 03:39:40 +01:00
|
|
|
continue;
|
2012-01-13 14:22:31 +01:00
|
|
|
}
|
2013-02-13 16:46:23 +01:00
|
|
|
tupindex = lazy_vacuum_page(onerel, tblk, buf, tupindex, vacrelstats,
|
|
|
|
&vmbuffer);
|
2008-09-30 12:52:14 +02:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/* Now that we've compacted the page, record its available space */
|
|
|
|
page = BufferGetPage(buf);
|
2008-09-30 12:52:14 +02:00
|
|
|
freespace = PageGetHeapFreeSpace(page);
|
|
|
|
|
2006-04-01 01:32:07 +02:00
|
|
|
UnlockReleaseBuffer(buf);
|
2008-09-30 12:52:14 +02:00
|
|
|
RecordPageWithFreeSpace(onerel, tblk, freespace);
|
2001-07-14 00:55:59 +02:00
|
|
|
npages++;
|
|
|
|
}
|
|
|
|
|
2013-02-13 16:46:23 +01:00
|
|
|
if (BufferIsValid(vmbuffer))
|
|
|
|
{
|
|
|
|
ReleaseBuffer(vmbuffer);
|
|
|
|
vmbuffer = InvalidBuffer;
|
|
|
|
}
|
|
|
|
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(elevel,
|
2003-09-25 08:58:07 +02:00
|
|
|
(errmsg("\"%s\": removed %d row versions in %d pages",
|
2003-07-20 23:56:35 +02:00
|
|
|
RelationGetRelationName(onerel),
|
|
|
|
tupindex, npages),
|
2005-10-04 00:52:26 +02:00
|
|
|
errdetail("%s.",
|
|
|
|
pg_rusage_show(&ru0))));
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_vacuum_page() -- free dead tuples on a page
|
|
|
|
* and repair its fragmentation.
|
|
|
|
*
|
2007-09-20 19:56:33 +02:00
|
|
|
* Caller must hold pin and buffer cleanup lock on the buffer.
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
|
|
|
* tupindex is the index in vacrelstats->dead_tuples of the first dead
|
|
|
|
* tuple for this page. We assume the rest follow sequentially.
|
|
|
|
* The return value is the first tupindex after the tuples of this page.
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
|
2013-02-13 16:46:23 +01:00
|
|
|
int tupindex, LVRelStats *vacrelstats, Buffer *vmbuffer)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
Page page = BufferGetPage(buffer);
|
2007-09-20 19:56:33 +02:00
|
|
|
OffsetNumber unused[MaxOffsetNumber];
|
|
|
|
int uncnt = 0;
|
2013-05-29 22:58:43 +02:00
|
|
|
TransactionId visibility_cutoff_xid;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
START_CRIT_SECTION();
|
2006-04-01 01:32:07 +02:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
for (; tupindex < vacrelstats->num_dead_tuples; tupindex++)
|
|
|
|
{
|
2001-10-25 07:50:21 +02:00
|
|
|
BlockNumber tblk;
|
|
|
|
OffsetNumber toff;
|
2007-09-20 19:56:33 +02:00
|
|
|
ItemId itemid;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
|
|
|
|
if (tblk != blkno)
|
|
|
|
break; /* past end of tuples for this block */
|
|
|
|
toff = ItemPointerGetOffsetNumber(&vacrelstats->dead_tuples[tupindex]);
|
|
|
|
itemid = PageGetItemId(page, toff);
|
2007-09-13 00:10:26 +02:00
|
|
|
ItemIdSetUnused(itemid);
|
2007-09-20 19:56:33 +02:00
|
|
|
unused[uncnt++] = toff;
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
2007-09-20 19:56:33 +02:00
|
|
|
PageRepairFragmentation(page);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2013-04-30 09:15:49 +02:00
|
|
|
/*
|
|
|
|
* Mark buffer dirty before we write WAL.
|
|
|
|
*/
|
|
|
|
MarkBufferDirty(buffer);
|
|
|
|
|
2013-12-13 12:52:47 +01:00
|
|
|
/* XLOG stuff */
|
|
|
|
if (RelationNeedsWAL(onerel))
|
|
|
|
{
|
|
|
|
XLogRecPtr recptr;
|
|
|
|
|
|
|
|
recptr = log_heap_clean(onerel, buffer,
|
|
|
|
NULL, 0, NULL, 0,
|
|
|
|
unused, uncnt,
|
|
|
|
vacrelstats->latestRemovedXid);
|
|
|
|
PageSetLSN(page, recptr);
|
|
|
|
}
|
|
|
|
|
2014-06-20 11:06:42 +02:00
|
|
|
/*
|
|
|
|
* End critical section, so we safely can do visibility tests (which
|
|
|
|
* possibly need to perform IO and allocate memory!). If we crash now the
|
|
|
|
* page (including the corresponding vm bit) might not be marked all
|
|
|
|
* visible, but that's fine. A later vacuum will fix that.
|
|
|
|
*/
|
|
|
|
END_CRIT_SECTION();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Now that we have removed the dead tuples from the page, once again
|
|
|
|
* check if the page has become all-visible. The page is already marked
|
|
|
|
* dirty, exclusively locked, and, if needed, a full page image has been
|
|
|
|
* emitted in the log_heap_clean() above.
|
|
|
|
*/
|
|
|
|
if (heap_page_is_all_visible(onerel, buffer, &visibility_cutoff_xid))
|
|
|
|
PageSetAllVisible(page);
|
|
|
|
|
2013-02-13 16:46:23 +01:00
|
|
|
/*
|
2014-04-17 16:47:50 +02:00
|
|
|
* All the changes to the heap page have been done. If the all-visible
|
|
|
|
* flag is now set, also set the VM bit.
|
2013-02-13 16:46:23 +01:00
|
|
|
*/
|
2014-04-17 16:47:50 +02:00
|
|
|
if (PageIsAllVisible(page) &&
|
|
|
|
!visibilitymap_test(onerel, blkno, vmbuffer))
|
2013-02-13 16:46:23 +01:00
|
|
|
{
|
|
|
|
Assert(BufferIsValid(*vmbuffer));
|
2013-03-22 14:54:07 +01:00
|
|
|
visibilitymap_set(onerel, blkno, buffer, InvalidXLogRecPtr, *vmbuffer,
|
2013-05-29 22:58:43 +02:00
|
|
|
visibility_cutoff_xid);
|
2013-02-13 16:46:23 +01:00
|
|
|
}
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
return tupindex;
|
|
|
|
}
|
|
|
|
|
2011-11-08 03:39:40 +01:00
|
|
|
/*
|
|
|
|
* lazy_check_needs_freeze() -- scan page to see if any tuples
|
|
|
|
* need to be cleaned to avoid wraparound
|
|
|
|
*
|
|
|
|
* Returns true if the page needs to be vacuumed using cleanup lock.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
lazy_check_needs_freeze(Buffer buf)
|
|
|
|
{
|
|
|
|
Page page;
|
|
|
|
OffsetNumber offnum,
|
|
|
|
maxoff;
|
|
|
|
HeapTupleHeader tupleheader;
|
|
|
|
|
|
|
|
page = BufferGetPage(buf);
|
|
|
|
|
|
|
|
if (PageIsNew(page) || PageIsEmpty(page))
|
|
|
|
{
|
|
|
|
/* PageIsNew probably shouldn't happen... */
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
for (offnum = FirstOffsetNumber;
|
|
|
|
offnum <= maxoff;
|
|
|
|
offnum = OffsetNumberNext(offnum))
|
|
|
|
{
|
|
|
|
ItemId itemid;
|
|
|
|
|
|
|
|
itemid = PageGetItemId(page, offnum);
|
|
|
|
|
|
|
|
if (!ItemIdIsNormal(itemid))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
|
|
|
|
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
if (heap_tuple_needs_freeze(tupleheader, FreezeLimit,
|
2013-09-16 20:45:00 +02:00
|
|
|
MultiXactCutoff, buf))
|
2011-11-08 03:39:40 +01:00
|
|
|
return true;
|
2012-06-10 21:20:04 +02:00
|
|
|
} /* scan along page */
|
2011-11-08 03:39:40 +01:00
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2001-07-18 02:46:25 +02:00
|
|
|
/*
|
2006-05-03 00:25:10 +02:00
|
|
|
* lazy_vacuum_index() -- vacuum one index relation.
|
2001-07-18 02:46:25 +02:00
|
|
|
*
|
2006-05-03 00:25:10 +02:00
|
|
|
* Delete all the index entries pointing to tuples listed in
|
|
|
|
* vacrelstats->dead_tuples, and update running statistics.
|
2001-07-18 02:46:25 +02:00
|
|
|
*/
|
|
|
|
static void
|
2006-05-03 00:25:10 +02:00
|
|
|
lazy_vacuum_index(Relation indrel,
|
|
|
|
IndexBulkDeleteResult **stats,
|
|
|
|
LVRelStats *vacrelstats)
|
2001-07-18 02:46:25 +02:00
|
|
|
{
|
2006-05-03 00:25:10 +02:00
|
|
|
IndexVacuumInfo ivinfo;
|
2005-10-04 00:52:26 +02:00
|
|
|
PGRUsage ru0;
|
2001-07-18 02:46:25 +02:00
|
|
|
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_init(&ru0);
|
2001-07-18 02:46:25 +02:00
|
|
|
|
2006-05-03 00:25:10 +02:00
|
|
|
ivinfo.index = indrel;
|
2009-03-24 21:17:18 +01:00
|
|
|
ivinfo.analyze_only = false;
|
2009-06-07 00:13:52 +02:00
|
|
|
ivinfo.estimated_count = true;
|
2006-05-03 00:25:10 +02:00
|
|
|
ivinfo.message_level = elevel;
|
2009-06-07 00:13:52 +02:00
|
|
|
ivinfo.num_heap_tuples = vacrelstats->old_rel_tuples;
|
2007-05-30 22:12:03 +02:00
|
|
|
ivinfo.strategy = vac_strategy;
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2006-05-03 00:25:10 +02:00
|
|
|
/* Do bulk deletion */
|
|
|
|
*stats = index_bulk_delete(&ivinfo, *stats,
|
|
|
|
lazy_tid_reaped, (void *) vacrelstats);
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(elevel,
|
2006-05-03 00:25:10 +02:00
|
|
|
(errmsg("scanned index \"%s\" to remove %d row versions",
|
2005-10-15 04:49:52 +02:00
|
|
|
RelationGetRelationName(indrel),
|
2006-05-03 00:25:10 +02:00
|
|
|
vacrelstats->num_dead_tuples),
|
|
|
|
errdetail("%s.", pg_rusage_show(&ru0))));
|
2001-07-18 02:46:25 +02:00
|
|
|
}
|
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
/*
|
2006-05-03 00:25:10 +02:00
|
|
|
* lazy_cleanup_index() -- do post-vacuum cleanup for one index relation.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
static void
|
2006-05-03 00:25:10 +02:00
|
|
|
lazy_cleanup_index(Relation indrel,
|
|
|
|
IndexBulkDeleteResult *stats,
|
|
|
|
LVRelStats *vacrelstats)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
2006-05-03 00:25:10 +02:00
|
|
|
IndexVacuumInfo ivinfo;
|
2005-10-04 00:52:26 +02:00
|
|
|
PGRUsage ru0;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_init(&ru0);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2006-05-03 00:25:10 +02:00
|
|
|
ivinfo.index = indrel;
|
2009-03-24 21:17:18 +01:00
|
|
|
ivinfo.analyze_only = false;
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
ivinfo.estimated_count = (vacrelstats->scanned_pages < vacrelstats->rel_pages);
|
2006-05-03 00:25:10 +02:00
|
|
|
ivinfo.message_level = elevel;
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
ivinfo.num_heap_tuples = vacrelstats->new_rel_tuples;
|
2007-05-30 22:12:03 +02:00
|
|
|
ivinfo.strategy = vac_strategy;
|
2003-02-22 01:45:05 +01:00
|
|
|
|
2006-05-03 00:25:10 +02:00
|
|
|
stats = index_vacuum_cleanup(&ivinfo, stats);
|
2003-02-22 01:45:05 +01:00
|
|
|
|
|
|
|
if (!stats)
|
|
|
|
return;
|
|
|
|
|
2009-06-07 00:13:52 +02:00
|
|
|
/*
|
2009-06-11 16:49:15 +02:00
|
|
|
* Now update statistics in pg_class, but only if the index says the count
|
|
|
|
* is accurate.
|
2009-06-07 00:13:52 +02:00
|
|
|
*/
|
|
|
|
if (!stats->estimated_count)
|
|
|
|
vac_update_relstats(indrel,
|
2011-10-14 23:23:01 +02:00
|
|
|
stats->num_pages,
|
|
|
|
stats->num_index_tuples,
|
|
|
|
0,
|
|
|
|
false,
|
Improve concurrency of foreign key locking
This patch introduces two additional lock modes for tuples: "SELECT FOR
KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each
other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
FOR UPDATE". UPDATE commands that do not modify the values stored in
the columns that are part of the key of the tuple now grab a SELECT FOR
NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
with tuple locks of the FOR KEY SHARE variety.
Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
means the concurrency improvement applies to them, which is the whole
point of this patch.
The added tuple lock semantics require some rejiggering of the multixact
module, so that the locking level that each transaction is holding can
be stored alongside its Xid. Also, multixacts now need to persist
across server restarts and crashes, because they can now represent not
only tuple locks, but also tuple updates. This means we need more
careful tracking of lifetime of pg_multixact SLRU files; since they now
persist longer, we require more infrastructure to figure out when they
can be removed. pg_upgrade also needs to be careful to copy
pg_multixact files over from the old server to the new, or at least part
of multixact.c state, depending on the versions of the old and new
servers.
Tuple time qualification rules (HeapTupleSatisfies routines) need to be
careful not to consider tuples with the "is multi" infomask bit set as
being only locked; they might need to look up MultiXact values (i.e.
possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
whereas they previously were assured to only use information readily
available from the tuple header. This is considered acceptable, because
the extra I/O would involve cases that would previously cause some
commands to block waiting for concurrent transactions to finish.
Another important change is the fact that locking tuples that have
previously been updated causes the future versions to be marked as
locked, too; this is essential for correctness of foreign key checks.
This causes additional WAL-logging, also (there was previously a single
WAL record for a locked tuple; now there are as many as updated copies
of the tuple there exist.)
With all this in place, contention related to tuples being checked by
foreign key rules should be much reduced.
As a bonus, the old behavior that a subtransaction grabbing a stronger
tuple lock than the parent (sub)transaction held on a given tuple and
later aborting caused the weaker lock to be lost, has been fixed.
Many new spec files were added for isolation tester framework, to ensure
overall behavior is sane. There's probably room for several more tests.
There were several reviewers of this patch; in particular, Noah Misch
and Andres Freund spent considerable time in it. Original idea for the
patch came from Simon Riggs, after a problem report by Joel Jacobson.
Most code is from me, with contributions from Marti Raudsepp, Alexander
Shulgin, Noah Misch and Andres Freund.
This patch was discussed in several pgsql-hackers threads; the most
important start at the following message-ids:
AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
1290721684-sup-3951@alvh.no-ip.org
1294953201-sup-2099@alvh.no-ip.org
1320343602-sup-2290@alvh.no-ip.org
1339690386-sup-8927@alvh.no-ip.org
4FE5FF020200002500048A3D@gw.wicourts.gov
4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
|
|
|
InvalidTransactionId,
|
2014-10-30 18:03:22 +01:00
|
|
|
InvalidMultiXactId,
|
|
|
|
false);
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
|
2003-07-20 23:56:35 +02:00
|
|
|
ereport(elevel,
|
2005-10-15 04:49:52 +02:00
|
|
|
(errmsg("index \"%s\" now contains %.0f row versions in %u pages",
|
|
|
|
RelationGetRelationName(indrel),
|
|
|
|
stats->num_index_tuples,
|
|
|
|
stats->num_pages),
|
|
|
|
errdetail("%.0f index row versions were removed.\n"
|
|
|
|
"%u index pages have been deleted, %u are currently reusable.\n"
|
|
|
|
"%s.",
|
|
|
|
stats->tuples_removed,
|
|
|
|
stats->pages_deleted, stats->pages_free,
|
|
|
|
pg_rusage_show(&ru0))));
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
|
2003-02-22 01:45:05 +01:00
|
|
|
pfree(stats);
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_truncate_heap - try to truncate off any empty pages at the end
|
|
|
|
*/
|
2010-02-09 22:43:30 +01:00
|
|
|
static void
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
2001-10-25 07:50:21 +02:00
|
|
|
BlockNumber old_rel_pages = vacrelstats->rel_pages;
|
|
|
|
BlockNumber new_rel_pages;
|
2005-10-04 00:52:26 +02:00
|
|
|
PGRUsage ru0;
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
int lock_retry;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2005-10-04 00:52:26 +02:00
|
|
|
pg_rusage_init(&ru0);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/*
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
* Loop until no more truncating can be done.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
do
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
/*
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
* We need full exclusive lock on the relation in order to do
|
|
|
|
* truncation. If we can't get it, give up rather than waiting --- we
|
|
|
|
* don't want to block other backends, and we don't want to deadlock
|
|
|
|
* (which is quite possible considering we already hold a lower-grade
|
|
|
|
* lock).
|
Fix VACUUM so that it always updates pg_class.reltuples/relpages.
When we added the ability for vacuum to skip heap pages by consulting the
visibility map, we made it just not update the reltuples/relpages
statistics if it skipped any pages. But this could leave us with extremely
out-of-date stats for a table that contains any unchanging areas,
especially for TOAST tables which never get processed by ANALYZE. In
particular this could result in autovacuum making poor decisions about when
to process the table, as in recent report from Florian Helmberger. And in
general it's a bad idea to not update the stats at all. Instead, use the
previous values of reltuples/relpages as an estimate of the tuple density
in unvisited pages. This approach results in a "moving average" estimate
of reltuples, which should converge to the correct value over multiple
VACUUM and ANALYZE cycles even when individual measurements aren't very
good.
This new method for updating reltuples is used by both VACUUM and ANALYZE,
with the result that we no longer need the grotty interconnections that
caused ANALYZE to not update the stats depending on what had happened
in the parent VACUUM command.
Also, fix the logic for skipping all-visible pages during VACUUM so that it
looks ahead rather than behind to decide what to do, as per a suggestion
from Greg Stark. This eliminates useless scanning of all-visible pages at
the start of the relation or just after a not-all-visible page. In
particular, the first few pages of the relation will not be invariably
included in the scanned pages, which seems to help in not overweighting
them in the reltuples estimate.
Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
|
|
|
*/
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
vacrelstats->lock_waiter_detected = false;
|
|
|
|
lock_retry = 0;
|
|
|
|
while (true)
|
|
|
|
{
|
|
|
|
if (ConditionalLockRelation(onerel, AccessExclusiveLock))
|
|
|
|
break;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
/*
|
|
|
|
* Check for interrupts while trying to (re-)acquire the exclusive
|
|
|
|
* lock.
|
|
|
|
*/
|
|
|
|
CHECK_FOR_INTERRUPTS();
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2013-04-29 20:05:26 +02:00
|
|
|
if (++lock_retry > (VACUUM_TRUNCATE_LOCK_TIMEOUT /
|
|
|
|
VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL))
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We failed to establish the lock in the specified number of
|
2013-04-29 20:05:26 +02:00
|
|
|
* retries. This means we give up truncating.
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
*/
|
|
|
|
vacrelstats->lock_waiter_detected = true;
|
2013-04-29 20:05:26 +02:00
|
|
|
ereport(elevel,
|
|
|
|
(errmsg("\"%s\": stopping truncate due to conflicting lock request",
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
RelationGetRelationName(onerel))));
|
|
|
|
return;
|
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2013-04-29 20:05:26 +02:00
|
|
|
pg_usleep(VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL);
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
}
|
2008-12-17 10:15:03 +01:00
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
/*
|
|
|
|
* Now that we have exclusive lock, look to see if the rel has grown
|
|
|
|
* whilst we were vacuuming with non-exclusive lock. If so, give up;
|
|
|
|
* the newly added pages presumably contain non-deletable tuples.
|
|
|
|
*/
|
|
|
|
new_rel_pages = RelationGetNumberOfBlocks(onerel);
|
|
|
|
if (new_rel_pages != old_rel_pages)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Note: we intentionally don't update vacrelstats->rel_pages with
|
|
|
|
* the new rel size here. If we did, it would amount to assuming
|
|
|
|
* that the new pages are empty, which is unlikely. Leaving the
|
|
|
|
* numbers alone amounts to assuming that the new pages have the
|
|
|
|
* same tuple density as existing ones, which is less unlikely.
|
|
|
|
*/
|
|
|
|
UnlockRelation(onerel, AccessExclusiveLock);
|
|
|
|
return;
|
|
|
|
}
|
2007-09-10 23:40:03 +02:00
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
/*
|
|
|
|
* Scan backwards from the end to verify that the end pages actually
|
|
|
|
* contain no tuples. This is *necessary*, not optional, because
|
|
|
|
* other backends could have added tuples to these pages whilst we
|
|
|
|
* were vacuuming.
|
|
|
|
*/
|
|
|
|
new_rel_pages = count_nondeletable_pages(onerel, vacrelstats);
|
2004-12-01 20:00:56 +01:00
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
if (new_rel_pages >= old_rel_pages)
|
|
|
|
{
|
|
|
|
/* can't do anything after all */
|
|
|
|
UnlockRelation(onerel, AccessExclusiveLock);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Okay to truncate.
|
|
|
|
*/
|
|
|
|
RelationTruncate(onerel, new_rel_pages);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We can release the exclusive lock as soon as we have truncated.
|
|
|
|
* Other backends can't safely access the relation until they have
|
|
|
|
* processed the smgr invalidation that smgrtruncate sent out ... but
|
|
|
|
* that should happen as part of standard invalidation processing once
|
|
|
|
* they acquire lock on the relation.
|
|
|
|
*/
|
|
|
|
UnlockRelation(onerel, AccessExclusiveLock);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Update statistics. Here, it *is* correct to adjust rel_pages
|
|
|
|
* without also touching reltuples, since the tuple count wasn't
|
|
|
|
* changed by the truncation.
|
|
|
|
*/
|
|
|
|
vacrelstats->pages_removed += old_rel_pages - new_rel_pages;
|
|
|
|
vacrelstats->rel_pages = new_rel_pages;
|
|
|
|
|
|
|
|
ereport(elevel,
|
|
|
|
(errmsg("\"%s\": truncated %u to %u pages",
|
|
|
|
RelationGetRelationName(onerel),
|
|
|
|
old_rel_pages, new_rel_pages),
|
|
|
|
errdetail("%s.",
|
|
|
|
pg_rusage_show(&ru0))));
|
|
|
|
old_rel_pages = new_rel_pages;
|
|
|
|
} while (new_rel_pages > vacrelstats->nonempty_pages &&
|
|
|
|
vacrelstats->lock_waiter_detected);
|
2001-07-14 00:55:59 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2007-09-16 04:37:46 +02:00
|
|
|
* Rescan end pages to verify that they are (still) empty of tuples.
|
2001-07-14 00:55:59 +02:00
|
|
|
*
|
|
|
|
* Returns number of nondeletable pages (last nonempty page + 1).
|
|
|
|
*/
|
|
|
|
static BlockNumber
|
Fix recently-understood problems with handling of XID freezing, particularly
in PITR scenarios. We now WAL-log the replacement of old XIDs with
FrozenTransactionId, so that such replacement is guaranteed to propagate to
PITR slave databases. Also, rather than relying on hint-bit updates to be
preserved, pg_clog is not truncated until all instances of an XID are known to
have been replaced by FrozenTransactionId. Add new GUC variables and
pg_autovacuum columns to allow management of the freezing policy, so that
users can trade off the size of pg_clog against the amount of freezing work
done. Revise the already-existing code that forces autovacuum of tables
approaching the wraparound point to make it more bulletproof; also, revise the
autovacuum logic so that anti-wraparound vacuuming is done per-table rather
than per-database. initdb forced because of changes in pg_class, pg_database,
and pg_autovacuum catalogs. Heikki Linnakangas, Simon Riggs, and Tom Lane.
2006-11-05 23:42:10 +01:00
|
|
|
count_nondeletable_pages(Relation onerel, LVRelStats *vacrelstats)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
BlockNumber blkno;
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
instr_time starttime;
|
|
|
|
|
|
|
|
/* Initialize the starttime if we check for conflicting lock requests */
|
|
|
|
INSTR_TIME_SET_CURRENT(starttime);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/* Strange coding of loop control is needed because blkno is unsigned */
|
|
|
|
blkno = vacrelstats->rel_pages;
|
|
|
|
while (blkno > vacrelstats->nonempty_pages)
|
|
|
|
{
|
|
|
|
Buffer buf;
|
|
|
|
Page page;
|
|
|
|
OffsetNumber offnum,
|
|
|
|
maxoff;
|
2007-09-16 04:37:46 +02:00
|
|
|
bool hastup;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
/*
|
|
|
|
* Check if another process requests a lock on our relation. We are
|
|
|
|
* holding an AccessExclusiveLock here, so they will be waiting. We
|
2013-04-29 20:05:26 +02:00
|
|
|
* only do this once per VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL, and we
|
|
|
|
* only check if that interval has elapsed once every 32 blocks to
|
|
|
|
* keep the number of system calls and actual shared lock table
|
|
|
|
* lookups to a minimum.
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
*/
|
|
|
|
if ((blkno % 32) == 0)
|
|
|
|
{
|
2013-04-29 20:05:26 +02:00
|
|
|
instr_time currenttime;
|
|
|
|
instr_time elapsed;
|
|
|
|
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
INSTR_TIME_SET_CURRENT(currenttime);
|
|
|
|
elapsed = currenttime;
|
|
|
|
INSTR_TIME_SUBTRACT(elapsed, starttime);
|
|
|
|
if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
|
2013-04-29 20:05:26 +02:00
|
|
|
>= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
{
|
|
|
|
if (LockHasWaitersRelation(onerel, AccessExclusiveLock))
|
|
|
|
{
|
|
|
|
ereport(elevel,
|
2013-04-29 20:05:26 +02:00
|
|
|
(errmsg("\"%s\": suspending truncate due to conflicting lock request",
|
Fix performance problems with autovacuum truncation in busy workloads.
In situations where there are over 8MB of empty pages at the end of
a table, the truncation work for trailing empty pages takes longer
than deadlock_timeout, and there is frequent access to the table by
processes other than autovacuum, there was a problem with the
autovacuum worker process being canceled by the deadlock checking
code. The truncation work done by autovacuum up that point was
lost, and the attempt tried again by a later autovacuum worker. The
attempts could continue indefinitely without making progress,
consuming resources and blocking other processes for up to
deadlock_timeout each time.
This patch has the autovacuum worker checking whether it is
blocking any other thread at 20ms intervals. If such a condition
develops, the autovacuum worker will persist the work it has done
so far, release its lock on the table, and sleep in 50ms intervals
for up to 5 seconds, hoping to be able to re-acquire the lock and
try again. If it is unable to get the lock in that time, it moves
on and a worker will try to continue later from the point this one
left off.
While this patch doesn't change the rules about when and what to
truncate, it does cause the truncation to occur sooner, with less
blocking, and with the consumption of fewer resources when there is
contention for the table's lock.
The only user-visible change other than improved performance is
that the table size during truncation may change incrementally
instead of just once.
This problem exists in all supported versions but is infrequently
reported, although some reports of performance problems when
autovacuum runs might be caused by this. Initial commit is just the
master branch, but this should probably be backpatched once the
build farm and general developer usage confirm that there are no
surprising effects.
Jan Wieck
2012-12-11 21:33:08 +01:00
|
|
|
RelationGetRelationName(onerel))));
|
|
|
|
|
|
|
|
vacrelstats->lock_waiter_detected = true;
|
|
|
|
return blkno;
|
|
|
|
}
|
|
|
|
starttime = currenttime;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-09-10 19:58:45 +02:00
|
|
|
/*
|
|
|
|
* We don't insert a vacuum delay point here, because we have an
|
2007-11-15 22:14:46 +01:00
|
|
|
* exclusive lock on the table which we want to hold for as short a
|
|
|
|
* time as possible. We still need to check for interrupts however.
|
2007-09-10 19:58:45 +02:00
|
|
|
*/
|
2007-09-12 04:05:48 +02:00
|
|
|
CHECK_FOR_INTERRUPTS();
|
2004-02-06 20:36:18 +01:00
|
|
|
|
2001-07-14 00:55:59 +02:00
|
|
|
blkno--;
|
|
|
|
|
Unite ReadBufferWithFork, ReadBufferWithStrategy, and ZeroOrReadBuffer
functions into one ReadBufferExtended function, that takes the strategy
and mode as argument. There's three modes, RBM_NORMAL which is the default
used by plain ReadBuffer(), RBM_ZERO, which replaces ZeroOrReadBuffer, and
a new mode RBM_ZERO_ON_ERROR, which allows callers to read corrupt pages
without throwing an error. The FSM needs the new mode to recover from
corrupt pages, which could happend if we crash after extending an FSM file,
and the new page is "torn".
Add fork number to some error messages in bufmgr.c, that still lacked it.
2008-10-31 16:05:00 +01:00
|
|
|
buf = ReadBufferExtended(onerel, MAIN_FORKNUM, blkno,
|
|
|
|
RBM_NORMAL, vac_strategy);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/* In this phase we only need shared access to the buffer */
|
|
|
|
LockBuffer(buf, BUFFER_LOCK_SHARE);
|
|
|
|
|
|
|
|
page = BufferGetPage(buf);
|
|
|
|
|
|
|
|
if (PageIsNew(page) || PageIsEmpty(page))
|
|
|
|
{
|
2004-04-26 01:50:58 +02:00
|
|
|
/* PageIsNew probably shouldn't happen... */
|
2006-04-01 01:32:07 +02:00
|
|
|
UnlockReleaseBuffer(buf);
|
2001-07-14 00:55:59 +02:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
hastup = false;
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
for (offnum = FirstOffsetNumber;
|
|
|
|
offnum <= maxoff;
|
|
|
|
offnum = OffsetNumberNext(offnum))
|
|
|
|
{
|
|
|
|
ItemId itemid;
|
|
|
|
|
|
|
|
itemid = PageGetItemId(page, offnum);
|
|
|
|
|
2007-09-16 04:37:46 +02:00
|
|
|
/*
|
|
|
|
* Note: any non-unused item should be taken as a reason to keep
|
|
|
|
* this page. We formerly thought that DEAD tuples could be
|
|
|
|
* thrown away, but that's not so, because we'd not have cleaned
|
|
|
|
* out their index entries.
|
|
|
|
*/
|
|
|
|
if (ItemIdIsUsed(itemid))
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
|
|
|
hastup = true;
|
|
|
|
break; /* can stop scanning */
|
|
|
|
}
|
2001-10-25 07:50:21 +02:00
|
|
|
} /* scan along page */
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2006-04-01 01:32:07 +02:00
|
|
|
UnlockReleaseBuffer(buf);
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
/* Done scanning if we found a tuple here */
|
|
|
|
if (hastup)
|
|
|
|
return blkno + 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we fall out of the loop, all the previously-thought-to-be-empty
|
2007-09-16 04:37:46 +02:00
|
|
|
* pages still are; we need not bother to look at the last known-nonempty
|
2005-10-15 04:49:52 +02:00
|
|
|
* page.
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
return vacrelstats->nonempty_pages;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_space_alloc - space allocation decisions for lazy vacuum
|
|
|
|
*
|
|
|
|
* See the comments at the head of this file for rationale.
|
|
|
|
*/
|
|
|
|
static void
|
2006-09-13 19:47:08 +02:00
|
|
|
lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
2005-08-21 01:26:37 +02:00
|
|
|
long maxtuples;
|
2014-05-06 18:12:18 +02:00
|
|
|
int vac_work_mem = IsAutoVacuumWorkerProcess() &&
|
|
|
|
autovacuum_work_mem != -1 ?
|
|
|
|
autovacuum_work_mem : maintenance_work_mem;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
2006-09-13 19:47:08 +02:00
|
|
|
if (vacrelstats->hasindex)
|
|
|
|
{
|
2013-12-12 12:42:39 +01:00
|
|
|
maxtuples = (vac_work_mem * 1024L) / sizeof(ItemPointerData);
|
2006-09-04 23:40:23 +02:00
|
|
|
maxtuples = Min(maxtuples, INT_MAX);
|
|
|
|
maxtuples = Min(maxtuples, MaxAllocSize / sizeof(ItemPointerData));
|
2007-09-24 05:52:55 +02:00
|
|
|
|
|
|
|
/* curious coding here to ensure the multiplication can't overflow */
|
|
|
|
if ((BlockNumber) (maxtuples / LAZY_ALLOC_TUPLES) > relblocks)
|
|
|
|
maxtuples = relblocks * LAZY_ALLOC_TUPLES;
|
|
|
|
|
2006-09-04 23:40:23 +02:00
|
|
|
/* stay sane if small maintenance_work_mem */
|
|
|
|
maxtuples = Max(maxtuples, MaxHeapTuplesPerPage);
|
2006-09-13 19:47:08 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2006-09-04 23:40:23 +02:00
|
|
|
maxtuples = MaxHeapTuplesPerPage;
|
|
|
|
}
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
vacrelstats->num_dead_tuples = 0;
|
2005-08-21 01:26:37 +02:00
|
|
|
vacrelstats->max_dead_tuples = (int) maxtuples;
|
2001-07-14 00:55:59 +02:00
|
|
|
vacrelstats->dead_tuples = (ItemPointer)
|
|
|
|
palloc(maxtuples * sizeof(ItemPointerData));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_record_dead_tuple - remember one deletable tuple
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
lazy_record_dead_tuple(LVRelStats *vacrelstats,
|
|
|
|
ItemPointer itemptr)
|
|
|
|
{
|
|
|
|
/*
|
2001-10-25 07:50:21 +02:00
|
|
|
* The array shouldn't overflow under normal behavior, but perhaps it
|
2004-02-03 18:34:04 +01:00
|
|
|
* could if we are given a really small maintenance_work_mem. In that
|
2007-09-20 19:56:33 +02:00
|
|
|
* case, just forget the last few tuples (we'll get 'em next time).
|
2001-07-14 00:55:59 +02:00
|
|
|
*/
|
|
|
|
if (vacrelstats->num_dead_tuples < vacrelstats->max_dead_tuples)
|
|
|
|
{
|
|
|
|
vacrelstats->dead_tuples[vacrelstats->num_dead_tuples] = *itemptr;
|
|
|
|
vacrelstats->num_dead_tuples++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* lazy_tid_reaped() -- is a particular tid deletable?
|
|
|
|
*
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
* This has the right signature to be an IndexBulkDeleteCallback.
|
|
|
|
*
|
2001-07-14 00:55:59 +02:00
|
|
|
* Assumes dead_tuples array is in sorted order.
|
|
|
|
*/
|
|
|
|
static bool
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
lazy_tid_reaped(ItemPointer itemptr, void *state)
|
2001-07-14 00:55:59 +02:00
|
|
|
{
|
Restructure index AM interface for index building and index tuple deletion,
per previous discussion on pghackers. Most of the duplicate code in
different AMs' ambuild routines has been moved out to a common routine
in index.c; this means that all index types now do the right things about
inserting recently-dead tuples, etc. (I also removed support for EXTEND
INDEX in the ambuild routines, since that's about to go away anyway, and
it cluttered the code a lot.) The retail indextuple deletion routines have
been replaced by a "bulk delete" routine in which the indexscan is inside
the access method. I haven't pushed this change as far as it should go yet,
but it should allow considerable simplification of the internal bookkeeping
for deletions. Also, add flag columns to pg_am to eliminate various
hardcoded tests on AM OIDs, and remove unused pg_am columns.
Fix rtree and gist index types to not attempt to store NULLs; before this,
gist usually crashed, while rtree managed not to crash but computed wacko
bounding boxes for NULL entries (which might have had something to do with
the performance problems we've heard about occasionally).
Add AtEOXact routines to hash, rtree, and gist, all of which have static
state that needs to be reset after an error. We discovered this need long
ago for btree, but missed the other guys.
Oh, one more thing: concurrent VACUUM is now the default.
2001-07-16 00:48:19 +02:00
|
|
|
LVRelStats *vacrelstats = (LVRelStats *) state;
|
2001-10-25 07:50:21 +02:00
|
|
|
ItemPointer res;
|
2001-07-14 00:55:59 +02:00
|
|
|
|
|
|
|
res = (ItemPointer) bsearch((void *) itemptr,
|
|
|
|
(void *) vacrelstats->dead_tuples,
|
|
|
|
vacrelstats->num_dead_tuples,
|
|
|
|
sizeof(ItemPointerData),
|
|
|
|
vac_cmp_itemptr);
|
|
|
|
|
|
|
|
return (res != NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Comparator routines for use with qsort() and bsearch().
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
vac_cmp_itemptr(const void *left, const void *right)
|
|
|
|
{
|
|
|
|
BlockNumber lblk,
|
|
|
|
rblk;
|
|
|
|
OffsetNumber loff,
|
|
|
|
roff;
|
|
|
|
|
|
|
|
lblk = ItemPointerGetBlockNumber((ItemPointer) left);
|
|
|
|
rblk = ItemPointerGetBlockNumber((ItemPointer) right);
|
|
|
|
|
|
|
|
if (lblk < rblk)
|
|
|
|
return -1;
|
|
|
|
if (lblk > rblk)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
loff = ItemPointerGetOffsetNumber((ItemPointer) left);
|
|
|
|
roff = ItemPointerGetOffsetNumber((ItemPointer) right);
|
|
|
|
|
|
|
|
if (loff < roff)
|
|
|
|
return -1;
|
|
|
|
if (loff > roff)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2013-02-13 16:46:23 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if every tuple in the given page is visible to all current and future
|
|
|
|
* transactions. Also return the visibility_cutoff_xid which is the highest
|
|
|
|
* xmin amongst the visible tuples.
|
|
|
|
*/
|
|
|
|
static bool
|
2013-07-22 19:26:33 +02:00
|
|
|
heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cutoff_xid)
|
2013-02-13 16:46:23 +01:00
|
|
|
{
|
2013-05-29 22:58:43 +02:00
|
|
|
Page page = BufferGetPage(buf);
|
2013-02-13 16:46:23 +01:00
|
|
|
OffsetNumber offnum,
|
2013-05-29 22:58:43 +02:00
|
|
|
maxoff;
|
|
|
|
bool all_visible = true;
|
2013-02-13 16:46:23 +01:00
|
|
|
|
|
|
|
*visibility_cutoff_xid = InvalidTransactionId;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is a stripped down version of the line pointer scan in
|
2013-05-29 22:58:43 +02:00
|
|
|
* lazy_scan_heap(). So if you change anything here, also check that code.
|
2013-02-13 16:46:23 +01:00
|
|
|
*/
|
|
|
|
maxoff = PageGetMaxOffsetNumber(page);
|
|
|
|
for (offnum = FirstOffsetNumber;
|
2013-05-29 22:58:43 +02:00
|
|
|
offnum <= maxoff && all_visible;
|
|
|
|
offnum = OffsetNumberNext(offnum))
|
2013-02-13 16:46:23 +01:00
|
|
|
{
|
2013-05-29 22:58:43 +02:00
|
|
|
ItemId itemid;
|
|
|
|
HeapTupleData tuple;
|
2013-02-13 16:46:23 +01:00
|
|
|
|
|
|
|
itemid = PageGetItemId(page, offnum);
|
|
|
|
|
|
|
|
/* Unused or redirect line pointers are of no interest */
|
|
|
|
if (!ItemIdIsUsed(itemid) || ItemIdIsRedirected(itemid))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
ItemPointerSet(&(tuple.t_self), BufferGetBlockNumber(buf), offnum);
|
|
|
|
|
|
|
|
/*
|
2013-05-29 22:58:43 +02:00
|
|
|
* Dead line pointers can have index pointers pointing to them. So
|
|
|
|
* they can't be treated as visible
|
2013-02-13 16:46:23 +01:00
|
|
|
*/
|
|
|
|
if (ItemIdIsDead(itemid))
|
|
|
|
{
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
Assert(ItemIdIsNormal(itemid));
|
|
|
|
|
|
|
|
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
|
2013-07-22 19:26:33 +02:00
|
|
|
tuple.t_len = ItemIdGetLength(itemid);
|
|
|
|
tuple.t_tableOid = RelationGetRelid(rel);
|
2013-02-13 16:46:23 +01:00
|
|
|
|
2013-07-22 19:26:33 +02:00
|
|
|
switch (HeapTupleSatisfiesVacuum(&tuple, OldestXmin, buf))
|
2013-02-13 16:46:23 +01:00
|
|
|
{
|
|
|
|
case HEAPTUPLE_LIVE:
|
|
|
|
{
|
|
|
|
TransactionId xmin;
|
|
|
|
|
|
|
|
/* Check comments in lazy_scan_heap. */
|
2013-12-22 21:49:09 +01:00
|
|
|
if (!HeapTupleHeaderXminCommitted(tuple.t_data))
|
2013-02-13 16:46:23 +01:00
|
|
|
{
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2013-05-29 22:58:43 +02:00
|
|
|
* The inserter definitely committed. But is it old enough
|
|
|
|
* that everyone sees it as committed?
|
2013-02-13 16:46:23 +01:00
|
|
|
*/
|
|
|
|
xmin = HeapTupleHeaderGetXmin(tuple.t_data);
|
|
|
|
if (!TransactionIdPrecedes(xmin, OldestXmin))
|
|
|
|
{
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Track newest xmin on page. */
|
|
|
|
if (TransactionIdFollows(xmin, *visibility_cutoff_xid))
|
|
|
|
*visibility_cutoff_xid = xmin;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case HEAPTUPLE_DEAD:
|
|
|
|
case HEAPTUPLE_RECENTLY_DEAD:
|
|
|
|
case HEAPTUPLE_INSERT_IN_PROGRESS:
|
|
|
|
case HEAPTUPLE_DELETE_IN_PROGRESS:
|
|
|
|
all_visible = false;
|
|
|
|
break;
|
|
|
|
|
|
|
|
default:
|
|
|
|
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
|
|
|
|
break;
|
|
|
|
}
|
2013-05-29 22:58:43 +02:00
|
|
|
} /* scan along page */
|
2013-02-13 16:46:23 +01:00
|
|
|
|
|
|
|
return all_visible;
|
|
|
|
}
|