rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/backend/commands/analyze.c

2798 lines
81 KiB
C
Raw Normal View History

Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
/*-------------------------------------------------------------------------
*
* analyze.c
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
* the Postgres statistics generator
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Update copyrights for 2013 Fully update git head, and update back branches in ./COPYRIGHT and legal.sgml files.
2013-01-01 23:15:01 +01:00
* Portions Copyright (c) 1996-2013, PostgreSQL Global Development Group
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
* Portions Copyright (c) 1994, Regents of the University of California
*
*
* IDENTIFICATION
Remove cvs keywords from all files.
2010-09-20 22:08:53 +02:00
* src/backend/commands/analyze.c
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
*-------------------------------------------------------------------------
*/
Avoid repeated detoasting (and possible memory leaks) when processing a toasted datum in VACUUM ANALYZE.
2000-12-02 20:38:34 +01:00
#include "postgres.h"
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
#include <math.h>
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Improve concurrency of foreign key locking This patch introduces two additional lock modes for tuples: "SELECT FOR KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each other, in contrast with already existing "SELECT FOR SHARE" and "SELECT FOR UPDATE". UPDATE commands that do not modify the values stored in the columns that are part of the key of the tuple now grab a SELECT FOR NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently with tuple locks of the FOR KEY SHARE variety. Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this means the concurrency improvement applies to them, which is the whole point of this patch. The added tuple lock semantics require some rejiggering of the multixact module, so that the locking level that each transaction is holding can be stored alongside its Xid. Also, multixacts now need to persist across server restarts and crashes, because they can now represent not only tuple locks, but also tuple updates. This means we need more careful tracking of lifetime of pg_multixact SLRU files; since they now persist longer, we require more infrastructure to figure out when they can be removed. pg_upgrade also needs to be careful to copy pg_multixact files over from the old server to the new, or at least part of multixact.c state, depending on the versions of the old and new servers. Tuple time qualification rules (HeapTupleSatisfies routines) need to be careful not to consider tuples with the "is multi" infomask bit set as being only locked; they might need to look up MultiXact values (i.e. possibly do pg_multixact I/O) to find out the Xid that updated a tuple, whereas they previously were assured to only use information readily available from the tuple header. This is considered acceptable, because the extra I/O would involve cases that would previously cause some commands to block waiting for concurrent transactions to finish. Another important change is the fact that locking tuples that have previously been updated causes the future versions to be marked as locked, too; this is essential for correctness of foreign key checks. This causes additional WAL-logging, also (there was previously a single WAL record for a locked tuple; now there are as many as updated copies of the tuple there exist.) With all this in place, contention related to tuples being checked by foreign key rules should be much reduced. As a bonus, the old behavior that a subtransaction grabbing a stronger tuple lock than the parent (sub)transaction held on a given tuple and later aborting caused the weaker lock to be lost, has been fixed. Many new spec files were added for isolation tester framework, to ensure overall behavior is sane. There's probably room for several more tests. There were several reviewers of this patch; in particular, Noah Misch and Andres Freund spent considerable time in it. Original idea for the patch came from Simon Riggs, after a problem report by Joel Jacobson. Most code is from me, with contributions from Marti Raudsepp, Alexander Shulgin, Noah Misch and Andres Freund. This patch was discussed in several pgsql-hackers threads; the most important start at the following message-ids: AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com 1290721684-sup-3951@alvh.no-ip.org 1294953201-sup-2099@alvh.no-ip.org 1320343602-sup-2290@alvh.no-ip.org 1339690386-sup-8927@alvh.no-ip.org 4FE5FF020200002500048A3D@gw.wicourts.gov 4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
#include "access/multixact.h"
Allow include files to compile own their own. Strip unused include files out unused include files, and add needed includes to C files. The next step is to remove unused include files in C files.
2006-07-13 18:49:20 +02:00
#include "access/transam.h"
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
#include "access/tupconvert.h"
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
#include "access/tuptoaster.h"
Measure the number of all-visible pages for use in index-only scan costing. Add a column pg_class.relallvisible to remember the number of pages that were all-visible according to the visibility map as of the last VACUUM (or ANALYZE, or some other operations that update pg_class.relpages). Use relallvisible/relpages, instead of an arbitrary constant, to estimate how many heap page fetches can be avoided during an index-only scan. This is pretty primitive and will no doubt see refinements once we've acquired more field experience with the index-only scan mechanism, but it's way better than using a constant. Note: I had to adjust an underspecified query in the window.sql regression test, because it was changing answers when the plan changed to use an index-only scan. Some of the adjacent tests perhaps should be adjusted as well, but I didn't do that here.
2011-10-14 23:23:01 +02:00
#include "access/visibilitymap.h"
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
#include "access/xact.h"
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
#include "catalog/index.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "catalog/indexing.h"
Make all comparisons done for/with statistics use the default collation. While this will give wrong answers when estimating selectivity for a comparison operator that's using a non-default collation, the estimation error probably won't be large; and anyway the former approach created estimation errors of its own by trying to use a histogram that might have been computed with some other collation. So we'll adopt this simplified approach for now and perhaps improve it sometime in the future. This patch incorporates changes from Andres Freund to make sure that selfuncs.c passes a valid collation OID to any datatype-specific function it calls, in case that function wants collation information. Said OID will now always be DEFAULT_COLLATION_OID, but at least we won't get errors.
2011-03-12 22:30:36 +01:00
#include "catalog/pg_collation.h"
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
#include "catalog/pg_inherits_fn.h"
Improve error messages emitted when VACUUM and ANALYZE skip a table. Per gripe from Clodoaldo Pinto Neto on Message-ID: <a595de7a0801060326qbfc790ax2a60573043c2e2be@mail.gmail.com>
2008-02-20 15:31:35 +01:00
#include "catalog/pg_namespace.h"
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
#include "commands/dbcommands.h"
Teach ANALYZE to clear pg_class.relhassubclass when appropriate. In the past, relhassubclass always remained true if a relation had ever had child relations, even if the last subclass was long gone. While this had only marginal performance implications in most cases, it was annoying, and I'm now considering some planner changes that would raise the cost of a false positive. It was previously impractical to fix this because of race condition concerns. However, given the recent change that made tablecmds.c take ShareExclusiveLock on relations that are gaining a child (commit fbcf4b92aa64d4577bcf25925b055316b978744a), we can now allow ANALYZE to clear the flag when it's no longer relevant. There is no additional locking cost to do so, since ANALYZE takes ShareExclusiveLock anyway.
2011-09-02 20:29:31 +02:00
#include "commands/tablecmds.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "commands/vacuum.h"
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
#include "executor/executor.h"
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
#include "foreign/fdwapi.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "miscadmin.h"
Move exprType(), exprTypmod(), expression_tree_walker(), and related routines into nodes/nodeFuncs, so as to reduce wanton cross-subsystem #includes inside the backend. There's probably more that should be done along this line, but this is a start anyway.
2008-08-26 00:42:34 +02:00
#include "nodes/nodeFuncs.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "parser/parse_oper.h"
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
2002-08-02 20:15:10 +02:00
#include "parser/parse_relation.h"
Integrate autovacuum functionality into the backend. There's still a few loose ends to be dealt with, but it seems to work. Alvaro Herrera, based on the contrib code by Matthew O'Connor.
2005-07-14 07:13:45 +02:00
#include "pgstat.h"
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
#include "postmaster/autovacuum.h"
Move BufferGetPageSize and BufferGetPage from bufpage.h to bufmgr.h. It is more logical that way, and also it reduces the amount of unnecessary includes in bufpage.h, which is widely used. Zdenek Kotala. My previous patch to bufpage.h should also have credited him as author, but I forgot (sorry about that).
2008-06-09 00:00:48 +02:00
#include "storage/bufmgr.h"
Avoid having autovacuum workers wait for relation locks. Waiting for relation locks can lead to starvation - it pins down an autovacuum worker for as long as the lock is held. But if we're doing an anti-wraparound vacuum, then we still wait; maintenance can no longer be put off. To assist with troubleshooting, if log_autovacuum_min_duration >= 0, we log whenever an autovacuum or autoanalyze is skipped for this reason. Per a gripe by Josh Berkus, and ensuing discussion.
2011-02-08 04:04:29 +01:00
#include "storage/lmgr.h"
Clean up the #include mess a little. walsender.h should depend on xlog.h, not vice versa. (Actually, the inclusion was circular until a couple hours ago, which was even sillier; but Bruce broke it in the expedient rather than logically correct direction.) Because of that poor decision, plus blind application of pgrminclude, we had a situation where half the system was depending on xlog.h to include such unrelated stuff as array.h and guc.h. Clean up the header inclusion, and manually revert a lot of what pgrminclude had done so things build again. This episode reinforces my feeling that pgrminclude should not be run without adult supervision. Inclusion changes in header files in particular need to be reviewed with great care. More generally, it'd be good if we had a clearer notion of module layering to dictate which headers can sanely include which others ... but that's a big task for another day.
2011-09-04 07:13:16 +02:00
#include "storage/proc.h"
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
#include "storage/procarray.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "utils/acl.h"
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
#include "utils/attoptcache.h"
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
#include "utils/datum.h"
Clean up the #include mess a little. walsender.h should depend on xlog.h, not vice versa. (Actually, the inclusion was circular until a couple hours ago, which was even sillier; but Bruce broke it in the expedient rather than logically correct direction.) Because of that poor decision, plus blind application of pgrminclude, we had a situation where half the system was depending on xlog.h to include such unrelated stuff as array.h and guc.h. Clean up the header inclusion, and manually revert a lot of what pgrminclude had done so things build again. This episode reinforces my feeling that pgrminclude should not be run without adult supervision. Inclusion changes in header files in particular need to be reviewed with great care. More generally, it'd be good if we had a clearer notion of module layering to dictate which headers can sanely include which others ... but that's a big task for another day.
2011-09-04 07:13:16 +02:00
#include "utils/guc.h"
Make VACUUM handle schema-qualified relation names properly.
2002-04-02 03:03:07 +02:00
#include "utils/lsyscache.h"
For some reason access/tupmacs.h has been #including utils/memutils.h, which is neither needed by nor related to that header. Remove the bogus inclusion and instead include the header in those C files that actually need it. Also fix unnecessary inclusions and bad inclusion order in tsearch2 files.
2005-05-06 19:24:55 +02:00
#include "utils/memutils.h"
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
#include "utils/pg_rusage.h"
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
#include "utils/sortsupport.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
#include "utils/syscache.h"
Move Timestamp/Interval typedefs and basic macros into datatype/timestamp.h. As per my recent proposal, this refactors things so that these typedefs and macros are available in a header that can be included in frontend-ish code. I also changed various headers that were undesirably including utils/timestamp.h to include datatype/timestamp.h instead. Unsurprisingly, this showed that half the system was getting utils/timestamp.h by way of xlog.h. No actual code changes here, just header refactoring.
2011-09-09 19:23:41 +02:00
#include "utils/timestamp.h"
Move the HTSU_Result enum definition into snapshot.h, to avoid including tqual.h into heapam.h. This makes all inclusion of tqual.h explicit. I also sorted alphabetically the includes on some source files.
2008-03-26 22:10:39 +01:00
#include "utils/tqual.h"
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
{
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
BlockNumber N; /* number of blocks, known in advance */
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
int n; /* desired sample size */
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
BlockNumber t; /* current block number */
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
int m; /* blocks selected so far */
} BlockSamplerData;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
typedef BlockSamplerData *BlockSampler;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/* Per-index data for ANALYZE */
typedef struct AnlIndexData
{
IndexInfo *indexInfo; /* BuildIndexInfo result */
double tupleFract; /* fraction of rows for partial index */
VacAttrStats **vacattrstats; /* index attrs to analyze */
int attr_cnt;
} AnlIndexData;
Instead of having a configure-time DEFAULT_ATTSTATTARGET, store -1 in attstattarget to indicate 'use the default'. The default is now a GUC variable default_statistics_target, and so may be changed on the fly. Along the way we gain the ability to have pg_dump dump the per-column statistics target when it's not the default. Patch by Neil Conway, with some kibitzing from Tom Lane.
2002-07-31 19:19:54 +02:00
/* Default statistics target (GUC parameter) */
Increase the default value of default_statistics_target from 10 to 100, and its maximum value from 1000 to 10000. ALTER TABLE SET STATISTICS similarly now allows a value up to 10000. Per discussion.
2008-12-13 20:13:44 +01:00
int default_statistics_target = 100;
Instead of having a configure-time DEFAULT_ATTSTATTARGET, store -1 in attstattarget to indicate 'use the default'. The default is now a GUC variable default_statistics_target, and so may be changed on the fly. Along the way we gain the ability to have pg_dump dump the per-column statistics target when it's not the default. Patch by Neil Conway, with some kibitzing from Tom Lane.
2002-07-31 19:19:54 +02:00
Make large sequential scans and VACUUMs work in a limited-size "ring" of buffers, rather than blowing out the whole shared-buffer arena. Aside from avoiding cache spoliation, this fixes the problem that VACUUM formerly tended to cause a WAL flush for every page it modified, because we had it hacked to use only a single buffer. Those flushes will now occur only once per ring-ful. The exact ring size, and the threshold for seqscans to switch into the ring usage pattern, remain under debate; but the infrastructure seems done. The key bit of infrastructure is a new optional BufferAccessStrategy object that can be passed to ReadBuffer operations; this replaces the former StrategyHintVacuum API. This patch also changes the buffer usage-count methodology a bit: we now advance usage_count when first pinning a buffer, rather than when last unpinning it. To preserve the behavior that a buffer's lifetime starts to decrease when it's released, the clock sweep code is modified to not decrement usage_count of pinned buffers. Work not done in this commit: teach GiST and GIN indexes to use the vacuum BufferAccessStrategy for vacuum-driven fetches. Original patch by Simon, reworked by Heikki and again by Tom.
2007-05-30 22:12:03 +02:00
/* A few variables that don't seem worth passing around as parameters */
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
static MemoryContext anl_context = NULL;
Make large sequential scans and VACUUMs work in a limited-size "ring" of buffers, rather than blowing out the whole shared-buffer arena. Aside from avoiding cache spoliation, this fixes the problem that VACUUM formerly tended to cause a WAL flush for every page it modified, because we had it hacked to use only a single buffer. Those flushes will now occur only once per ring-ful. The exact ring size, and the threshold for seqscans to switch into the ring usage pattern, remain under debate; but the infrastructure seems done. The key bit of infrastructure is a new optional BufferAccessStrategy object that can be passed to ReadBuffer operations; this replaces the former StrategyHintVacuum API. This patch also changes the buffer usage-count methodology a bit: we now advance usage_count when first pinning a buffer, rather than when last unpinning it. To preserve the behavior that a buffer's lifetime starts to decrease when it's released, the clock sweep code is modified to not decrement usage_count of pinned buffers. Work not done in this commit: teach GiST and GIN indexes to use the vacuum BufferAccessStrategy for vacuum-driven fetches. Original patch by Simon, reworked by Heikki and again by Tom.
2007-05-30 22:12:03 +02:00
static BufferAccessStrategy vac_strategy;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
static void do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
AcquireSampleRowsFunc acquirefunc, BlockNumber relpages,
bool inh, int elevel);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
static void BlockSampler_Init(BlockSampler bs, BlockNumber nblocks,
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
int samplesize);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
static bool BlockSampler_HasMore(BlockSampler bs);
static BlockNumber BlockSampler_Next(BlockSampler bs);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
static void compute_index_stats(Relation onerel, double totalrows,
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
AnlIndexData *indexdata, int nindexes,
HeapTuple *rows, int numrows,
MemoryContext col_context);
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
static VacAttrStats *examine_attribute(Relation onerel, int attnum,
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
Node *index_expr);
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
static int acquire_sample_rows(Relation onerel, int elevel,
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
HeapTuple *rows, int targrows,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
double *totalrows, double *totaldeadrows);
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
static int compare_rows(const void *a, const void *b);
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
static int acquire_inherited_sample_rows(Relation onerel, int elevel,
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
HeapTuple *rows, int targrows,
double *totalrows, double *totaldeadrows);
static void update_attstats(Oid relid, bool inh,
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
int natts, VacAttrStats **vacattrstats);
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
/*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* analyze_rel() -- analyze one relation
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
void
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Relation onerel;
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
int elevel;
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
AcquireSampleRowsFunc acquirefunc = NULL;
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
BlockNumber relpages = 0;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
/* Select logging level */
Provide a parenthesized-options syntax for VACUUM, analogous to that recently adopted for EXPLAIN. This will allow additional options to be implemented in future without having to make them fully-reserved keywords. The old syntax remains available for existing options, however. Itagaki Takahiro
2009-11-16 22:32:07 +01:00
if (vacstmt->options & VACOPT_VERBOSE)
Commit to match discussed elog() changes. Only update is that LOG is now just below FATAL in server_min_messages. Added more text to highlight ordering difference between it and client_min_messages. --------------------------------------------------------------------------- REALLYFATAL => PANIC STOP => PANIC New INFO level the prints to client by default New LOG level the prints to server log by default Cause VACUUM information to print only to the client NOTICE => INFO where purely information messages are sent DEBUG => LOG for purely server status messages DEBUG removed, kept as backward compatible DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1 added DebugLvl removed in favor of new DEBUG[1-5] symbols New server_min_messages GUC parameter with values: DEBUG[5-1], INFO, NOTICE, ERROR, LOG, FATAL, PANIC New client_min_messages GUC parameter with values: DEBUG[5-1], LOG, INFO, NOTICE, ERROR, FATAL, PANIC Server startup now logged with LOG instead of DEBUG Remove debug_level GUC parameter elog() numbers now start at 10 Add test to print error message if older elog() values are passed to elog() Bootstrap mode now has a -d that requires an argument, like postmaster
2002-03-02 22:39:36 +01:00
elevel = INFO;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
else
Make debug_ GUC varables output DEBUG1 rather than LOG, and mention in docs that CLIENT/LOG_MIN_MESSAGES now controls debug_* output location. Doc changes included.
2003-05-27 19:49:47 +02:00
elevel = DEBUG2;
Change made to elog: o Change all current CVS messages of NOTICE to WARNING. We were going to do this just before 7.3 beta but it has to be done now, as you will see below. o Change current INFO messages that should be controlled by client_min_messages to NOTICE. o Force remaining INFO messages, like from EXPLAIN, VACUUM VERBOSE, etc. to always go to the client. o Remove INFO from the client_min_messages options and add NOTICE. Seems we do need three non-ERROR elog levels to handle the various behaviors we need for these messages. Regression passed.
2002-03-06 07:10:59 +01:00
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
/* Set up static variables */
Make large sequential scans and VACUUMs work in a limited-size "ring" of buffers, rather than blowing out the whole shared-buffer arena. Aside from avoiding cache spoliation, this fixes the problem that VACUUM formerly tended to cause a WAL flush for every page it modified, because we had it hacked to use only a single buffer. Those flushes will now occur only once per ring-ful. The exact ring size, and the threshold for seqscans to switch into the ring usage pattern, remain under debate; but the infrastructure seems done. The key bit of infrastructure is a new optional BufferAccessStrategy object that can be passed to ReadBuffer operations; this replaces the former StrategyHintVacuum API. This patch also changes the buffer usage-count methodology a bit: we now advance usage_count when first pinning a buffer, rather than when last unpinning it. To preserve the behavior that a buffer's lifetime starts to decrease when it's released, the clock sweep code is modified to not decrement usage_count of pinned buffers. Work not done in this commit: teach GiST and GIN indexes to use the vacuum BufferAccessStrategy for vacuum-driven fetches. Original patch by Simon, reworked by Heikki and again by Tom.
2007-05-30 22:12:03 +02:00
vac_strategy = bstrategy;
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
/*
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
* Check for user-requested abort.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
Restructure backend SIGINT/SIGTERM handling so that 'die' interrupts are treated more like 'cancel' interrupts: the signal handler sets a flag that is examined at well-defined spots, rather than trying to cope with an interrupt that might happen anywhere. See pghackers discussion of 1/12/01.
2001-01-14 06:08:17 +01:00
CHECK_FOR_INTERRUPTS();
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
/*
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
* Open the relation, getting ShareUpdateExclusiveLock to ensure that two
* ANALYZEs don't run on it concurrently. (This also locks out a
* concurrent VACUUM, which doesn't matter much at the moment but might
* matter if we ever try to accumulate stats on dead tuples.) If the rel
* has been dropped since we last saw it, we don't need to process it.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
Avoid having autovacuum workers wait for relation locks. Waiting for relation locks can lead to starvation - it pins down an autovacuum worker for as long as the lock is held. But if we're doing an anti-wraparound vacuum, then we still wait; maintenance can no longer be put off. To assist with troubleshooting, if log_autovacuum_min_duration >= 0, we log whenever an autovacuum or autoanalyze is skipped for this reason. Per a gripe by Josh Berkus, and ensuing discussion.
2011-02-08 04:04:29 +01:00
if (!(vacstmt->options & VACOPT_NOWAIT))
onerel = try_relation_open(relid, ShareUpdateExclusiveLock);
else if (ConditionalLockRelationOid(relid, ShareUpdateExclusiveLock))
onerel = try_relation_open(relid, NoLock);
else
{
onerel = NULL;
if (IsAutoVacuumWorkerProcess() && Log_autovacuum_min_duration >= 0)
ereport(LOG,
(errcode(ERRCODE_LOCK_NOT_AVAILABLE),
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
errmsg("skipping analyze of \"%s\" --- lock not available",
vacstmt->relation->relname)));
Avoid having autovacuum workers wait for relation locks. Waiting for relation locks can lead to starvation - it pins down an autovacuum worker for as long as the lock is held. But if we're doing an anti-wraparound vacuum, then we still wait; maintenance can no longer be put off. To assist with troubleshooting, if log_autovacuum_min_duration >= 0, we log whenever an autovacuum or autoanalyze is skipped for this reason. Per a gripe by Josh Berkus, and ensuing discussion.
2011-02-08 04:04:29 +01:00
}
Now that we've rearranged relation open to get a lock before touching the rel, it's easy to get rid of the narrow race-condition window that used to exist in VACUUM and CLUSTER. Did some minor code-beautification work in the same area, too.
2006-08-18 18:09:13 +02:00
if (!onerel)
Change SearchSysCache coding conventions so that a reference count is maintained for each cache entry. A cache entry will not be freed until the matching ReleaseSysCache call has been executed. This eliminates worries about cache entries getting dropped while still in use. See my posting to pg-hackers of even date for more info.
2000-11-16 23:30:52 +01:00
return;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
/*
Now that we've rearranged relation open to get a lock before touching the rel, it's easy to get rid of the narrow race-condition window that used to exist in VACUUM and CLUSTER. Did some minor code-beautification work in the same area, too.
2006-08-18 18:09:13 +02:00
* Check permissions --- this should match vacuum's check!
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
Make VACUUM handle schema-qualified relation names properly.
2002-04-02 03:03:07 +02:00
if (!(pg_class_ownercheck(RelationGetRelid(onerel), GetUserId()) ||
First batch of object rename commands.
2003-06-27 16:45:32 +02:00
(pg_database_ownercheck(MyDatabaseId, GetUserId()) && !onerel->rd_rel->relisshared)))
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Make VACUUM handle schema-qualified relation names properly.
2002-04-02 03:03:07 +02:00
/* No need for a WARNING if we already complained during VACUUM */
Provide a parenthesized-options syntax for VACUUM, analogous to that recently adopted for EXPLAIN. This will allow additional options to be implemented in future without having to make them fully-reserved keywords. The old syntax remains available for existing options, however. Itagaki Takahiro
2009-11-16 22:32:07 +01:00
if (!(vacstmt->options & VACOPT_VACUUM))
Improve error messages emitted when VACUUM and ANALYZE skip a table. Per gripe from Clodoaldo Pinto Neto on Message-ID: <a595de7a0801060326qbfc790ax2a60573043c2e2be@mail.gmail.com>
2008-02-20 15:31:35 +01:00
{
if (onerel->rd_rel->relisshared)
ereport(WARNING,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
(errmsg("skipping \"%s\" --- only superuser can analyze it",
RelationGetRelationName(onerel))));
Improve error messages emitted when VACUUM and ANALYZE skip a table. Per gripe from Clodoaldo Pinto Neto on Message-ID: <a595de7a0801060326qbfc790ax2a60573043c2e2be@mail.gmail.com>
2008-02-20 15:31:35 +01:00
else if (onerel->rd_rel->relnamespace == PG_CATALOG_NAMESPACE)
ereport(WARNING,
(errmsg("skipping \"%s\" --- only superuser or database owner can analyze it",
RelationGetRelationName(onerel))));
else
ereport(WARNING,
(errmsg("skipping \"%s\" --- only table or database owner can analyze it",
RelationGetRelationName(onerel))));
}
Change ANALYZE to take ShareUpdateExclusiveLock not AccessShareLock on the table being analyzed. This prevents two ANALYZEs from running concurrently on the same table and possibly suffering concurrent-update failures while trying to store their results into pg_statistic. The downside is that a database-wide ANALYZE executed within a transaction block will hold ShareUpdateExclusiveLock on many tables simultaneously, which could lead to concurrency issues or even deadlock against another such ANALYZE. However, this seems a corner case of less importance than getting unexpected errors from a foreground ANALYZE when autovacuum elects to analyze the same table concurrently. Per discussion.
2006-09-18 00:50:31 +02:00
relation_close(onerel, ShareUpdateExclusiveLock);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
return;
}
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
/*
* Silently ignore tables that are temp tables of other backends ---
* trying to analyze these is rather pointless, since their contents are
* probably not up-to-date on disk. (We don't throw a warning here; it
* would just lead to chatter during a database-wide ANALYZE.)
*/
if (RELATION_IS_OTHER_TEMP(onerel))
{
relation_close(onerel, ShareUpdateExclusiveLock);
return;
}
/*
* We can ANALYZE any table except pg_statistic. See update_attstats
*/
if (RelationGetRelid(onerel) == StatisticRelationId)
{
relation_close(onerel, ShareUpdateExclusiveLock);
return;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Add a materialized view relations. A materialized view has a rule just like a view and a heap and other physical properties like a table. The rule is only used to populate the table, references in queries refer to the materialized data. This is a minimal implementation, but should still be useful in many cases. Currently data is only populated "on demand" by the CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW statements. It is expected that future releases will add incremental updates with various timings, and that a more refined concept of defining what is "fresh" data will be developed. At some point it may even be possible to have queries use a materialized in place of references to underlying tables, but that requires the other above-mentioned features to be working first. Much of the documentation work by Robert Haas. Review by Noah Misch, Thom Brown, Robert Haas, Marko Tiikkaja Security review by KaiGai Kohei, with a decision on how best to implement sepgsql still pending.
2013-03-04 01:23:31 +01:00
* Check that it's a plain table, materialized view, or foreign table; we
* used to do this in get_rel_oids() but seems safer to check after we've
* locked the relation.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Add a materialized view relations. A materialized view has a rule just like a view and a heap and other physical properties like a table. The rule is only used to populate the table, references in queries refer to the materialized data. This is a minimal implementation, but should still be useful in many cases. Currently data is only populated "on demand" by the CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW statements. It is expected that future releases will add incremental updates with various timings, and that a more refined concept of defining what is "fresh" data will be developed. At some point it may even be possible to have queries use a materialized in place of references to underlying tables, but that requires the other above-mentioned features to be working first. Much of the documentation work by Robert Haas. Review by Noah Misch, Thom Brown, Robert Haas, Marko Tiikkaja Security review by KaiGai Kohei, with a decision on how best to implement sepgsql still pending.
2013-03-04 01:23:31 +01:00
if (onerel->rd_rel->relkind == RELKIND_RELATION ||
onerel->rd_rel->relkind == RELKIND_MATVIEW)
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
{
/* Regular table, so we'll use the regular row acquisition function */
acquirefunc = acquire_sample_rows;
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
/* Also get regular table's size */
relpages = RelationGetNumberOfBlocks(onerel);
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
}
else if (onerel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
/*
* For a foreign table, call the FDW's hook function to see whether it
* supports analysis.
*/
FdwRoutine *fdwroutine;
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
bool ok = false;
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
Arrange to cache FdwRoutine structs in foreign tables' relcache entries. This saves several catalog lookups per reference. It's not all that exciting right now, because we'd managed to minimize the number of places that need to fetch the data; but the upcoming writable-foreign-tables patch needs this info in a lot more places.
2013-03-07 05:47:38 +01:00
fdwroutine = GetFdwRoutineForRelation(onerel, false);
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
if (fdwroutine->AnalyzeForeignTable != NULL)
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
ok = fdwroutine->AnalyzeForeignTable(onerel,
&acquirefunc,
&relpages);
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
if (!ok)
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
{
ereport(WARNING,
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
(errmsg("skipping \"%s\" --- cannot analyze this foreign table",
RelationGetRelationName(onerel))));
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
relation_close(onerel, ShareUpdateExclusiveLock);
return;
}
}
else
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Change made to elog: o Change all current CVS messages of NOTICE to WARNING. We were going to do this just before 7.3 beta but it has to be done now, as you will see below. o Change current INFO messages that should be controlled by client_min_messages to NOTICE. o Force remaining INFO messages, like from EXPLAIN, VACUUM VERBOSE, etc. to always go to the client. o Remove INFO from the client_min_messages options and add NOTICE. Seems we do need three non-ERROR elog levels to handle the various behaviors we need for these messages. Regression passed.
2002-03-06 07:10:59 +01:00
/* No need for a WARNING if we already complained during VACUUM */
Provide a parenthesized-options syntax for VACUUM, analogous to that recently adopted for EXPLAIN. This will allow additional options to be implemented in future without having to make them fully-reserved keywords. The old syntax remains available for existing options, however. Itagaki Takahiro
2009-11-16 22:32:07 +01:00
if (!(vacstmt->options & VACOPT_VACUUM))
Another round of error message editing, covering backend/commands/.
2003-07-20 23:56:35 +02:00
ereport(WARNING,
Basic foreign table support. Foreign tables are a core component of SQL/MED. This commit does not provide a working SQL/MED infrastructure, because foreign tables cannot yet be queried. Support for foreign table scans will need to be added in a future patch. However, this patch creates the necessary system catalog structure, syntax support, and support for ancillary operations such as COMMENT and SECURITY LABEL. Shigeru Hanada, heavily revised by Robert Haas
2011-01-02 05:48:11 +01:00
(errmsg("skipping \"%s\" --- cannot analyze non-tables or special system tables",
Another round of error message editing, covering backend/commands/.
2003-07-20 23:56:35 +02:00
RelationGetRelationName(onerel))));
Change ANALYZE to take ShareUpdateExclusiveLock not AccessShareLock on the table being analyzed. This prevents two ANALYZEs from running concurrently on the same table and possibly suffering concurrent-update failures while trying to store their results into pg_statistic. The downside is that a database-wide ANALYZE executed within a transaction block will hold ShareUpdateExclusiveLock on many tables simultaneously, which could lead to concurrency issues or even deadlock against another such ANALYZE. However, this seems a corner case of less importance than getting unexpected errors from a foreground ANALYZE when autovacuum elects to analyze the same table concurrently. Per discussion.
2006-09-18 00:50:31 +02:00
relation_close(onerel, ShareUpdateExclusiveLock);
Make VACUUM handle schema-qualified relation names properly.
2002-04-02 03:03:07 +02:00
return;
}
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/*
* OK, let's do it. First let other backends know I'm in ANALYZE.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
Move "hot" members of PGPROC into a separate PGXACT array. This speeds up snapshot-taking and reduces ProcArrayLock contention. Also, the PGPROC (and PGXACT) structures used by two-phase commit are now allocated as part of the main array, rather than in a separate array, and we keep ProcArray sorted in pointer order. These changes are intended to minimize the number of cache lines that must be pulled in to take a snapshot, and testing shows a substantial increase in performance on both read and write workloads at high concurrencies. Pavan Deolasee, Heikki Linnakangas, Robert Haas
2011-11-25 14:02:10 +01:00
MyPgXact->vacuumFlags |= PROC_IN_ANALYZE;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
LWLockRelease(ProcArrayLock);
/*
* Do the normal non-recursive ANALYZE.
*/
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
do_analyze_rel(onerel, vacstmt, acquirefunc, relpages, false, elevel);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/*
* If there are child tables, do recursive ANALYZE.
*/
if (onerel->rd_rel->relhassubclass)
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
do_analyze_rel(onerel, vacstmt, acquirefunc, relpages, true, elevel);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/*
* Close source relation now, but keep lock so that no one deletes it
* before we commit. (If someone did, they'd fail to clean up the entries
* we made in pg_statistic. Also, releasing the lock before commit would
* expose us to concurrent-update failures in update_attstats.)
*/
relation_close(onerel, NoLock);
/*
Update comments that became out-of-date with the PGXACT struct. When the "hot" members of PGPROC were split off to separate PGXACT structs, many PGPROC fields referred to in comments were moved to PGXACT, but the comments were neglected in the commit. Mostly this is just a search/replace of PGPROC with PGXACT, but the way the dummy PGPROC entries are created for prepared transactions changed more, making some of the comments totally bogus. Noah Misch
2012-05-14 09:22:44 +02:00
* Reset my PGXACT flag. Note: we need this here, and not in vacuum_rel,
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
* because the vacuum flag is cleared by the end-of-xact code.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
Move "hot" members of PGPROC into a separate PGXACT array. This speeds up snapshot-taking and reduces ProcArrayLock contention. Also, the PGPROC (and PGXACT) structures used by two-phase commit are now allocated as part of the main array, rather than in a separate array, and we keep ProcArray sorted in pointer order. These changes are intended to minimize the number of cache lines that must be pulled in to take a snapshot, and testing shows a substantial increase in performance on both read and write workloads at high concurrencies. Pavan Deolasee, Heikki Linnakangas, Robert Haas
2011-11-25 14:02:10 +01:00
MyPgXact->vacuumFlags &= ~PROC_IN_ANALYZE;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
LWLockRelease(ProcArrayLock);
}
/*
* do_analyze_rel() -- analyze one relation, recursively or not
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
*
* Note that "acquirefunc" is only relevant for the non-inherited case.
* If we supported foreign tables in inheritance trees,
* acquire_inherited_sample_rows would need to determine the appropriate
* acquirefunc for each child table.
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
*/
static void
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
AcquireSampleRowsFunc acquirefunc, BlockNumber relpages,
bool inh, int elevel)
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
{
int attr_cnt,
tcnt,
i,
ind;
Relation *Irel;
int nindexes;
bool hasindex;
VacAttrStats **vacattrstats;
AnlIndexData *indexdata;
int targrows,
numrows;
double totalrows,
totaldeadrows;
HeapTuple *rows;
PGRUsage ru0;
TimestampTz starttime = 0;
MemoryContext caller_context;
Oid save_userid;
int save_sec_context;
int save_nestlevel;
if (inh)
ereport(elevel,
(errmsg("analyzing \"%s.%s\" inheritance tree",
get_namespace_name(RelationGetNamespace(onerel)),
RelationGetRelationName(onerel))));
else
ereport(elevel,
(errmsg("analyzing \"%s.%s\"",
get_namespace_name(RelationGetNamespace(onerel)),
RelationGetRelationName(onerel))));
/*
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* Set up a working context so that we can easily free whatever junk gets
* created.
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
*/
anl_context = AllocSetContextCreate(CurrentMemoryContext,
"Analyze",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
caller_context = MemoryContextSwitchTo(anl_context);
Make standard maintenance operations (including VACUUM, ANALYZE, REINDEX, and CLUSTER) execute as the table owner rather than the calling user, using the same privilege-switching mechanism already used for SECURITY DEFINER functions. The purpose of this change is to ensure that user-defined functions used in index definitions cannot acquire the privileges of a superuser account that is performing routine maintenance. While a function used in an index is supposed to be IMMUTABLE and thus not able to do anything very interesting, there are several easy ways around that restriction; and even if we could plug them all, there would remain a risk of reading sensitive information and broadcasting it through a covert channel such as CPU usage. To prevent bypassing this security measure, execution of SET SESSION AUTHORIZATION and SET ROLE is now forbidden within a SECURITY DEFINER context. Thanks to Itagaki Takahiro for reporting this vulnerability. Security: CVE-2007-6600
2008-01-03 22:23:15 +01:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Switch to the table owner's userid, so that any index functions are run
Prevent indirect security attacks via changing session-local state within an allegedly immutable index function. It was previously recognized that we had to prevent such a function from executing SET/RESET ROLE/SESSION AUTHORIZATION, or it could trivially obtain the privileges of the session user. However, since there is in general no privilege checking for changes of session-local state, it is also possible for such a function to change settings in a way that might subvert later operations in the same session. Examples include changing search_path to cause an unexpected function to be called, or replacing an existing prepared statement with another one that will execute a function of the attacker's choosing. The present patch secures VACUUM, ANALYZE, and CREATE INDEX/REINDEX against these threats, which are the same places previously deemed to need protection against the SET ROLE issue. GUC changes are still allowed, since there are many useful cases for that, but we prevent security problems by forcing a rollback of any GUC change after completing the operation. Other cases are handled by throwing an error if any change is attempted; these include temp table creation, closing a cursor, and creating or deleting a prepared statement. (In 7.4, the infrastructure to roll back GUC changes doesn't exist, so we settle for rejecting changes of "search_path" in these contexts.) Original report and patch by Gurjeet Singh, additional analysis by Tom Lane. Security: CVE-2009-4136
2009-12-09 22:57:51 +01:00
* as that user. Also lock down security-restricted operations and
* arrange to make GUC variable changes local to this command.
Make standard maintenance operations (including VACUUM, ANALYZE, REINDEX, and CLUSTER) execute as the table owner rather than the calling user, using the same privilege-switching mechanism already used for SECURITY DEFINER functions. The purpose of this change is to ensure that user-defined functions used in index definitions cannot acquire the privileges of a superuser account that is performing routine maintenance. While a function used in an index is supposed to be IMMUTABLE and thus not able to do anything very interesting, there are several easy ways around that restriction; and even if we could plug them all, there would remain a risk of reading sensitive information and broadcasting it through a covert channel such as CPU usage. To prevent bypassing this security measure, execution of SET SESSION AUTHORIZATION and SET ROLE is now forbidden within a SECURITY DEFINER context. Thanks to Itagaki Takahiro for reporting this vulnerability. Security: CVE-2007-6600
2008-01-03 22:23:15 +01:00
*/
Prevent indirect security attacks via changing session-local state within an allegedly immutable index function. It was previously recognized that we had to prevent such a function from executing SET/RESET ROLE/SESSION AUTHORIZATION, or it could trivially obtain the privileges of the session user. However, since there is in general no privilege checking for changes of session-local state, it is also possible for such a function to change settings in a way that might subvert later operations in the same session. Examples include changing search_path to cause an unexpected function to be called, or replacing an existing prepared statement with another one that will execute a function of the attacker's choosing. The present patch secures VACUUM, ANALYZE, and CREATE INDEX/REINDEX against these threats, which are the same places previously deemed to need protection against the SET ROLE issue. GUC changes are still allowed, since there are many useful cases for that, but we prevent security problems by forcing a rollback of any GUC change after completing the operation. Other cases are handled by throwing an error if any change is attempted; these include temp table creation, closing a cursor, and creating or deleting a prepared statement. (In 7.4, the infrastructure to roll back GUC changes doesn't exist, so we settle for rejecting changes of "search_path" in these contexts.) Original report and patch by Gurjeet Singh, additional analysis by Tom Lane. Security: CVE-2009-4136
2009-12-09 22:57:51 +01:00
GetUserIdAndSecContext(&save_userid, &save_sec_context);
SetUserIdAndSecContext(onerel->rd_rel->relowner,
save_sec_context | SECURITY_RESTRICTED_OPERATION);
save_nestlevel = NewGUCNestLevel();
Make standard maintenance operations (including VACUUM, ANALYZE, REINDEX, and CLUSTER) execute as the table owner rather than the calling user, using the same privilege-switching mechanism already used for SECURITY DEFINER functions. The purpose of this change is to ensure that user-defined functions used in index definitions cannot acquire the privileges of a superuser account that is performing routine maintenance. While a function used in an index is supposed to be IMMUTABLE and thus not able to do anything very interesting, there are several easy ways around that restriction; and even if we could plug them all, there would remain a risk of reading sensitive information and broadcasting it through a covert channel such as CPU usage. To prevent bypassing this security measure, execution of SET SESSION AUTHORIZATION and SET ROLE is now forbidden within a SECURITY DEFINER context. Thanks to Itagaki Takahiro for reporting this vulnerability. Security: CVE-2007-6600
2008-01-03 22:23:15 +01:00
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
/* measure elapsed time iff autovacuum logging requires it */
Simplify and rename some GUC variables, per various recent discussions: * stats_start_collector goes away; we always start the collector process, unless prevented by a problem with setting up the stats UDP socket. * stats_reset_on_server_start goes away; it seems useless in view of the availability of pg_stat_reset(). * stats_block_level and stats_row_level are merged into a single variable "track_counts", which controls all reports sent to the collector process. * stats_command_string is renamed to track_activities. * log_autovacuum is renamed to log_autovacuum_min_duration to better reflect its meaning. The log_autovacuum change is not a compatibility issue since it didn't exist before 8.3 anyway. The other changes need to be release-noted.
2007-09-24 05:12:23 +02:00
if (IsAutoVacuumWorkerProcess() && Log_autovacuum_min_duration >= 0)
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
{
pg_rusage_init(&ru0);
Simplify and rename some GUC variables, per various recent discussions: * stats_start_collector goes away; we always start the collector process, unless prevented by a problem with setting up the stats UDP socket. * stats_reset_on_server_start goes away; it seems useless in view of the availability of pg_stat_reset(). * stats_block_level and stats_row_level are merged into a single variable "track_counts", which controls all reports sent to the collector process. * stats_command_string is renamed to track_activities. * log_autovacuum is renamed to log_autovacuum_min_duration to better reflect its meaning. The log_autovacuum change is not a compatibility issue since it didn't exist before 8.3 anyway. The other changes need to be release-noted.
2007-09-24 05:12:23 +02:00
if (Log_autovacuum_min_duration > 0)
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
starttime = GetCurrentTimestamp();
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* Determine which columns to analyze
*
* Note that system attributes are never analyzed.
*/
if (vacstmt->va_cols != NIL)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Reimplement the linked list data structure used throughout the backend. In the past, we used a 'Lispy' linked list implementation: a "list" was merely a pointer to the head node of the list. The problem with that design is that it makes lappend() and length() linear time. This patch fixes that problem (and others) by maintaining a count of the list length and a pointer to the tail node along with each head node pointer. A "list" is now a pointer to a structure containing some meta-data about the list; the head and tail pointers in that structure refer to ListCell structures that maintain the actual linked list of nodes. The function names of the list API have also been changed to, I hope, be more logically consistent. By default, the old function names are still available; they will be disabled-by-default once the rest of the tree has been updated to use the new API names.
2004-05-26 06:41:50 +02:00
ListCell *le;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Reimplement the linked list data structure used throughout the backend. In the past, we used a 'Lispy' linked list implementation: a "list" was merely a pointer to the head node of the list. The problem with that design is that it makes lappend() and length() linear time. This patch fixes that problem (and others) by maintaining a count of the list length and a pointer to the tail node along with each head node pointer. A "list" is now a pointer to a structure containing some meta-data about the list; the head and tail pointers in that structure refer to ListCell structures that maintain the actual linked list of nodes. The function names of the list API have also been changed to, I hope, be more logically consistent. By default, the old function names are still available; they will be disabled-by-default once the rest of the tree has been updated to use the new API names.
2004-05-26 06:41:50 +02:00
vacattrstats = (VacAttrStats **) palloc(list_length(vacstmt->va_cols) *
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
sizeof(VacAttrStats *));
tcnt = 0;
foreach(le, vacstmt->va_cols)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
char *col = strVal(lfirst(le));
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
2002-08-02 20:15:10 +02:00
i = attnameAttNum(onerel, col, false);
Add error location info to ResTarget parse nodes. Allows error cursor to be supplied for various mistakes involving INSERT and UPDATE target columns.
2006-03-23 01:19:30 +01:00
if (i == InvalidAttrNumber)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
pgindent run for 8.2.
2006-10-04 02:30:14 +02:00
errmsg("column \"%s\" of relation \"%s\" does not exist",
col, RelationGetRelationName(onerel))));
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
vacattrstats[tcnt] = examine_attribute(onerel, i, NULL);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (vacattrstats[tcnt] != NULL)
tcnt++;
}
attr_cnt = tcnt;
}
else
{
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
2002-08-02 20:15:10 +02:00
attr_cnt = onerel->rd_att->natts;
Tweak palloc/repalloc to allow zero bytes to be requested, as per recent proposal. Eliminate several dozen now-unnecessary hacks to avoid palloc(0). (It's likely there are more that I didn't find.)
2004-06-05 21:48:09 +02:00
vacattrstats = (VacAttrStats **)
palloc(attr_cnt * sizeof(VacAttrStats *));
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
tcnt = 0;
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
2002-08-02 20:15:10 +02:00
for (i = 1; i <= attr_cnt; i++)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
vacattrstats[tcnt] = examine_attribute(onerel, i, NULL);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (vacattrstats[tcnt] != NULL)
tcnt++;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
attr_cnt = tcnt;
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Open all indexes of the relation, and see if there are any analyzable
* columns in the indexes. We do not analyze index columns if there was
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
* an explicit column list in the ANALYZE command, however. If we are
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* doing a recursive scan, we don't want to touch the parent's indexes at
* all.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
if (!inh)
vac_open_indexes(onerel, AccessShareLock, &nindexes, &Irel);
else
{
Irel = NULL;
nindexes = 0;
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
hasindex = (nindexes > 0);
indexdata = NULL;
if (hasindex)
{
indexdata = (AnlIndexData *) palloc0(nindexes * sizeof(AnlIndexData));
for (ind = 0; ind < nindexes; ind++)
{
AnlIndexData *thisdata = &indexdata[ind];
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
IndexInfo *indexInfo;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
thisdata->indexInfo = indexInfo = BuildIndexInfo(Irel[ind]);
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
thisdata->tupleFract = 1.0; /* fix later if partial */
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
if (indexInfo->ii_Expressions != NIL && vacstmt->va_cols == NIL)
{
Reimplement the linked list data structure used throughout the backend. In the past, we used a 'Lispy' linked list implementation: a "list" was merely a pointer to the head node of the list. The problem with that design is that it makes lappend() and length() linear time. This patch fixes that problem (and others) by maintaining a count of the list length and a pointer to the tail node along with each head node pointer. A "list" is now a pointer to a structure containing some meta-data about the list; the head and tail pointers in that structure refer to ListCell structures that maintain the actual linked list of nodes. The function names of the list API have also been changed to, I hope, be more logically consistent. By default, the old function names are still available; they will be disabled-by-default once the rest of the tree has been updated to use the new API names.
2004-05-26 06:41:50 +02:00
ListCell *indexpr_item = list_head(indexInfo->ii_Expressions);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
thisdata->vacattrstats = (VacAttrStats **)
palloc(indexInfo->ii_NumIndexAttrs * sizeof(VacAttrStats *));
tcnt = 0;
for (i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
{
int keycol = indexInfo->ii_KeyAttrNumbers[i];
if (keycol == 0)
{
/* Found an index expression */
Node *indexkey;
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (indexpr_item == NULL) /* shouldn't happen */
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
elog(ERROR, "too few entries in indexprs list");
Reimplement the linked list data structure used throughout the backend. In the past, we used a 'Lispy' linked list implementation: a "list" was merely a pointer to the head node of the list. The problem with that design is that it makes lappend() and length() linear time. This patch fixes that problem (and others) by maintaining a count of the list length and a pointer to the tail node along with each head node pointer. A "list" is now a pointer to a structure containing some meta-data about the list; the head and tail pointers in that structure refer to ListCell structures that maintain the actual linked list of nodes. The function names of the list API have also been changed to, I hope, be more logically consistent. By default, the old function names are still available; they will be disabled-by-default once the rest of the tree has been updated to use the new API names.
2004-05-26 06:41:50 +02:00
indexkey = (Node *) lfirst(indexpr_item);
indexpr_item = lnext(indexpr_item);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
thisdata->vacattrstats[tcnt] =
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
examine_attribute(Irel[ind], i + 1, indexkey);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
if (thisdata->vacattrstats[tcnt] != NULL)
tcnt++;
}
}
thisdata->attr_cnt = tcnt;
}
}
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Determine how many rows we need to sample, using the worst case from
* all analyzable columns. We use a lower bound of 100 rows to avoid
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
targrows = 100;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
for (i = 0; i < attr_cnt; i++)
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (targrows < vacattrstats[i]->minrows)
targrows = vacattrstats[i]->minrows;
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
for (ind = 0; ind < nindexes; ind++)
{
AnlIndexData *thisdata = &indexdata[ind];
for (i = 0; i < thisdata->attr_cnt; i++)
{
if (targrows < thisdata->vacattrstats[i]->minrows)
targrows = thisdata->vacattrstats[i]->minrows;
}
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* Acquire the sample rows
*/
rows = (HeapTuple *) palloc(targrows * sizeof(HeapTuple));
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
if (inh)
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
numrows = acquire_inherited_sample_rows(onerel, elevel,
rows, targrows,
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
&totalrows, &totaldeadrows);
else
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
numrows = (*acquirefunc) (onerel, elevel,
rows, targrows,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
&totalrows, &totaldeadrows);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Compute the statistics. Temporary results during the calculations for
* each column are stored in a child context. The calc routines are
* responsible to make sure that whatever they store into the VacAttrStats
* structure is allocated in anl_context.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (numrows > 0)
{
MemoryContext col_context,
old_context;
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
col_context = AllocSetContextCreate(anl_context,
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
"Analyze Column",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
old_context = MemoryContextSwitchTo(col_context);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
for (i = 0; i < attr_cnt; i++)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
VacAttrStats *stats = vacattrstats[i];
Collect and use element-frequency statistics for arrays. This patch improves selectivity estimation for the array <@, &&, and @> (containment and overlaps) operators. It enables collection of statistics about individual array element values by ANALYZE, and introduces operator-specific estimators that use these stats. In addition, ScalarArrayOpExpr constructs of the forms "const = ANY/ALL (array_column)" and "const <> ANY/ALL (array_column)" are estimated by treating them as variants of the containment operators. Since we still collect scalar-style stats about the array values as a whole, the pg_stats view is expanded to show both these stats and the array-style stats in separate columns. This creates an incompatible change in how stats for tsvector columns are displayed in pg_stats: the stats about lexemes are now displayed in the array-related columns instead of the original scalar-related columns. There are a few loose ends here, notably that it'd be nice to be able to suppress either the scalar-style stats or the array-element stats for columns for which they're not useful. But the patch is in good enough shape to commit for wider testing. Alexander Korotkov, reviewed by Noah Misch and Nathan Boley
2012-03-04 02:20:19 +01:00
AttributeOpts *aopt;
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
stats->rows = rows;
stats->tupDesc = onerel->rd_att;
(*stats->compute_stats) (stats,
std_fetch_func,
numrows,
totalrows);
Add ALTER TABLE ... ALTER COLUMN ... SET STATISTICS DISTINCT Robert Haas
2009-08-03 00:14:53 +02:00
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
/*
* If the appropriate flavor of the n_distinct option is
* specified, override with the corresponding value.
*/
Collect and use element-frequency statistics for arrays. This patch improves selectivity estimation for the array <@, &&, and @> (containment and overlaps) operators. It enables collection of statistics about individual array element values by ANALYZE, and introduces operator-specific estimators that use these stats. In addition, ScalarArrayOpExpr constructs of the forms "const = ANY/ALL (array_column)" and "const <> ANY/ALL (array_column)" are estimated by treating them as variants of the containment operators. Since we still collect scalar-style stats about the array values as a whole, the pg_stats view is expanded to show both these stats and the array-style stats in separate columns. This creates an incompatible change in how stats for tsvector columns are displayed in pg_stats: the stats about lexemes are now displayed in the array-related columns instead of the original scalar-related columns. There are a few loose ends here, notably that it'd be nice to be able to suppress either the scalar-style stats or the array-element stats for columns for which they're not useful. But the patch is in good enough shape to commit for wider testing. Alexander Korotkov, reviewed by Noah Misch and Nathan Boley
2012-03-04 02:20:19 +01:00
aopt = get_attribute_options(onerel->rd_id, stats->attr->attnum);
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
if (aopt != NULL)
{
Collect and use element-frequency statistics for arrays. This patch improves selectivity estimation for the array <@, &&, and @> (containment and overlaps) operators. It enables collection of statistics about individual array element values by ANALYZE, and introduces operator-specific estimators that use these stats. In addition, ScalarArrayOpExpr constructs of the forms "const = ANY/ALL (array_column)" and "const <> ANY/ALL (array_column)" are estimated by treating them as variants of the containment operators. Since we still collect scalar-style stats about the array values as a whole, the pg_stats view is expanded to show both these stats and the array-style stats in separate columns. This creates an incompatible change in how stats for tsvector columns are displayed in pg_stats: the stats about lexemes are now displayed in the array-related columns instead of the original scalar-related columns. There are a few loose ends here, notably that it'd be nice to be able to suppress either the scalar-style stats or the array-element stats for columns for which they're not useful. But the patch is in good enough shape to commit for wider testing. Alexander Korotkov, reviewed by Noah Misch and Nathan Boley
2012-03-04 02:20:19 +01:00
float8 n_distinct;
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
Collect and use element-frequency statistics for arrays. This patch improves selectivity estimation for the array <@, &&, and @> (containment and overlaps) operators. It enables collection of statistics about individual array element values by ANALYZE, and introduces operator-specific estimators that use these stats. In addition, ScalarArrayOpExpr constructs of the forms "const = ANY/ALL (array_column)" and "const <> ANY/ALL (array_column)" are estimated by treating them as variants of the containment operators. Since we still collect scalar-style stats about the array values as a whole, the pg_stats view is expanded to show both these stats and the array-style stats in separate columns. This creates an incompatible change in how stats for tsvector columns are displayed in pg_stats: the stats about lexemes are now displayed in the array-related columns instead of the original scalar-related columns. There are a few loose ends here, notably that it'd be nice to be able to suppress either the scalar-style stats or the array-element stats for columns for which they're not useful. But the patch is in good enough shape to commit for wider testing. Alexander Korotkov, reviewed by Noah Misch and Nathan Boley
2012-03-04 02:20:19 +01:00
n_distinct = inh ? aopt->n_distinct_inherited : aopt->n_distinct;
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
if (n_distinct != 0.0)
stats->stadistinct = n_distinct;
}
Add ALTER TABLE ... ALTER COLUMN ... SET STATISTICS DISTINCT Robert Haas
2009-08-03 00:14:53 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
MemoryContextResetAndDeleteChildren(col_context);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
if (hasindex)
compute_index_stats(onerel, totalrows,
indexdata, nindexes,
rows, numrows,
col_context);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
MemoryContextSwitchTo(old_context);
MemoryContextDelete(col_context);
/*
* Emit the completed stats rows into pg_statistic, replacing any
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* previous statistics for the target columns. (If there are stats in
* pg_statistic for columns we didn't process, we leave them alone.)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
update_attstats(RelationGetRelid(onerel), inh,
attr_cnt, vacattrstats);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
for (ind = 0; ind < nindexes; ind++)
{
AnlIndexData *thisdata = &indexdata[ind];
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
update_attstats(RelationGetRelid(Irel[ind]), false,
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
thisdata->attr_cnt, thisdata->vacattrstats);
}
}
/*
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
* Update pages/tuples stats in pg_class ... but not if we're doing
* inherited stats.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
if (!inh)
Make relhasrules and relhastriggers work like relhasindex, namely we let VACUUM reset them to false rather than trying to clean 'em up during DROP.
2008-11-10 01:49:37 +01:00
vac_update_relstats(onerel,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
relpages,
Measure the number of all-visible pages for use in index-only scan costing. Add a column pg_class.relallvisible to remember the number of pages that were all-visible according to the visibility map as of the last VACUUM (or ANALYZE, or some other operations that update pg_class.relpages). Use relallvisible/relpages, instead of an arbitrary constant, to estimate how many heap page fetches can be avoided during an index-only scan. This is pretty primitive and will no doubt see refinements once we've acquired more field experience with the index-only scan mechanism, but it's way better than using a constant. Note: I had to adjust an underspecified query in the window.sql regression test, because it was changing answers when the plan changed to use an index-only scan. Some of the adjacent tests perhaps should be adjusted as well, but I didn't do that here.
2011-10-14 23:23:01 +02:00
totalrows,
visibilitymap_count(onerel),
hasindex,
Improve concurrency of foreign key locking This patch introduces two additional lock modes for tuples: "SELECT FOR KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each other, in contrast with already existing "SELECT FOR SHARE" and "SELECT FOR UPDATE". UPDATE commands that do not modify the values stored in the columns that are part of the key of the tuple now grab a SELECT FOR NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently with tuple locks of the FOR KEY SHARE variety. Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this means the concurrency improvement applies to them, which is the whole point of this patch. The added tuple lock semantics require some rejiggering of the multixact module, so that the locking level that each transaction is holding can be stored alongside its Xid. Also, multixacts now need to persist across server restarts and crashes, because they can now represent not only tuple locks, but also tuple updates. This means we need more careful tracking of lifetime of pg_multixact SLRU files; since they now persist longer, we require more infrastructure to figure out when they can be removed. pg_upgrade also needs to be careful to copy pg_multixact files over from the old server to the new, or at least part of multixact.c state, depending on the versions of the old and new servers. Tuple time qualification rules (HeapTupleSatisfies routines) need to be careful not to consider tuples with the "is multi" infomask bit set as being only locked; they might need to look up MultiXact values (i.e. possibly do pg_multixact I/O) to find out the Xid that updated a tuple, whereas they previously were assured to only use information readily available from the tuple header. This is considered acceptable, because the extra I/O would involve cases that would previously cause some commands to block waiting for concurrent transactions to finish. Another important change is the fact that locking tuples that have previously been updated causes the future versions to be marked as locked, too; this is essential for correctness of foreign key checks. This causes additional WAL-logging, also (there was previously a single WAL record for a locked tuple; now there are as many as updated copies of the tuple there exist.) With all this in place, contention related to tuples being checked by foreign key rules should be much reduced. As a bonus, the old behavior that a subtransaction grabbing a stronger tuple lock than the parent (sub)transaction held on a given tuple and later aborting caused the weaker lock to be lost, has been fixed. Many new spec files were added for isolation tester framework, to ensure overall behavior is sane. There's probably room for several more tests. There were several reviewers of this patch; in particular, Noah Misch and Andres Freund spent considerable time in it. Original idea for the patch came from Simon Riggs, after a problem report by Joel Jacobson. Most code is from me, with contributions from Marti Raudsepp, Alexander Shulgin, Noah Misch and Andres Freund. This patch was discussed in several pgsql-hackers threads; the most important start at the following message-ids: AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com 1290721684-sup-3951@alvh.no-ip.org 1294953201-sup-2099@alvh.no-ip.org 1320343602-sup-2290@alvh.no-ip.org 1339690386-sup-8927@alvh.no-ip.org 4FE5FF020200002500048A3D@gw.wicourts.gov 4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
InvalidTransactionId,
InvalidMultiXactId);
Improve vacuum code to track minimum Xids per table instead of per database. To this end, add a couple of columns to pg_class, relminxid and relvacuumxid, based on which we calculate the pg_database columns after each vacuum. We now force all databases to be vacuumed, even template ones. A backend noticing too old a database (meaning pg_database.datminxid is in danger of falling behind Xid wraparound) will signal the postmaster, which in turn will start an autovacuum iteration to process the offending database. In principle this is only there to cope with frozen (non-connectable) databases without forcing users to set them to connectable, but it could force regular user database to go through a database-wide vacuum at any time. Maybe we should warn users about this somehow. Of course the real solution will be to use autovacuum all the time ;-) There are some additional improvements we could have in this area: for example the vacuum code could be smarter about not updating pg_database for each table when called by autovacuum, and do it only once the whole autovacuum iteration is done. I updated the system catalogs documentation, but I didn't modify the maintenance section. Also having some regression tests for this would be nice but it's not really a very straightforward thing to do. Catalog version bumped due to system catalog changes.
2006-07-10 18:20:52 +02:00
Don't reset pg_class.reltuples and relpages in VACUUM, if any pages were skipped. We could update relpages anyway, but it seems better to only update it together with reltuples, because we use the reltuples/relpages ratio in the planner. Also don't update n_live_tuples in pgstat. ANALYZE in VACUUM ANALYZE now needs to update pg_class, if the VACUUM-phase didn't do so. Added some boolean-passing to let analyze_rel know if it should update pg_class or not. I also moved the relcache invalidation (to update rd_targblock) from vac_update_relstats to where RelationTruncate is called, because vac_update_relstats is not called for partial vacuums anymore. It's more obvious to send the invalidation close to the truncation that requires it. Per report by Ned T. Crigler.
2008-12-17 10:15:03 +01:00
/*
* Same for indexes. Vacuum always scans all indexes, so if we're part of
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* VACUUM ANALYZE, don't overwrite the accurate count already inserted by
Don't reset pg_class.reltuples and relpages in VACUUM, if any pages were skipped. We could update relpages anyway, but it seems better to only update it together with reltuples, because we use the reltuples/relpages ratio in the planner. Also don't update n_live_tuples in pgstat. ANALYZE in VACUUM ANALYZE now needs to update pg_class, if the VACUUM-phase didn't do so. Added some boolean-passing to let analyze_rel know if it should update pg_class or not. I also moved the relcache invalidation (to update rd_targblock) from vac_update_relstats to where RelationTruncate is called, because vac_update_relstats is not called for partial vacuums anymore. It's more obvious to send the invalidation close to the truncation that requires it. Per report by Ned T. Crigler.
2008-12-17 10:15:03 +01:00
* VACUUM.
*/
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
if (!inh && !(vacstmt->options & VACOPT_VACUUM))
Don't reset pg_class.reltuples and relpages in VACUUM, if any pages were skipped. We could update relpages anyway, but it seems better to only update it together with reltuples, because we use the reltuples/relpages ratio in the planner. Also don't update n_live_tuples in pgstat. ANALYZE in VACUUM ANALYZE now needs to update pg_class, if the VACUUM-phase didn't do so. Added some boolean-passing to let analyze_rel know if it should update pg_class or not. I also moved the relcache invalidation (to update rd_targblock) from vac_update_relstats to where RelationTruncate is called, because vac_update_relstats is not called for partial vacuums anymore. It's more obvious to send the invalidation close to the truncation that requires it. Per report by Ned T. Crigler.
2008-12-17 10:15:03 +01:00
{
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
for (ind = 0; ind < nindexes; ind++)
{
AnlIndexData *thisdata = &indexdata[ind];
double totalindexrows;
totalindexrows = ceil(thisdata->tupleFract * totalrows);
Make relhasrules and relhastriggers work like relhasindex, namely we let VACUUM reset them to false rather than trying to clean 'em up during DROP.
2008-11-10 01:49:37 +01:00
vac_update_relstats(Irel[ind],
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
RelationGetNumberOfBlocks(Irel[ind]),
Measure the number of all-visible pages for use in index-only scan costing. Add a column pg_class.relallvisible to remember the number of pages that were all-visible according to the visibility map as of the last VACUUM (or ANALYZE, or some other operations that update pg_class.relpages). Use relallvisible/relpages, instead of an arbitrary constant, to estimate how many heap page fetches can be avoided during an index-only scan. This is pretty primitive and will no doubt see refinements once we've acquired more field experience with the index-only scan mechanism, but it's way better than using a constant. Note: I had to adjust an underspecified query in the window.sql regression test, because it was changing answers when the plan changed to use an index-only scan. Some of the adjacent tests perhaps should be adjusted as well, but I didn't do that here.
2011-10-14 23:23:01 +02:00
totalindexrows,
0,
false,
Improve concurrency of foreign key locking This patch introduces two additional lock modes for tuples: "SELECT FOR KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each other, in contrast with already existing "SELECT FOR SHARE" and "SELECT FOR UPDATE". UPDATE commands that do not modify the values stored in the columns that are part of the key of the tuple now grab a SELECT FOR NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently with tuple locks of the FOR KEY SHARE variety. Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this means the concurrency improvement applies to them, which is the whole point of this patch. The added tuple lock semantics require some rejiggering of the multixact module, so that the locking level that each transaction is holding can be stored alongside its Xid. Also, multixacts now need to persist across server restarts and crashes, because they can now represent not only tuple locks, but also tuple updates. This means we need more careful tracking of lifetime of pg_multixact SLRU files; since they now persist longer, we require more infrastructure to figure out when they can be removed. pg_upgrade also needs to be careful to copy pg_multixact files over from the old server to the new, or at least part of multixact.c state, depending on the versions of the old and new servers. Tuple time qualification rules (HeapTupleSatisfies routines) need to be careful not to consider tuples with the "is multi" infomask bit set as being only locked; they might need to look up MultiXact values (i.e. possibly do pg_multixact I/O) to find out the Xid that updated a tuple, whereas they previously were assured to only use information readily available from the tuple header. This is considered acceptable, because the extra I/O would involve cases that would previously cause some commands to block waiting for concurrent transactions to finish. Another important change is the fact that locking tuples that have previously been updated causes the future versions to be marked as locked, too; this is essential for correctness of foreign key checks. This causes additional WAL-logging, also (there was previously a single WAL record for a locked tuple; now there are as many as updated copies of the tuple there exist.) With all this in place, contention related to tuples being checked by foreign key rules should be much reduced. As a bonus, the old behavior that a subtransaction grabbing a stronger tuple lock than the parent (sub)transaction held on a given tuple and later aborting caused the weaker lock to be lost, has been fixed. Many new spec files were added for isolation tester framework, to ensure overall behavior is sane. There's probably room for several more tests. There were several reviewers of this patch; in particular, Noah Misch and Andres Freund spent considerable time in it. Original idea for the patch came from Simon Riggs, after a problem report by Joel Jacobson. Most code is from me, with contributions from Marti Raudsepp, Alexander Shulgin, Noah Misch and Andres Freund. This patch was discussed in several pgsql-hackers threads; the most important start at the following message-ids: AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com 1290721684-sup-3951@alvh.no-ip.org 1294953201-sup-2099@alvh.no-ip.org 1320343602-sup-2290@alvh.no-ip.org 1339690386-sup-8927@alvh.no-ip.org 4FE5FF020200002500048A3D@gw.wicourts.gov 4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
InvalidTransactionId,
InvalidMultiXactId);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Revise pgstat's tracking of tuple changes to improve the reliability of decisions about when to auto-analyze. The previous code depended on n_live_tuples + n_dead_tuples - last_anl_tuples, where all three of these numbers could be bad estimates from ANALYZE itself. Even worse, in the presence of a steady flow of HOT updates and matching HOT-tuple reclamations, auto-analyze might never trigger at all, even if all three numbers are exactly right, because n_dead_tuples could hold steady. To fix, replace last_anl_tuples with an accurately tracked count of the total number of committed tuple inserts + updates + deletes since the last ANALYZE on the table. This can still be compared to the same threshold as before, but it's much more trustworthy than the old computation. Tracking this requires one more intra-transaction counter per modified table within backends, but no additional memory space in the stats collector. There probably isn't any measurable speed difference; if anything it might be a bit faster than before, since I was able to eliminate some per-tuple arithmetic operations in favor of adding sums once per (sub)transaction. Also, simplify the logic around pgstat vacuum and analyze reporting messages by not trying to fold VACUUM ANALYZE into a single pgstat message. The original thought behind this patch was to allow scheduling of analyzes on parent tables by artificially inflating their changes_since_analyze count. I've left that for a separate patch since this change seems to stand on its own merit.
2009-12-30 21:32:14 +01:00
/*
Pgindent run before 9.1 beta2.
2011-06-09 20:32:50 +02:00
* Report ANALYZE to the stats collector, too. However, if doing
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
* inherited stats we shouldn't report, because the stats collector only
* tracks per-table stats.
Revise pgstat's tracking of tuple changes to improve the reliability of decisions about when to auto-analyze. The previous code depended on n_live_tuples + n_dead_tuples - last_anl_tuples, where all three of these numbers could be bad estimates from ANALYZE itself. Even worse, in the presence of a steady flow of HOT updates and matching HOT-tuple reclamations, auto-analyze might never trigger at all, even if all three numbers are exactly right, because n_dead_tuples could hold steady. To fix, replace last_anl_tuples with an accurately tracked count of the total number of committed tuple inserts + updates + deletes since the last ANALYZE on the table. This can still be compared to the same threshold as before, but it's much more trustworthy than the old computation. Tracking this requires one more intra-transaction counter per modified table within backends, but no additional memory space in the stats collector. There probably isn't any measurable speed difference; if anything it might be a bit faster than before, since I was able to eliminate some per-tuple arithmetic operations in favor of adding sums once per (sub)transaction. Also, simplify the logic around pgstat vacuum and analyze reporting messages by not trying to fold VACUUM ANALYZE into a single pgstat message. The original thought behind this patch was to allow scheduling of analyzes on parent tables by artificially inflating their changes_since_analyze count. I've left that for a separate patch since this change seems to stand on its own merit.
2009-12-30 21:32:14 +01:00
*/
Dept of second thoughts: recursive case in ANALYZE shouldn't emit a pgstats message. This might need to be done differently later, but with the current logic that's what should happen.
2009-12-30 22:21:33 +01:00
if (!inh)
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
pgstat_report_analyze(onerel, totalrows, totaldeadrows);
Revise pgstat's tracking of tuple changes to improve the reliability of decisions about when to auto-analyze. The previous code depended on n_live_tuples + n_dead_tuples - last_anl_tuples, where all three of these numbers could be bad estimates from ANALYZE itself. Even worse, in the presence of a steady flow of HOT updates and matching HOT-tuple reclamations, auto-analyze might never trigger at all, even if all three numbers are exactly right, because n_dead_tuples could hold steady. To fix, replace last_anl_tuples with an accurately tracked count of the total number of committed tuple inserts + updates + deletes since the last ANALYZE on the table. This can still be compared to the same threshold as before, but it's much more trustworthy than the old computation. Tracking this requires one more intra-transaction counter per modified table within backends, but no additional memory space in the stats collector. There probably isn't any measurable speed difference; if anything it might be a bit faster than before, since I was able to eliminate some per-tuple arithmetic operations in favor of adding sums once per (sub)transaction. Also, simplify the logic around pgstat vacuum and analyze reporting messages by not trying to fold VACUUM ANALYZE into a single pgstat message. The original thought behind this patch was to allow scheduling of analyzes on parent tables by artificially inflating their changes_since_analyze count. I've left that for a separate patch since this change seems to stand on its own merit.
2009-12-30 21:32:14 +01:00
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
/* If this isn't part of VACUUM ANALYZE, let index AMs do cleanup */
Provide a parenthesized-options syntax for VACUUM, analogous to that recently adopted for EXPLAIN. This will allow additional options to be implemented in future without having to make them fully-reserved keywords. The old syntax remains available for existing options, however. Itagaki Takahiro
2009-11-16 22:32:07 +01:00
if (!(vacstmt->options & VACOPT_VACUUM))
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
{
for (ind = 0; ind < nindexes; ind++)
{
IndexBulkDeleteResult *stats;
IndexVacuumInfo ivinfo;
ivinfo.index = Irel[ind];
ivinfo.analyze_only = true;
Improve the IndexVacuumInfo/IndexBulkDeleteResult API to allow somewhat sane behavior in cases where we don't know the heap tuple count accurately; in particular partial vacuum, but this also makes the API a bit more useful for ANALYZE. This patch adds "estimated_count" flags to both structs so that an approximate count can be flagged as such, and adjusts the logic so that approximate counts are not used for updating pg_class.reltuples. This fixes my previous complaint that VACUUM was putting ridiculous values into pg_class.reltuples for indexes. The actual impact of that bug is limited, because the planner only pays attention to reltuples for an index if the index is partial; which probably explains why beta testers hadn't noticed a degradation in plan quality from it. But it needs to be fixed. The whole thing is a bit messy and should be redesigned in future, because reltuples now has the potential to drift quite far away from reality when a long period elapses with no non-partial vacuums. But this is as good as it's going to get for 8.4.
2009-06-07 00:13:52 +02:00
ivinfo.estimated_count = true;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
ivinfo.message_level = elevel;
Improve the IndexVacuumInfo/IndexBulkDeleteResult API to allow somewhat sane behavior in cases where we don't know the heap tuple count accurately; in particular partial vacuum, but this also makes the API a bit more useful for ANALYZE. This patch adds "estimated_count" flags to both structs so that an approximate count can be flagged as such, and adjusts the logic so that approximate counts are not used for updating pg_class.reltuples. This fixes my previous complaint that VACUUM was putting ridiculous values into pg_class.reltuples for indexes. The actual impact of that bug is limited, because the planner only pays attention to reltuples for an index if the index is partial; which probably explains why beta testers hadn't noticed a degradation in plan quality from it. But it needs to be fixed. The whole thing is a bit messy and should be redesigned in future, because reltuples now has the potential to drift quite far away from reality when a long period elapses with no non-partial vacuums. But this is as good as it's going to get for 8.4.
2009-06-07 00:13:52 +02:00
ivinfo.num_heap_tuples = onerel->rd_rel->reltuples;
Implement "fastupdate" support for GIN indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. This patch also removes GIN support for amgettuple-style index scans. The API defined for amgettuple is difficult to support with fastupdate, and the previously committed partial-match feature didn't really work with it either. We might eventually figure a way to put back amgettuple support, but it won't happen for 8.4. catversion bumped because of change in GIN's pg_am entry, and because the format of GIN indexes changed on-disk (there's a metapage now, and possibly a pending list). Teodor Sigaev
2009-03-24 21:17:18 +01:00
ivinfo.strategy = vac_strategy;
stats = index_vacuum_cleanup(&ivinfo, NULL);
if (stats)
pfree(stats);
}
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/* Done with indexes */
Adjust index locking rules as per my proposal of earlier today. You now are supposed to take some kind of lock on an index whenever you are going to access the index contents, rather than relying only on a lock on the parent table.
2004-10-01 01:21:26 +02:00
vac_close_indexes(nindexes, Irel, NoLock);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
/* Log the action if appropriate */
Simplify and rename some GUC variables, per various recent discussions: * stats_start_collector goes away; we always start the collector process, unless prevented by a problem with setting up the stats UDP socket. * stats_reset_on_server_start goes away; it seems useless in view of the availability of pg_stat_reset(). * stats_block_level and stats_row_level are merged into a single variable "track_counts", which controls all reports sent to the collector process. * stats_command_string is renamed to track_activities. * log_autovacuum is renamed to log_autovacuum_min_duration to better reflect its meaning. The log_autovacuum change is not a compatibility issue since it didn't exist before 8.3 anyway. The other changes need to be release-noted.
2007-09-24 05:12:23 +02:00
if (IsAutoVacuumWorkerProcess() && Log_autovacuum_min_duration >= 0)
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
{
Simplify and rename some GUC variables, per various recent discussions: * stats_start_collector goes away; we always start the collector process, unless prevented by a problem with setting up the stats UDP socket. * stats_reset_on_server_start goes away; it seems useless in view of the availability of pg_stat_reset(). * stats_block_level and stats_row_level are merged into a single variable "track_counts", which controls all reports sent to the collector process. * stats_command_string is renamed to track_activities. * log_autovacuum is renamed to log_autovacuum_min_duration to better reflect its meaning. The log_autovacuum change is not a compatibility issue since it didn't exist before 8.3 anyway. The other changes need to be release-noted.
2007-09-24 05:12:23 +02:00
if (Log_autovacuum_min_duration == 0 ||
Implement rate-limiting logic on how often backends will attempt to send messages to the stats collector. This avoids the problem that enabling stats_row_level for autovacuum has a significant overhead for short read-only transactions, as noted by Arjen van der Meijden. We can avoid an extra gettimeofday call by piggybacking on the one done for WAL-logging xact commit or abort (although that doesn't help read-only transactions, since they don't WAL-log anything). In my proposal for this, I noted that we could change the WAL log entries for commit/abort to record full TimestampTz precision, instead of only time_t as at present. That's not done in this patch, but will be committed separately.
2007-04-30 05:23:49 +02:00
TimestampDifferenceExceeds(starttime, GetCurrentTimestamp(),
Simplify and rename some GUC variables, per various recent discussions: * stats_start_collector goes away; we always start the collector process, unless prevented by a problem with setting up the stats UDP socket. * stats_reset_on_server_start goes away; it seems useless in view of the availability of pg_stat_reset(). * stats_block_level and stats_row_level are merged into a single variable "track_counts", which controls all reports sent to the collector process. * stats_command_string is renamed to track_activities. * log_autovacuum is renamed to log_autovacuum_min_duration to better reflect its meaning. The log_autovacuum change is not a compatibility issue since it didn't exist before 8.3 anyway. The other changes need to be release-noted.
2007-09-24 05:12:23 +02:00
Log_autovacuum_min_duration))
Enable configurable log of autovacuum actions. Initial patch from Simon Riggs, additional code and docs by me. Per discussion.
2007-04-18 18:44:18 +02:00
ereport(LOG,
(errmsg("automatic analyze of table \"%s.%s.%s\" system usage: %s",
get_database_name(MyDatabaseId),
get_namespace_name(RelationGetNamespace(onerel)),
RelationGetRelationName(onerel),
pg_rusage_show(&ru0))));
}
Rearrange vacuum-related bits in PGPROC as a bitmask, to better support having several of them. Add two more flags: whether the process is executing an ANALYZE, and whether a vacuum is for Xid wraparound (which is obviously only set by autovacuum). Sneakily move the worker's recently-acquired PostAuthDelay to a more useful place.
2007-10-24 22:55:36 +02:00
Prevent indirect security attacks via changing session-local state within an allegedly immutable index function. It was previously recognized that we had to prevent such a function from executing SET/RESET ROLE/SESSION AUTHORIZATION, or it could trivially obtain the privileges of the session user. However, since there is in general no privilege checking for changes of session-local state, it is also possible for such a function to change settings in a way that might subvert later operations in the same session. Examples include changing search_path to cause an unexpected function to be called, or replacing an existing prepared statement with another one that will execute a function of the attacker's choosing. The present patch secures VACUUM, ANALYZE, and CREATE INDEX/REINDEX against these threats, which are the same places previously deemed to need protection against the SET ROLE issue. GUC changes are still allowed, since there are many useful cases for that, but we prevent security problems by forcing a rollback of any GUC change after completing the operation. Other cases are handled by throwing an error if any change is attempted; these include temp table creation, closing a cursor, and creating or deleting a prepared statement. (In 7.4, the infrastructure to roll back GUC changes doesn't exist, so we settle for rejecting changes of "search_path" in these contexts.) Original report and patch by Gurjeet Singh, additional analysis by Tom Lane. Security: CVE-2009-4136
2009-12-09 22:57:51 +01:00
/* Roll back any GUC changes executed by index functions */
AtEOXact_GUC(false, save_nestlevel);
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/* Restore current context and release memory */
MemoryContextSwitchTo(caller_context);
MemoryContextDelete(anl_context);
anl_context = NULL;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/*
* Compute statistics about indexes of a relation
*/
static void
compute_index_stats(Relation onerel, double totalrows,
AnlIndexData *indexdata, int nindexes,
HeapTuple *rows, int numrows,
MemoryContext col_context)
{
MemoryContext ind_context,
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
old_context;
Convert index-related tuple handling routines from char 'n'/' ' to bool convention for isnull flags. Also, remove the useless InsertIndexResult return struct from index AM aminsert calls --- there is no reason for the caller to know where in the index the tuple was inserted, and we were wasting a palloc cycle per insert to deliver this uninteresting value (plus nontrivial complexity in some AMs). I forced initdb because of the change in the signature of the aminsert routines, even though nothing really looks at those pg_proc entries...
2005-03-21 02:24:04 +01:00
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
int ind,
i;
ind_context = AllocSetContextCreate(anl_context,
"Analyze Index",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
old_context = MemoryContextSwitchTo(ind_context);
for (ind = 0; ind < nindexes; ind++)
{
AnlIndexData *thisdata = &indexdata[ind];
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
IndexInfo *indexInfo = thisdata->indexInfo;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
int attr_cnt = thisdata->attr_cnt;
TupleTableSlot *slot;
EState *estate;
ExprContext *econtext;
List *predicate;
Datum *exprvals;
bool *exprnulls;
int numindexrows,
tcnt,
rowno;
double totalindexrows;
/* Ignore index if no columns to analyze and not partial */
if (attr_cnt == 0 && indexInfo->ii_Predicate == NIL)
continue;
/*
* Need an EState for evaluation of index expressions and
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* partial-index predicates. Create it in the per-index context to be
* sure it gets cleaned up at the bottom of the loop.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
estate = CreateExecutorState();
econtext = GetPerTupleExprContext(estate);
/* Need a slot to hold the current heap tuple, too */
Revise TupleTableSlot code to avoid unnecessary construction and disassembly of tuples when passing data up through multiple plan nodes. A slot can now hold either a normal "physical" HeapTuple, or a "virtual" tuple consisting of Datum/isnull arrays. Upper plan levels can usually just copy the Datum arrays, avoiding heap_formtuple() and possible subsequent nocachegetattr() calls to extract the data again. This work extends Atsushi Ogawa's earlier patch, which provided the key idea of adding Datum arrays to TupleTableSlots. (I believe however that something like this was foreseen way back in Berkeley days --- see the old comment on ExecProject.) A test case involving many levels of join of fairly wide tables (about 80 columns altogether) showed about 3x overall speedup, though simple queries will probably not be helped very much. I have also duplicated some code in heaptuple.c in order to provide versions of heap_formtuple and friends that use "bool" arrays to indicate null attributes, instead of the old convention of "char" arrays containing either 'n' or ' '. This provides a better match to the convention used by ExecEvalExpr. While I have not made a concerted effort to get rid of uses of the old routines, I think they should be deprecated and eventually removed.
2005-03-16 22:38:10 +01:00
slot = MakeSingleTupleTableSlot(RelationGetDescr(onerel));
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/* Arrange for econtext's scan tuple to be the tuple under test */
econtext->ecxt_scantuple = slot;
/* Set up execution state for predicate. */
predicate = (List *)
ExecPrepareExpr((Expr *) indexInfo->ii_Predicate,
estate);
/* Compute and save index expression values */
Tweak palloc/repalloc to allow zero bytes to be requested, as per recent proposal. Eliminate several dozen now-unnecessary hacks to avoid palloc(0). (It's likely there are more that I didn't find.)
2004-06-05 21:48:09 +02:00
exprvals = (Datum *) palloc(numrows * attr_cnt * sizeof(Datum));
exprnulls = (bool *) palloc(numrows * attr_cnt * sizeof(bool));
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
numindexrows = 0;
tcnt = 0;
for (rowno = 0; rowno < numrows; rowno++)
{
HeapTuple heapTuple = rows[rowno];
Repair memory leakage while ANALYZE-ing complex index expressions. The general design of memory management in Postgres is that intermediate results computed by an expression are not freed until the end of the tuple cycle. For expression indexes, ANALYZE has to re-evaluate each expression for each of its sample rows, and it wasn't bothering to free intermediate results until the end of processing of that index. This could lead to very substantial leakage if the intermediate results were large, as in a recent example from Jakub Ouhrabka. Fix by doing ResetExprContext for each sample row. This necessitates adding a datumCopy step to ensure that the final expression value isn't recycled too. Some quick testing suggests that this change adds at worst about 10% to the time needed to analyze a table with an expression index; which is annoying, but seems a tolerable price to pay to avoid unexpected out-of-memory problems. Back-patch to all supported branches.
2010-11-09 17:28:18 +01:00
/*
* Reset the per-tuple context each time, to reclaim any cruft
* left behind by evaluating the predicate or index expressions.
*/
ResetExprContext(econtext);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/* Set up for predicate or expression evaluation */
ExecStoreTuple(heapTuple, slot, InvalidBuffer, false);
/* If index is partial, check predicate */
if (predicate != NIL)
{
if (!ExecQual(predicate, econtext, false))
continue;
}
numindexrows++;
if (attr_cnt > 0)
{
/*
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
* Evaluate the index row to compute expression values. We
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* could do this by hand, but FormIndexDatum is convenient.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
FormIndexDatum(indexInfo,
Revise TupleTableSlot code to avoid unnecessary construction and disassembly of tuples when passing data up through multiple plan nodes. A slot can now hold either a normal "physical" HeapTuple, or a "virtual" tuple consisting of Datum/isnull arrays. Upper plan levels can usually just copy the Datum arrays, avoiding heap_formtuple() and possible subsequent nocachegetattr() calls to extract the data again. This work extends Atsushi Ogawa's earlier patch, which provided the key idea of adding Datum arrays to TupleTableSlots. (I believe however that something like this was foreseen way back in Berkeley days --- see the old comment on ExecProject.) A test case involving many levels of join of fairly wide tables (about 80 columns altogether) showed about 3x overall speedup, though simple queries will probably not be helped very much. I have also duplicated some code in heaptuple.c in order to provide versions of heap_formtuple and friends that use "bool" arrays to indicate null attributes, instead of the old convention of "char" arrays containing either 'n' or ' '. This provides a better match to the convention used by ExecEvalExpr. While I have not made a concerted effort to get rid of uses of the old routines, I think they should be deprecated and eventually removed.
2005-03-16 22:38:10 +01:00
slot,
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
estate,
Convert index-related tuple handling routines from char 'n'/' ' to bool convention for isnull flags. Also, remove the useless InsertIndexResult return struct from index AM aminsert calls --- there is no reason for the caller to know where in the index the tuple was inserted, and we were wasting a palloc cycle per insert to deliver this uninteresting value (plus nontrivial complexity in some AMs). I forced initdb because of the change in the signature of the aminsert routines, even though nothing really looks at those pg_proc entries...
2005-03-21 02:24:04 +01:00
values,
isnull);
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/*
Repair memory leakage while ANALYZE-ing complex index expressions. The general design of memory management in Postgres is that intermediate results computed by an expression are not freed until the end of the tuple cycle. For expression indexes, ANALYZE has to re-evaluate each expression for each of its sample rows, and it wasn't bothering to free intermediate results until the end of processing of that index. This could lead to very substantial leakage if the intermediate results were large, as in a recent example from Jakub Ouhrabka. Fix by doing ResetExprContext for each sample row. This necessitates adding a datumCopy step to ensure that the final expression value isn't recycled too. Some quick testing suggests that this change adds at worst about 10% to the time needed to analyze a table with an expression index; which is annoying, but seems a tolerable price to pay to avoid unexpected out-of-memory problems. Back-patch to all supported branches.
2010-11-09 17:28:18 +01:00
* Save just the columns we care about. We copy the values
* into ind_context from the estate's per-tuple context.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
for (i = 0; i < attr_cnt; i++)
{
VacAttrStats *stats = thisdata->vacattrstats[i];
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
int attnum = stats->attr->attnum;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
Repair memory leakage while ANALYZE-ing complex index expressions. The general design of memory management in Postgres is that intermediate results computed by an expression are not freed until the end of the tuple cycle. For expression indexes, ANALYZE has to re-evaluate each expression for each of its sample rows, and it wasn't bothering to free intermediate results until the end of processing of that index. This could lead to very substantial leakage if the intermediate results were large, as in a recent example from Jakub Ouhrabka. Fix by doing ResetExprContext for each sample row. This necessitates adding a datumCopy step to ensure that the final expression value isn't recycled too. Some quick testing suggests that this change adds at worst about 10% to the time needed to analyze a table with an expression index; which is annoying, but seems a tolerable price to pay to avoid unexpected out-of-memory problems. Back-patch to all supported branches.
2010-11-09 17:28:18 +01:00
if (isnull[attnum - 1])
{
exprvals[tcnt] = (Datum) 0;
exprnulls[tcnt] = true;
}
else
{
exprvals[tcnt] = datumCopy(values[attnum - 1],
stats->attrtype->typbyval,
stats->attrtype->typlen);
exprnulls[tcnt] = false;
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
tcnt++;
}
}
}
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Having counted the number of rows that pass the predicate in the
* sample, we can estimate the total number of rows in the index.
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
*/
thisdata->tupleFract = (double) numindexrows / (double) numrows;
totalindexrows = ceil(thisdata->tupleFract * totalrows);
/*
* Now we can compute the statistics for the expression columns.
*/
if (numindexrows > 0)
{
MemoryContextSwitchTo(col_context);
for (i = 0; i < attr_cnt; i++)
{
VacAttrStats *stats = thisdata->vacattrstats[i];
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
AttributeOpts *aopt =
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
get_attribute_options(stats->attr->attrelid,
stats->attr->attnum);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
stats->exprvals = exprvals + i;
stats->exprnulls = exprnulls + i;
stats->rowstride = attr_cnt;
(*stats->compute_stats) (stats,
ind_fetch_func,
numindexrows,
totalindexrows);
Replace ALTER TABLE ... SET STATISTICS DISTINCT with a more general mechanism. Attributes can now have options, just as relations and tablespaces do, and the reloptions code is used to parse, validate, and store them. For simplicity and because these options are not performance critical, we store them in a separate cache rather than the main relcache. Thanks to Alex Hunsaker for the review.
2010-01-22 17:40:19 +01:00
/*
* If the n_distinct option is specified, it overrides the
* above computation. For indices, we always use just
* n_distinct, not n_distinct_inherited.
*/
if (aopt != NULL && aopt->n_distinct != 0.0)
stats->stadistinct = aopt->n_distinct;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
MemoryContextResetAndDeleteChildren(col_context);
}
}
/* And clean up */
MemoryContextSwitchTo(ind_context);
Revise TupleTableSlot code to avoid unnecessary construction and disassembly of tuples when passing data up through multiple plan nodes. A slot can now hold either a normal "physical" HeapTuple, or a "virtual" tuple consisting of Datum/isnull arrays. Upper plan levels can usually just copy the Datum arrays, avoiding heap_formtuple() and possible subsequent nocachegetattr() calls to extract the data again. This work extends Atsushi Ogawa's earlier patch, which provided the key idea of adding Datum arrays to TupleTableSlots. (I believe however that something like this was foreseen way back in Berkeley days --- see the old comment on ExecProject.) A test case involving many levels of join of fairly wide tables (about 80 columns altogether) showed about 3x overall speedup, though simple queries will probably not be helped very much. I have also duplicated some code in heaptuple.c in order to provide versions of heap_formtuple and friends that use "bool" arrays to indicate null attributes, instead of the old convention of "char" arrays containing either 'n' or ' '. This provides a better match to the convention used by ExecEvalExpr. While I have not made a concerted effort to get rid of uses of the old routines, I think they should be deprecated and eventually removed.
2005-03-16 22:38:10 +01:00
ExecDropSingleTupleTableSlot(slot);
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
FreeExecutorState(estate);
MemoryContextResetAndDeleteChildren(ind_context);
}
MemoryContextSwitchTo(old_context);
MemoryContextDelete(ind_context);
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* examine_attribute -- pre-analysis of a single column
*
* Determine whether the column is analyzable; if so, create and initialize
* a VacAttrStats struct for it. If not, return NULL.
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
*
* If index_expr isn't NULL, then we're trying to analyze an expression index,
* and index_expr is the expression tree representing the column's data.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
static VacAttrStats *
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
examine_attribute(Relation onerel, int attnum, Node *index_expr)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Form_pg_attribute attr = onerel->rd_att->attrs[attnum - 1];
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
HeapTuple typtuple;
VacAttrStats *stats;
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
int i;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
bool ok;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/* Never analyze dropped columns */
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
2002-08-02 20:15:10 +02:00
if (attr->attisdropped)
return NULL;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Don't analyze column if user has specified not to */
Instead of having a configure-time DEFAULT_ATTSTATTARGET, store -1 in attstattarget to indicate 'use the default'. The default is now a GUC variable default_statistics_target, and so may be changed on the fly. Along the way we gain the ability to have pg_dump dump the per-column statistics target when it's not the default. Patch by Neil Conway, with some kibitzing from Tom Lane.
2002-07-31 19:19:54 +02:00
if (attr->attstattarget == 0)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
return NULL;
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Create the VacAttrStats struct. Note that we only have a copy of the
* fixed fields of the pg_attribute tuple.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Add new palloc0 call as merge of palloc and MemSet(0).
2002-11-13 01:39:48 +01:00
stats = (VacAttrStats *) palloc0(sizeof(VacAttrStats));
Support column-level privileges, as required by SQL standard. Stephen Frost, with help from KaiGai Kohei and others
2009-01-22 21:16:10 +01:00
stats->attr = (Form_pg_attribute) palloc(ATTRIBUTE_FIXED_PART_SIZE);
memcpy(stats->attr, attr, ATTRIBUTE_FIXED_PART_SIZE);
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
/*
* When analyzing an expression index, believe the expression tree's type
* not the column datatype --- the latter might be the opckeytype storage
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* type of the opclass, which is not interesting for our purposes. (Note:
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
* if we did anything with non-expression index columns, we'd need to
* figure out where to get the correct type info from, but for now that's
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* not a problem.) It's not clear whether anyone will care about the
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
* typmod, but we store that too just in case.
*/
if (index_expr)
{
stats->attrtypid = exprType(index_expr);
stats->attrtypmod = exprTypmod(index_expr);
}
else
{
stats->attrtypid = attr->atttypid;
stats->attrtypmod = attr->atttypmod;
}
Avoid possibly accessing off the end of memory in examine_attribute(). Since the last couple of columns of pg_type are often NULL, sizeof(FormData_pg_type) can be an overestimate of the actual size of the tuple data part. Therefore memcpy'ing that much out of the catalog cache, as analyze.c was doing, poses a small risk of copying past the end of memory and incurring SIGSEGV. No such crash has been identified in the field, but we've certainly seen the equivalent happen in other code paths, so patch this one all the way back. Per valgrind testing by Noah Misch, though this is not his proposed patch. I chose to use SearchSysCacheCopy1 rather than inventing special-purpose infrastructure for copying only the minimal part of a pg_type tuple.
2011-09-06 20:35:19 +02:00
typtuple = SearchSysCacheCopy1(TYPEOID,
ObjectIdGetDatum(stats->attrtypid));
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (!HeapTupleIsValid(typtuple))
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
elog(ERROR, "cache lookup failed for type %u", stats->attrtypid);
Avoid possibly accessing off the end of memory in examine_attribute(). Since the last couple of columns of pg_type are often NULL, sizeof(FormData_pg_type) can be an overestimate of the actual size of the tuple data part. Therefore memcpy'ing that much out of the catalog cache, as analyze.c was doing, poses a small risk of copying past the end of memory and incurring SIGSEGV. No such crash has been identified in the field, but we've certainly seen the equivalent happen in other code paths, so patch this one all the way back. Per valgrind testing by Noah Misch, though this is not his proposed patch. I chose to use SearchSysCacheCopy1 rather than inventing special-purpose infrastructure for copying only the minimal part of a pg_type tuple.
2011-09-06 20:35:19 +02:00
stats->attrtype = (Form_pg_type) GETSTRUCT(typtuple);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
stats->anl_context = anl_context;
stats->tupattnum = attnum;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* The fields describing the stats->stavalues[n] element types default to
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
* the type of the data being analyzed, but the type-specific typanalyze
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* function can change them if it wants to store something else.
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
*/
for (i = 0; i < STATISTIC_NUM_SLOTS; i++)
{
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
stats->statypid[i] = stats->attrtypid;
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
stats->statyplen[i] = stats->attrtype->typlen;
stats->statypbyval[i] = stats->attrtype->typbyval;
stats->statypalign[i] = stats->attrtype->typalign;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Call the type-specific typanalyze function. If none is specified, use
* std_typanalyze().
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
if (OidIsValid(stats->attrtype->typanalyze))
ok = DatumGetBool(OidFunctionCall1(stats->attrtype->typanalyze,
PointerGetDatum(stats)));
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
else
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
ok = std_typanalyze(stats);
if (!ok || stats->compute_stats == NULL || stats->minrows <= 0)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Avoid possibly accessing off the end of memory in examine_attribute(). Since the last couple of columns of pg_type are often NULL, sizeof(FormData_pg_type) can be an overestimate of the actual size of the tuple data part. Therefore memcpy'ing that much out of the catalog cache, as analyze.c was doing, poses a small risk of copying past the end of memory and incurring SIGSEGV. No such crash has been identified in the field, but we've certainly seen the equivalent happen in other code paths, so patch this one all the way back. Per valgrind testing by Noah Misch, though this is not his proposed patch. I chose to use SearchSysCacheCopy1 rather than inventing special-purpose infrastructure for copying only the minimal part of a pg_type tuple.
2011-09-06 20:35:19 +02:00
heap_freetuple(typtuple);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
pfree(stats->attr);
pfree(stats);
return NULL;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
return stats;
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/*
* BlockSampler_Init -- prepare for random sampling of blocknumbers
*
* BlockSampler is used for stage one of our new two-stage tuple
* sampling mechanism as discussed on pgsql-hackers 2004-04-02 (subject
* "Large DB"). It selects a random sample of samplesize blocks out of
* the nblocks blocks in the table. If the table has less than
* samplesize blocks, all blocks are selected.
*
* Since we know the total number of blocks in advance, we can use the
* straightforward Algorithm S from Knuth 3.4.2, rather than Vitter's
* algorithm.
*/
static void
BlockSampler_Init(BlockSampler bs, BlockNumber nblocks, int samplesize)
{
bs->N = nblocks; /* measured table size */
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* If we decide to reduce samplesize for tables that have less or not much
* more than samplesize blocks, here is the place to do it.
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
*/
bs->n = samplesize;
bs->t = 0; /* blocks scanned so far */
bs->m = 0; /* blocks selected so far */
}
static bool
BlockSampler_HasMore(BlockSampler bs)
{
return (bs->t < bs->N) && (bs->m < bs->n);
}
static BlockNumber
BlockSampler_Next(BlockSampler bs)
{
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
BlockNumber K = bs->N - bs->t; /* remaining blocks */
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
int k = bs->n - bs->m; /* blocks still to sample */
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
double p; /* probability to skip block */
double V; /* random */
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
Assert(BlockSampler_HasMore(bs)); /* hence K > 0 and k > 0 */
if ((BlockNumber) k >= K)
{
/* need all the rest */
bs->m++;
return bs->t++;
}
/*----------
* It is not obvious that this code matches Knuth's Algorithm S.
* Knuth says to skip the current block with probability 1 - k/K.
* If we are to skip, we should advance t (hence decrease K), and
* repeat the same probabilistic test for the next block. The naive
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
* implementation thus requires an anl_random_fract() call for each block
* number. But we can reduce this to one anl_random_fract() call per
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
* selected block, by noting that each time the while-test succeeds,
* we can reinterpret V as a uniform random number in the range 0 to p.
* Therefore, instead of choosing a new V, we just adjust p to be
* the appropriate fraction of its former value, and our next loop
* makes the appropriate probabilistic test.
*
* We have initially K > k > 0. If the loop reduces K to equal k,
* the next while-test must fail since p will become exactly zero
* (we assume there will not be roundoff error in the division).
* (Note: Knuth suggests a "<=" loop condition, but we use "<" just
* to be doubly sure about roundoff error.) Therefore K cannot become
* less than k, which means that we cannot fail to select enough blocks.
*----------
*/
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
V = anl_random_fract();
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
p = 1.0 - (double) k / (double) K;
while (V < p)
{
/* skip */
bs->t++;
K--; /* keep K == N - t */
/* adjust p to be new cutoff point in reduced range */
p *= 1.0 - (double) k / (double) K;
}
/* select */
bs->m++;
return bs->t++;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* acquire_sample_rows -- acquire a random sample of rows from the table
*
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
* Selected rows are returned in the caller-allocated array rows[], which
* must have at least targrows entries.
* The actual number of rows selected is returned as the function result.
* We also estimate the total numbers of live and dead rows in the table,
* and return them into *totalrows and *totaldeadrows, respectively.
*
* The returned list of tuples is in order by physical position in the table.
* (We will rely on this later to derive correlation estimates.)
*
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
* As of May 2004 we use a new two-stage method: Stage one selects up
* to targrows random blocks (or all blocks, if there aren't so many).
* Stage two scans these blocks and uses the Vitter algorithm to create
* a random sample of targrows rows (or less, if there are less in the
* sample of blocks). The two stages are executed simultaneously: each
* block is processed as soon as stage one returns its number and while
* the rows are read stage two controls which ones are to be inserted
* into the sample.
*
* Although every row has an equal chance of ending up in the final
* sample, this sampling method is not perfect: not every possible
* sample has an equal chance of being selected. For large relations
* the number of different blocks represented by the sample tends to be
* too small. We can live with that for now. Improvements are welcome.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*
Integrate autovacuum functionality into the backend. There's still a few loose ends to be dealt with, but it seems to work. Alvaro Herrera, based on the contrib code by Matthew O'Connor.
2005-07-14 07:13:45 +02:00
* An important property of this sampling method is that because we do
* look at a statistically unbiased set of blocks, we should get
* unbiased estimates of the average numbers of live and dead rows per
* block. The previous sampling method put too much credence in the row
* density near the start of the table.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
static int
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
acquire_sample_rows(Relation onerel, int elevel,
HeapTuple *rows, int targrows,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
double *totalrows, double *totaldeadrows)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
int numrows = 0; /* # rows now in reservoir */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
double samplerows = 0; /* total # rows collected */
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
double liverows = 0; /* # live rows seen */
Integrate autovacuum functionality into the backend. There's still a few loose ends to be dealt with, but it seems to work. Alvaro Herrera, based on the contrib code by Matthew O'Connor.
2005-07-14 07:13:45 +02:00
double deadrows = 0; /* # dead rows seen */
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
double rowstoskip = -1; /* -1 means not set yet */
BlockNumber totalblocks;
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
TransactionId OldestXmin;
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
BlockSamplerData bs;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
double rstate;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
Assert(targrows > 0);
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
totalblocks = RelationGetNumberOfBlocks(onerel);
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
/* Need a cutoff xmin for HeapTupleSatisfiesVacuum */
OldestXmin = GetOldestXmin(onerel->rd_rel->relisshared, true);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/* Prepare for sampling block numbers */
BlockSampler_Init(&bs, totalblocks, targrows);
/* Prepare for sampling rows */
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
rstate = anl_init_selection_state(targrows);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/* Outer loop over blocks to sample */
while (BlockSampler_HasMore(&bs))
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
BlockNumber targblock = BlockSampler_Next(&bs);
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Buffer targbuffer;
Page targpage;
OffsetNumber targoffset,
maxoffset;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Centralize implementation of delay code by creating a pg_usleep() subroutine in src/port/pgsleep.c. Remove platform dependencies from miscadmin.h and put them in port.h where they belong. Extend recent vacuum cost-based-delay patch to apply to VACUUM FULL, ANALYZE, and non-btree index vacuuming. By the way, where is the documentation for the cost-based-delay patch?
2004-02-10 04:42:45 +01:00
vacuum_delay_point();
Add CHECK_FOR_INTERRUPTS() in various strategic spots, per comments from Hiroshi.
2002-01-06 01:37:44 +01:00
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* We must maintain a pin on the target page's buffer to ensure that
* the maxoffset value stays good (else concurrent VACUUM might delete
* tuples out from under us). Hence, pin the page until we are done
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
* looking at it. We also choose to hold sharelock on the buffer
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* throughout --- we could release and re-acquire sharelock for each
* tuple, but since we aren't doing much work per tuple, the extra
* lock traffic is probably better avoided.
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
*/
Unite ReadBufferWithFork, ReadBufferWithStrategy, and ZeroOrReadBuffer functions into one ReadBufferExtended function, that takes the strategy and mode as argument. There's three modes, RBM_NORMAL which is the default used by plain ReadBuffer(), RBM_ZERO, which replaces ZeroOrReadBuffer, and a new mode RBM_ZERO_ON_ERROR, which allows callers to read corrupt pages without throwing an error. The FSM needs the new mode to recover from corrupt pages, which could happend if we crash after extending an FSM file, and the new page is "torn". Add fork number to some error messages in bufmgr.c, that still lacked it.
2008-10-31 16:05:00 +01:00
targbuffer = ReadBufferExtended(onerel, MAIN_FORKNUM, targblock,
RBM_NORMAL, vac_strategy);
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
LockBuffer(targbuffer, BUFFER_LOCK_SHARE);
targpage = BufferGetPage(targbuffer);
maxoffset = PageGetMaxOffsetNumber(targpage);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/* Inner loop over all tuples on the selected page */
for (targoffset = FirstOffsetNumber; targoffset <= maxoffset; targoffset++)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
ItemId itemid;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
HeapTupleData targtuple;
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
bool sample_it = false;
itemid = PageGetItemId(targpage, targoffset);
/*
* We ignore unused and redirect line pointers. DEAD line
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* pointers should be counted as dead, because we need vacuum to
* run to get rid of them. Note that this rule agrees with the
* way that heap_page_prune() counts things.
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
*/
if (!ItemIdIsNormal(itemid))
{
if (ItemIdIsDead(itemid))
deadrows += 1;
continue;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
ItemPointerSet(&targtuple.t_self, targblock, targoffset);
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
Adjust HeapTupleSatisfies* routines to take a HeapTuple. Previously, these functions took a HeapTupleHeader, but upcoming patches for logical replication will introduce new a new snapshot type under which the tuple's TID will be used to lookup (CMIN, CMAX) for visibility determination purposes. This makes that information available. Code churn is minimal since HeapTupleSatisfiesVisibility took the HeapTuple anyway, and deferenced it before calling the satisfies function. Independently of logical replication, this allows t_tableOid and t_self to be cross-checked via assertions in tqual.c. This seems like a useful way to make sure that all callers are setting these values properly, which has been previously put forward as desirable. Andres Freund, reviewed by Álvaro Herrera
2013-07-22 19:26:33 +02:00
targtuple.t_tableOid = RelationGetRelid(onerel);
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
targtuple.t_data = (HeapTupleHeader) PageGetItem(targpage, itemid);
targtuple.t_len = ItemIdGetLength(itemid);
Adjust HeapTupleSatisfies* routines to take a HeapTuple. Previously, these functions took a HeapTupleHeader, but upcoming patches for logical replication will introduce new a new snapshot type under which the tuple's TID will be used to lookup (CMIN, CMAX) for visibility determination purposes. This makes that information available. Code churn is minimal since HeapTupleSatisfiesVisibility took the HeapTuple anyway, and deferenced it before calling the satisfies function. Independently of logical replication, this allows t_tableOid and t_self to be cross-checked via assertions in tqual.c. This seems like a useful way to make sure that all callers are setting these values properly, which has been previously put forward as desirable. Andres Freund, reviewed by Álvaro Herrera
2013-07-22 19:26:33 +02:00
switch (HeapTupleSatisfiesVacuum(&targtuple,
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
OldestXmin,
targbuffer))
{
case HEAPTUPLE_LIVE:
sample_it = true;
liverows += 1;
break;
case HEAPTUPLE_DEAD:
case HEAPTUPLE_RECENTLY_DEAD:
/* Count dead and recently-dead rows */
deadrows += 1;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
/*
* Insert-in-progress rows are not counted. We assume
* that when the inserting transaction commits or aborts,
* it will send a stats message to increment the proper
* count. This works right only if that transaction ends
* after we finish analyzing the table; if things happen
* in the other order, its stats update will be
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* overwritten by ours. However, the error will be large
* only if the other transaction runs long enough to
* insert many tuples, so assuming it will finish after us
* is the safer option.
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
*
* A special case is that the inserting transaction might
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* be our own. In this case we should count and sample
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
* the row, to accommodate users who load a table and
* analyze it in one transaction. (pgstat_report_analyze
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* has to adjust the numbers we send to the stats
* collector to make this come out right.)
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
*/
if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(targtuple.t_data)))
{
sample_it = true;
liverows += 1;
}
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
/*
* We count delete-in-progress rows as still live, using
* the same reasoning given above; but we don't bother to
* include them in the sample.
*
* If the delete was done by our own transaction, however,
* we must count the row as dead to make
* pgstat_report_analyze's stats adjustments come out
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* right. (Note: this works out properly when the row was
* both inserted and deleted in our xact.)
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
*/
Improve concurrency of foreign key locking This patch introduces two additional lock modes for tuples: "SELECT FOR KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each other, in contrast with already existing "SELECT FOR SHARE" and "SELECT FOR UPDATE". UPDATE commands that do not modify the values stored in the columns that are part of the key of the tuple now grab a SELECT FOR NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently with tuple locks of the FOR KEY SHARE variety. Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this means the concurrency improvement applies to them, which is the whole point of this patch. The added tuple lock semantics require some rejiggering of the multixact module, so that the locking level that each transaction is holding can be stored alongside its Xid. Also, multixacts now need to persist across server restarts and crashes, because they can now represent not only tuple locks, but also tuple updates. This means we need more careful tracking of lifetime of pg_multixact SLRU files; since they now persist longer, we require more infrastructure to figure out when they can be removed. pg_upgrade also needs to be careful to copy pg_multixact files over from the old server to the new, or at least part of multixact.c state, depending on the versions of the old and new servers. Tuple time qualification rules (HeapTupleSatisfies routines) need to be careful not to consider tuples with the "is multi" infomask bit set as being only locked; they might need to look up MultiXact values (i.e. possibly do pg_multixact I/O) to find out the Xid that updated a tuple, whereas they previously were assured to only use information readily available from the tuple header. This is considered acceptable, because the extra I/O would involve cases that would previously cause some commands to block waiting for concurrent transactions to finish. Another important change is the fact that locking tuples that have previously been updated causes the future versions to be marked as locked, too; this is essential for correctness of foreign key checks. This causes additional WAL-logging, also (there was previously a single WAL record for a locked tuple; now there are as many as updated copies of the tuple there exist.) With all this in place, contention related to tuples being checked by foreign key rules should be much reduced. As a bonus, the old behavior that a subtransaction grabbing a stronger tuple lock than the parent (sub)transaction held on a given tuple and later aborting caused the weaker lock to be lost, has been fixed. Many new spec files were added for isolation tester framework, to ensure overall behavior is sane. There's probably room for several more tests. There were several reviewers of this patch; in particular, Noah Misch and Andres Freund spent considerable time in it. Original idea for the patch came from Simon Riggs, after a problem report by Joel Jacobson. Most code is from me, with contributions from Marti Raudsepp, Alexander Shulgin, Noah Misch and Andres Freund. This patch was discussed in several pgsql-hackers threads; the most important start at the following message-ids: AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com 1290721684-sup-3951@alvh.no-ip.org 1294953201-sup-2099@alvh.no-ip.org 1320343602-sup-2290@alvh.no-ip.org 1339690386-sup-8927@alvh.no-ip.org 4FE5FF020200002500048A3D@gw.wicourts.gov 4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-23 16:04:59 +01:00
if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(targtuple.t_data)))
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
deadrows += 1;
else
liverows += 1;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
break;
}
if (sample_it)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
/*
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
* The first targrows sample rows are simply copied into the
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
* reservoir. Then we start replacing tuples in the sample
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* until we reach the end of the relation. This algorithm is
* from Jeff Vitter's paper (see full citation below). It
* works by repeatedly computing the number of tuples to skip
* before selecting a tuple, which replaces a randomly chosen
* element of the reservoir (current set of tuples). At all
* times the reservoir is a true random sample of the tuples
* we've passed over so far, so when we fall off the end of
* the relation we're done.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
if (numrows < targrows)
rows[numrows++] = heap_copytuple(&targtuple);
else
{
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* t in Vitter's paper is the number of records already
* processed. If we need to compute a new S value, we
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* must use the not-yet-incremented value of samplerows as
* t.
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
*/
if (rowstoskip < 0)
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
rowstoskip = anl_get_next_S(samplerows, targrows,
&rstate);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
if (rowstoskip <= 0)
{
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Found a suitable tuple, so save it, replacing one
* old tuple at random
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
*/
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
int k = (int) (targrows * anl_random_fract());
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
Assert(k >= 0 && k < targrows);
heap_freetuple(rows[k]);
rows[k] = heap_copytuple(&targtuple);
}
rowstoskip -= 1;
}
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
samplerows += 1;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
Teach ANALYZE to distinguish dead and in-doubt tuples, which it formerly classed all as "dead"; also get it to count DEAD item pointers as dead rows, instead of ignoring them as before. Also improve matters so that tuples previously inserted or deleted by our own transaction are handled nicely: the stats collector's live-tuple and dead-tuple counts will end up correct after our transaction ends, regardless of whether we end in commit or abort. While there's more work that could be done to improve the counting of in-doubt tuples in both VACUUM and ANALYZE, this commit is enough to alleviate some known bad behaviors in 8.3; and the other stuff that's been discussed seems like research projects anyway. Pavan Deolasee and Tom Lane
2008-04-03 18:27:25 +02:00
/* Now release the lock and pin on the page */
UnlockReleaseBuffer(targbuffer);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* If we didn't find as many tuples as we wanted then we're done. No sort
* is needed, since they're already in order.
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
*
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
2005-11-22 19:17:34 +01:00
* Otherwise we need to sort the collected tuples by position
* (itempointer). It's not worth worrying about corner cases where the
* tuples are already sorted.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
if (numrows == targrows)
qsort((void *) rows, numrows, sizeof(HeapTuple), compare_rows);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Pgindent run before 9.1 beta2.
2011-06-09 20:32:50 +02:00
* Estimate total numbers of rows in relation. For live rows, use
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
* vac_estimate_reltuples; for dead rows, we have no source of old
* information, so we have to assume the density is the same in unseen
* pages as in the pages we scanned.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
*totalrows = vac_estimate_reltuples(onerel, true,
totalblocks,
bs.m,
liverows);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
if (bs.m > 0)
Fix VACUUM so that it always updates pg_class.reltuples/relpages. When we added the ability for vacuum to skip heap pages by consulting the visibility map, we made it just not update the reltuples/relpages statistics if it skipped any pages. But this could leave us with extremely out-of-date stats for a table that contains any unchanging areas, especially for TOAST tables which never get processed by ANALYZE. In particular this could result in autovacuum making poor decisions about when to process the table, as in recent report from Florian Helmberger. And in general it's a bad idea to not update the stats at all. Instead, use the previous values of reltuples/relpages as an estimate of the tuple density in unvisited pages. This approach results in a "moving average" estimate of reltuples, which should converge to the correct value over multiple VACUUM and ANALYZE cycles even when individual measurements aren't very good. This new method for updating reltuples is used by both VACUUM and ANALYZE, with the result that we no longer need the grotty interconnections that caused ANALYZE to not update the stats depending on what had happened in the parent VACUUM command. Also, fix the logic for skipping all-visible pages during VACUUM so that it looks ahead rather than behind to decide what to do, as per a suggestion from Greg Stark. This eliminates useless scanning of all-visible pages at the start of the relation or just after a not-all-visible page. In particular, the first few pages of the relation will not be invariably included in the scanned pages, which seems to help in not overweighting them in the reltuples estimate. Back-patch to 8.4, where the visibility map was introduced.
2011-05-30 23:05:26 +02:00
*totaldeadrows = floor((deadrows / bs.m) * totalblocks + 0.5);
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
else
Integrate autovacuum functionality into the backend. There's still a few loose ends to be dealt with, but it seems to work. Alvaro Herrera, based on the contrib code by Matthew O'Connor.
2005-07-14 07:13:45 +02:00
*totaldeadrows = 0.0;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Attached is a (very small) patch to make analyze display some interesting info in verbose mode about the analyzed relation (pages, rows per block and rows). Mark Kirkwood
2003-08-26 17:38:42 +02:00
/*
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
* Emit some interesting relation info
Attached is a (very small) patch to make analyze display some interesting info in verbose mode about the analyzed relation (pages, rows per block and rows). Mark Kirkwood
2003-08-26 17:38:42 +02:00
*/
Try to make recently-added analyze log message look something like the others in style.
2003-09-12 00:59:28 +02:00
ereport(elevel,
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
(errmsg("\"%s\": scanned %d of %u pages, "
"containing %.0f live rows and %.0f dead rows; "
"%d rows in sample, %.0f estimated total rows",
Try to make recently-added analyze log message look something like the others in style.
2003-09-12 00:59:28 +02:00
RelationGetRelationName(onerel),
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
bs.m, totalblocks,
liverows, deadrows,
numrows, *totalrows)));
Attached is a (very small) patch to make analyze display some interesting info in verbose mode about the analyzed relation (pages, rows per block and rows). Mark Kirkwood
2003-08-26 17:38:42 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
return numrows;
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Update random() usage so ranges are inclusive/exclusive as required.
2006-02-03 13:45:47 +01:00
/* Select a random value R uniformly distributed in (0 - 1) */
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
double
anl_random_fract(void)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Update random() usage so ranges are inclusive/exclusive as required.
2006-02-03 13:45:47 +01:00
return ((double) random() + 1) / ((double) MAX_RANDOM_VALUE + 2);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
/*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* These two routines embody Algorithm Z from "Random sampling with a
* reservoir" by Jeffrey S. Vitter, in ACM Trans. Math. Softw. 11, 1
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
* (Mar. 1985), Pages 37-57. Vitter describes his algorithm in terms
* of the count S of records to skip before processing another record.
* It is computed primarily based on t, the number of records already read.
* The only extra state needed between calls is W, a random state variable.
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
*
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
* anl_init_selection_state computes the initial W value.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
* Given that we've already read t records (t >= n), anl_get_next_S
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
* determines the number of records to skip before the next record is
* processed.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
double
anl_init_selection_state(int n)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
/* Initial value of W (for use when Algorithm Z is first applied) */
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
return exp(-log(anl_random_fract()) / n);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
double
anl_get_next_S(double t, int n, double *stateptr)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
double S;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* The magic constant here is T from Vitter's paper */
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
if (t <= (22.0 * n))
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
/* Process records using Algorithm X until t is large enough */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
double V,
quot;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
V = anl_random_fract(); /* Generate V */
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
S = 0;
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
t += 1;
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
/* Note: "num" in Vitter's code is always equal to t - n */
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
quot = (t - (double) n) / t;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Find min S satisfying (4.1) */
while (quot > V)
{
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
S += 1;
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
t += 1;
quot *= (t - (double) n) / t;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
}
else
{
/* Now apply Algorithm Z */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
double W = *stateptr;
double term = t - (double) n + 1;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
for (;;)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
double numer,
numer_lim,
denom;
double U,
X,
lhs,
rhs,
y,
tmp;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Generate U and X */
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
U = anl_random_fract();
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
X = t * (W - 1.0);
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
S = floor(X); /* S is tentatively set to floor(X) */
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Test if U <= h(S)/cg(X) in the manner of (6.3) */
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
tmp = (t + 1) / term;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
lhs = exp(log(((U * tmp * tmp) * (term + S)) / (t + X)) / n);
rhs = (((t + X) / (term + S)) * term) / t;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (lhs <= rhs)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
W = rhs / lhs;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
break;
}
/* Test if U <= f(S)/cg(X) */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
y = (((U * (t + 1)) / term) * (t + S + 1)) / (t + X);
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
if ((double) n < S)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
denom = t;
numer_lim = term + S;
}
else
{
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
denom = t - (double) n + S;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
numer_lim = t + 1;
}
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
for (numer = t + S; numer >= numer_lim; numer -= 1)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Don't assume that max offset number stays fixed on a page when we're not holding a pin on the page. Use double instead of long to count rows in relation, so that code still works for > LONG_MAX rows in rel.
2001-07-05 21:33:35 +02:00
y *= numer / denom;
denom -= 1;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
W = exp(-log(anl_random_fract()) / n); /* Generate W in advance */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
if (exp(log(y) / n) <= (t + X) / t)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
break;
}
*stateptr = W;
}
New two-stage sampling method for ANALYZE, as per discussions a few weeks ago. This should give significantly better results when the density of live tuples is not uniform throughout a table. Manfred Koizar, with minor kibitzing from Tom Lane.
2004-05-23 23:24:02 +02:00
return S;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
/*
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
* qsort comparator for sorting rows[] array
*/
static int
compare_rows(const void *a, const void *b)
{
Remove many -Wcast-qual warnings This addresses only those cases that are easy to fix by adding or moving a const qualifier or removing an unnecessary cast. There are many more complicated cases remaining.
2011-09-11 20:54:32 +02:00
HeapTuple ha = *(const HeapTuple *) a;
HeapTuple hb = *(const HeapTuple *) b;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
BlockNumber ba = ItemPointerGetBlockNumber(&ha->t_self);
OffsetNumber oa = ItemPointerGetOffsetNumber(&ha->t_self);
BlockNumber bb = ItemPointerGetBlockNumber(&hb->t_self);
OffsetNumber ob = ItemPointerGetOffsetNumber(&hb->t_self);
if (ba < bb)
return -1;
if (ba > bb)
return 1;
if (oa < ob)
return -1;
if (oa > ob)
return 1;
return 0;
}
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/*
* acquire_inherited_sample_rows -- acquire sample rows from inheritance tree
*
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
* This has the same API as acquire_sample_rows, except that rows are
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
* collected from all inheritance children as well as the specified table.
* We fail and return zero if there are no inheritance children.
*/
static int
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
acquire_inherited_sample_rows(Relation onerel, int elevel,
HeapTuple *rows, int targrows,
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
double *totalrows, double *totaldeadrows)
{
List *tableOIDs;
Relation *rels;
double *relblocks;
double totalblocks;
int numrows,
nrels,
i;
ListCell *lc;
/*
* Find all members of inheritance set. We only need AccessShareLock on
* the children.
*/
Tighten integrity checks on ALTER TABLE ... ALTER COLUMN ... RENAME. When a column is renamed, we recursively rename the same column in all descendent tables. But if one of those tables also inherits that column from a table outside the inheritance hierarchy rooted at the named table, we must throw an error. The previous coding correctly prohibited the rename when the parent had inherited the column from elsewhere, but overlooked the case where the parent was OK but a child table also inherited the same column from a second, unrelated parent. For now, not backpatched due to lack of complaints from the field. KaiGai Kohei, with further changes by me. Reviewed by Bernd Helme and Tom Lane.
2010-02-01 20:28:56 +01:00
tableOIDs =
find_all_inheritors(RelationGetRelid(onerel), AccessShareLock, NULL);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/*
* Check that there's at least one descendant, else fail. This could
* happen despite analyze_rel's relhassubclass check, if table once had a
Teach ANALYZE to clear pg_class.relhassubclass when appropriate. In the past, relhassubclass always remained true if a relation had ever had child relations, even if the last subclass was long gone. While this had only marginal performance implications in most cases, it was annoying, and I'm now considering some planner changes that would raise the cost of a false positive. It was previously impractical to fix this because of race condition concerns. However, given the recent change that made tablecmds.c take ShareExclusiveLock on relations that are gaining a child (commit fbcf4b92aa64d4577bcf25925b055316b978744a), we can now allow ANALYZE to clear the flag when it's no longer relevant. There is no additional locking cost to do so, since ANALYZE takes ShareExclusiveLock anyway.
2011-09-02 20:29:31 +02:00
* child but no longer does. In that case, we can clear the
* relhassubclass field so as not to make the same mistake again later.
* (This is safe because we hold ShareUpdateExclusiveLock.)
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
*/
if (list_length(tableOIDs) < 2)
{
Teach ANALYZE to clear pg_class.relhassubclass when appropriate. In the past, relhassubclass always remained true if a relation had ever had child relations, even if the last subclass was long gone. While this had only marginal performance implications in most cases, it was annoying, and I'm now considering some planner changes that would raise the cost of a false positive. It was previously impractical to fix this because of race condition concerns. However, given the recent change that made tablecmds.c take ShareExclusiveLock on relations that are gaining a child (commit fbcf4b92aa64d4577bcf25925b055316b978744a), we can now allow ANALYZE to clear the flag when it's no longer relevant. There is no additional locking cost to do so, since ANALYZE takes ShareExclusiveLock anyway.
2011-09-02 20:29:31 +02:00
/* CCI because we already updated the pg_class row in this command */
CommandCounterIncrement();
SetRelationHasSubclass(RelationGetRelid(onerel), false);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
return 0;
}
/*
* Count the blocks in all the relations. The result could overflow
* BlockNumber, so we use double arithmetic.
*/
rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
relblocks = (double *) palloc(list_length(tableOIDs) * sizeof(double));
totalblocks = 0;
nrels = 0;
foreach(lc, tableOIDs)
{
Oid childOID = lfirst_oid(lc);
Relation childrel;
/* We already got the needed lock */
childrel = heap_open(childOID, NoLock);
/* Ignore if temp table of another backend */
if (RELATION_IS_OTHER_TEMP(childrel))
{
/* ... but release the lock on it */
Assert(childrel != onerel);
heap_close(childrel, AccessShareLock);
continue;
}
rels[nrels] = childrel;
relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
totalblocks += relblocks[nrels];
nrels++;
}
/*
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* Now sample rows from each relation, proportionally to its fraction of
* the total block count. (This might be less than desirable if the child
* rels have radically different free-space percentages, but it's not
* clear that it's worth working harder.)
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
*/
numrows = 0;
*totalrows = 0;
*totaldeadrows = 0;
for (i = 0; i < nrels; i++)
{
Relation childrel = rels[i];
double childblocks = relblocks[i];
if (childblocks > 0)
{
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
int childtargrows;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
childtargrows = (int) rint(targrows * childblocks / totalblocks);
/* Make sure we don't overrun due to roundoff error */
childtargrows = Min(childtargrows, targrows - numrows);
if (childtargrows > 0)
{
int childrows;
double trows,
tdrows;
/* Fetch a random sample of the child's rows */
childrows = acquire_sample_rows(childrel,
Allow statistics to be collected for foreign tables. ANALYZE now accepts foreign tables and allows the table's FDW to control how the sample rows are collected. (But only manual ANALYZEs will touch foreign tables, for the moment, since among other things it's not very clear how to handle remote permissions checks in an auto-analyze.) contrib/file_fdw is extended to support this. Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
2012-04-06 21:02:35 +02:00
elevel,
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
rows + numrows,
childtargrows,
&trows,
Dept of second thoughts: improve the API for AnalyzeForeignTable. If we make the initially-called function return the table physical-size estimate, acquire_inherited_sample_rows will be able to use that to allocate numbers of samples among child tables, when the day comes that we want to support foreign tables in inheritance trees.
2012-04-06 22:04:10 +02:00
&tdrows);
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
/* We may need to convert from child's rowtype to parent's */
if (childrows > 0 &&
!equalTupleDescs(RelationGetDescr(childrel),
RelationGetDescr(onerel)))
{
TupleConversionMap *map;
map = convert_tuples_by_name(RelationGetDescr(childrel),
RelationGetDescr(onerel),
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
gettext_noop("could not convert row type"));
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
if (map != NULL)
{
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
int j;
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
for (j = 0; j < childrows; j++)
{
HeapTuple newtup;
newtup = do_convert_tuple(rows[numrows + j], map);
heap_freetuple(rows[numrows + j]);
rows[numrows + j] = newtup;
}
free_conversion_map(map);
}
}
/* And add to counts */
numrows += childrows;
*totalrows += trows;
*totaldeadrows += tdrows;
}
}
/*
* Note: we cannot release the child-table locks, since we may have
* pointers to their TOAST tables in the sampled rows.
*/
heap_close(childrel, NoLock);
}
return numrows;
}
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/*
* update_attstats() -- update attribute statistics for one relation
*
* Statistics are stored in several places: the pg_class row for the
* relation has stats about the whole relation, and there is a
* pg_statistic row for each (non-system) attribute that has ever
* been analyzed. The pg_class values are updated by VACUUM, not here.
*
* pg_statistic rows are just added or updated normally. This means
* that pg_statistic will probably contain some deleted rows at the
* completion of a vacuum cycle, unless it happens to get vacuumed last.
*
* To keep things simple, we punt for pg_statistic, and don't try
* to compute or store rows for pg_statistic itself in pg_statistic.
* This could possibly be made to work, but it's not worth the trouble.
* Note analyze_rel() has seen to it that we won't come here when
* vacuuming pg_statistic itself.
*
Change ANALYZE to take ShareUpdateExclusiveLock not AccessShareLock on the table being analyzed. This prevents two ANALYZEs from running concurrently on the same table and possibly suffering concurrent-update failures while trying to store their results into pg_statistic. The downside is that a database-wide ANALYZE executed within a transaction block will hold ShareUpdateExclusiveLock on many tables simultaneously, which could lead to concurrency issues or even deadlock against another such ANALYZE. However, this seems a corner case of less importance than getting unexpected errors from a foreground ANALYZE when autovacuum elects to analyze the same table concurrently. Per discussion.
2006-09-18 00:50:31 +02:00
* Note: there would be a race condition here if two backends could
* ANALYZE the same table concurrently. Presently, we lock that out
* by taking a self-exclusive lock on the relation in analyze_rel().
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
*/
static void
Add the ability to store inheritance-tree statistics in pg_statistic, and teach ANALYZE to compute such stats for tables that have subclasses. Per my proposal of yesterday. autovacuum still needs to be taught about running ANALYZE on parent tables when their subclasses change, but the feature is useful even without that.
2009-12-29 21:11:45 +01:00
update_attstats(Oid relid, bool inh, int natts, VacAttrStats **vacattrstats)
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
{
Relation sd;
int attno;
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
if (natts <= 0)
return; /* nothing to do */
Completion of project to use fixed OIDs for all system catalogs and indexes. Replace all heap_openr and index_openr calls by heap_open and index_open. Remove runtime lookups of catalog OID numbers in various places. Remove relcache's support for looking up system catalogs by name. Bulky but mostly very boring patch ...
2005-04-14 22:03:27 +02:00
sd = heap_open(StatisticRelationId, RowExclusiveLock);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
for (attno = 0; attno < natts; attno++)
{
VacAttrStats *stats = vacattrstats[attno];
HeapTuple stup,
oldtup;
int i,
k,
n;
Datum values[Natts_pg_statistic];
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
bool nulls[Natts_pg_statistic];
bool replaces[Natts_pg_statistic];
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/* Ignore attr if we weren't able to collect stats */
if (!stats->stats_valid)
continue;
/*
* Construct a new pg_statistic tuple
*/
for (i = 0; i < Natts_pg_statistic; ++i)
{
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
nulls[i] = false;
replaces[i] = true;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
}
Index tuple data arrays using Anum_xxx symbolic constants instead of "i++". We had already converted most places to this style, but this patch gets the last few that were still doing it the old way. The main advantage is that this exposes a greppable name for each target column, rather than having to rely on comments (which a couple of places failed to provide anyhow). Richard Hopkins, additional work by me to clean up update_attstats() too
2011-06-16 23:03:58 +02:00
values[Anum_pg_statistic_starelid - 1] = ObjectIdGetDatum(relid);
values[Anum_pg_statistic_staattnum - 1] = Int16GetDatum(stats->attr->attnum);
values[Anum_pg_statistic_stainherit - 1] = BoolGetDatum(inh);
values[Anum_pg_statistic_stanullfrac - 1] = Float4GetDatum(stats->stanullfrac);
values[Anum_pg_statistic_stawidth - 1] = Int32GetDatum(stats->stawidth);
values[Anum_pg_statistic_stadistinct - 1] = Float4GetDatum(stats->stadistinct);
i = Anum_pg_statistic_stakind1 - 1;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
for (k = 0; k < STATISTIC_NUM_SLOTS; k++)
{
values[i++] = Int16GetDatum(stats->stakind[k]); /* stakindN */
}
Index tuple data arrays using Anum_xxx symbolic constants instead of "i++". We had already converted most places to this style, but this patch gets the last few that were still doing it the old way. The main advantage is that this exposes a greppable name for each target column, rather than having to rely on comments (which a couple of places failed to provide anyhow). Richard Hopkins, additional work by me to clean up update_attstats() too
2011-06-16 23:03:58 +02:00
i = Anum_pg_statistic_staop1 - 1;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
for (k = 0; k < STATISTIC_NUM_SLOTS; k++)
{
values[i++] = ObjectIdGetDatum(stats->staop[k]); /* staopN */
}
Index tuple data arrays using Anum_xxx symbolic constants instead of "i++". We had already converted most places to this style, but this patch gets the last few that were still doing it the old way. The main advantage is that this exposes a greppable name for each target column, rather than having to rely on comments (which a couple of places failed to provide anyhow). Richard Hopkins, additional work by me to clean up update_attstats() too
2011-06-16 23:03:58 +02:00
i = Anum_pg_statistic_stanumbers1 - 1;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
for (k = 0; k < STATISTIC_NUM_SLOTS; k++)
{
int nnum = stats->numnumbers[k];
if (nnum > 0)
{
Datum *numdatums = (Datum *) palloc(nnum * sizeof(Datum));
ArrayType *arry;
for (n = 0; n < nnum; n++)
numdatums[n] = Float4GetDatum(stats->stanumbers[k][n]);
/* XXX knows more than it should about type float4: */
arry = construct_array(numdatums, nnum,
FLOAT4OID,
Allow float8, int8, and related datatypes to be passed by value on machines where Datum is 8 bytes wide. Since this will break old-style C functions (those still using version 0 calling convention) that have arguments or results of these types, provide a configure option to disable it and retain the old pass-by-reference behavior. Likewise, provide a configure option to disable the recently-committed float4 pass-by-value change. Zoltan Boszormenyi, plus configurability stuff by me.
2008-04-21 02:26:47 +02:00
sizeof(float4), FLOAT4PASSBYVAL, 'i');
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
values[i++] = PointerGetDatum(arry); /* stanumbersN */
}
else
{
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
nulls[i] = true;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
values[i++] = (Datum) 0;
}
}
Index tuple data arrays using Anum_xxx symbolic constants instead of "i++". We had already converted most places to this style, but this patch gets the last few that were still doing it the old way. The main advantage is that this exposes a greppable name for each target column, rather than having to rely on comments (which a couple of places failed to provide anyhow). Richard Hopkins, additional work by me to clean up update_attstats() too
2011-06-16 23:03:58 +02:00
i = Anum_pg_statistic_stavalues1 - 1;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
for (k = 0; k < STATISTIC_NUM_SLOTS; k++)
{
if (stats->numvalues[k] > 0)
{
ArrayType *arry;
arry = construct_array(stats->stavalues[k],
stats->numvalues[k],
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
stats->statypid[k],
stats->statyplen[k],
stats->statypbyval[k],
stats->statypalign[k]);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
values[i++] = PointerGetDatum(arry); /* stavaluesN */
}
else
{
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
nulls[i] = true;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
values[i++] = (Datum) 0;
}
}
/* Is there already a pg_statistic tuple for this attribute? */
Wrap calls to SearchSysCache and related functions using macros. The purpose of this change is to eliminate the need for every caller of SearchSysCache, SearchSysCacheCopy, SearchSysCacheExists, GetSysCacheOid, and SearchSysCacheList to know the maximum number of allowable keys for a syscache entry (currently 4). This will make it far easier to increase the maximum number of keys in a future release should we choose to do so, and it makes the code shorter, too. Design and review by Tom Lane.
2010-02-14 19:42:19 +01:00
oldtup = SearchSysCache3(STATRELATTINH,
ObjectIdGetDatum(relid),
Int16GetDatum(stats->attr->attnum),
BoolGetDatum(inh));
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
if (HeapTupleIsValid(oldtup))
{
/* Yes, replace it */
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
stup = heap_modify_tuple(oldtup,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
RelationGetDescr(sd),
values,
nulls,
replaces);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
ReleaseSysCache(oldtup);
simple_heap_update(sd, &stup->t_self, stup);
}
else
{
/* No, insert new tuple */
Remove all uses of the deprecated functions heap_formtuple, heap_modifytuple, and heap_deformtuple in favor of the newer functions heap_form_tuple et al (which do the same things but use bool control flags instead of arbitrary char values). Eliminate the former duplicate coding of these functions, reducing the deprecated functions to mere wrappers around the newer ones. We can't get rid of them entirely because add-on modules probably still contain many instances of the old coding style. Kris Jurka
2008-11-02 02:45:28 +01:00
stup = heap_form_tuple(RelationGetDescr(sd), values, nulls);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
simple_heap_insert(sd, stup);
}
/* update indexes too */
CatalogUpdateIndexes(sd, stup);
heap_freetuple(stup);
}
heap_close(sd, RowExclusiveLock);
}
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
/*
* Standard fetch function for use by compute_stats subroutines.
*
* This exists to provide some insulation between compute_stats routines
* and the actual storage of the sample data.
*/
static Datum
std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull)
{
int attnum = stats->tupattnum;
HeapTuple tuple = stats->rows[rownum];
TupleDesc tupDesc = stats->tupDesc;
return heap_getattr(tuple, attnum, tupDesc, isNull);
}
First steps towards statistics on expressional (nee functional) indexes. This commit teaches ANALYZE to store such stats in pg_statistic, but nothing is done yet about teaching the planner to use 'em. Also, repair longstanding oversight in separate ANALYZE command: it updated the pg_class.relpages and reltuples counts for the table proper, but not for indexes.
2004-02-15 22:01:39 +01:00
/*
* Fetch function for analyzing index expressions.
*
* We have not bothered to construct index tuples, instead the data is
* just in Datum arrays.
*/
static Datum
ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull)
{
int i;
/* exprvals and exprnulls are already offset for proper column */
i = rownum * stats->rowstride;
*isNull = stats->exprnulls[i];
return stats->exprvals[i];
}
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/*==========================================================================
*
* Code below this point represents the "standard" type-specific statistics
* analysis algorithms. This code can be replaced on a per-data-type basis
* by setting a nonzero value in pg_type.typanalyze.
*
*==========================================================================
*/
/*
* To avoid consuming too much memory during analysis and/or too much space
* in the resulting pg_statistic rows, we ignore varlena datums that are wider
* than WIDTH_THRESHOLD (after detoasting!). This is legitimate for MCV
* and distinct-value calculations since a wide value is unlikely to be
* duplicated at all, much less be a most-common value. For the same reason,
* ignoring wide values will not affect our estimates of histogram bin
* boundaries very much.
*/
#define WIDTH_THRESHOLD 1024
#define swapInt(a,b) do {int _tmp; _tmp=a; a=b; b=_tmp;} while(0)
#define swapDatum(a,b) do {Datum _tmp; _tmp=a; a=b; b=_tmp;} while(0)
/*
* Extra information used by the default analysis routines
*/
typedef struct
{
Oid eqopr; /* '=' operator for datatype, if any */
Oid eqfunc; /* and associated function */
Oid ltopr; /* '<' operator for datatype, if any */
} StdAnalyzeData;
typedef struct
{
Datum value; /* a data value */
int tupno; /* position index for tuple it came from */
} ScalarItem;
typedef struct
{
int count; /* # of duplicates */
int first; /* values[] index of first occurrence */
} ScalarMCVItem;
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
typedef struct
{
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
SortSupport ssup;
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
int *tupnoLink;
Re-run pgindent with updated list of typedefs. (Updated README should avoid this problem in the future.)
2007-11-15 23:25:18 +01:00
} CompareScalarsContext;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
static void compute_minimal_stats(VacAttrStatsP stats,
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
AnalyzeAttrFetchFunc fetchfunc,
int samplerows,
double totalrows);
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
static void compute_scalar_stats(VacAttrStatsP stats,
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
AnalyzeAttrFetchFunc fetchfunc,
int samplerows,
double totalrows);
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
static int compare_scalars(const void *a, const void *b, void *arg);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
static int compare_mcvs(const void *a, const void *b);
/*
* std_typanalyze -- the default type-specific typanalyze function
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Collect and use element-frequency statistics for arrays. This patch improves selectivity estimation for the array <@, &&, and @> (containment and overlaps) operators. It enables collection of statistics about individual array element values by ANALYZE, and introduces operator-specific estimators that use these stats. In addition, ScalarArrayOpExpr constructs of the forms "const = ANY/ALL (array_column)" and "const <> ANY/ALL (array_column)" are estimated by treating them as variants of the containment operators. Since we still collect scalar-style stats about the array values as a whole, the pg_stats view is expanded to show both these stats and the array-style stats in separate columns. This creates an incompatible change in how stats for tsvector columns are displayed in pg_stats: the stats about lexemes are now displayed in the array-related columns instead of the original scalar-related columns. There are a few loose ends here, notably that it'd be nice to be able to suppress either the scalar-style stats or the array-element stats for columns for which they're not useful. But the patch is in good enough shape to commit for wider testing. Alexander Korotkov, reviewed by Noah Misch and Nathan Boley
2012-03-04 02:20:19 +01:00
bool
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
std_typanalyze(VacAttrStats *stats)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
Form_pg_attribute attr = stats->attr;
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
Oid ltopr;
Oid eqopr;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
StdAnalyzeData *mystats;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/* If the attstattarget column is negative, use the default value */
/* NB: it is okay to scribble on stats->attr since it's a copy */
if (attr->attstattarget < 0)
attr->attstattarget = default_statistics_target;
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
/* Look for default "<" and "=" operators for column's type */
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
get_sort_group_operators(stats->attrtypid,
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
false, false, false,
Provide hashing support for arrays. The core of this patch is hash_array() and associated typcache infrastructure, which works just about exactly like the existing support for array comparison. In addition I did some work to ensure that the planner won't think that an array type is hashable unless its element type is hashable, and similarly for sorting. This includes adding a datatype parameter to op_hashjoinable and op_mergejoinable, and adding an explicit "hashable" flag to SortGroupClause. The lack of a cross-check on the element type was a pre-existing bug in mergejoin support --- but it didn't matter so much before, because if you couldn't sort the element type there wasn't any good alternative to failing anyhow. Now that we have the alternative of hashing the array type, there are cases where we can avoid a failure by being picky at the planner stage, so it's time to be picky. The issue of exactly how to combine the per-element hash values to produce an array hash is still open for discussion, but the rest of this is pretty solid, so I'll commit it as-is.
2010-10-31 02:55:20 +01:00
&ltopr, &eqopr, NULL,
NULL);
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
/* If column has no "=" operator, we can't do much of anything */
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
if (!OidIsValid(eqopr))
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
return false;
/* Save the operator info for compute_stats routines */
mystats = (StdAnalyzeData *) palloc(sizeof(StdAnalyzeData));
mystats->eqopr = eqopr;
Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items as per my recent proposal: 1. Fold SortClause and GroupClause into a single node type SortGroupClause. We were already relying on them to be struct-equivalent, so using two node tags wasn't accomplishing much except to get in the way of comparing items with equal(). 2. Add an "eqop" field to SortGroupClause to carry the associated equality operator. This is cheap for the parser to get at the same time it's looking up the sort operator, and storing it eliminates the need for repeated not-so-cheap lookups during planning. In future this will also let us represent GROUP/DISTINCT operations on datatypes that have hash opclasses but no btree opclasses (ie, they have equality but no natural sort order). The previous representation simply didn't work for that, since its only indicator of comparison semantics was a sort operator. 3. Add a hasDistinctOn boolean to struct Query to explicitly record whether the distinctClause came from DISTINCT or DISTINCT ON. This allows removing some complicated and not 100% bulletproof code that attempted to figure that out from the distinctClause alone. This patch doesn't in itself create any new capability, but it's necessary infrastructure for future attempts to use hash-based grouping for DISTINCT and UNION/INTERSECT/EXCEPT.
2008-08-02 23:32:01 +02:00
mystats->eqfunc = get_opcode(eqopr);
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
mystats->ltopr = ltopr;
stats->extra_data = mystats;
/*
* Determine which standard statistics algorithm to use
*/
if (OidIsValid(ltopr))
{
/* Seems to be a scalar datatype */
stats->compute_stats = compute_scalar_stats;
/*--------------------
* The following choice of minrows is based on the paper
* "Random sampling for histogram construction: how much is enough?"
* by Surajit Chaudhuri, Rajeev Motwani and Vivek Narasayya, in
* Proceedings of ACM SIGMOD International Conference on Management
* of Data, 1998, Pages 436-447. Their Corollary 1 to Theorem 5
* says that for table size n, histogram size k, maximum relative
* error in bin size f, and error probability gamma, the minimum
* random sample size is
* r = 4 * k * ln(2*n/gamma) / f^2
Increase the default value of default_statistics_target from 10 to 100, and its maximum value from 1000 to 10000. ALTER TABLE SET STATISTICS similarly now allows a value up to 10000. Per discussion.
2008-12-13 20:13:44 +01:00
* Taking f = 0.5, gamma = 0.01, n = 10^6 rows, we obtain
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
* r = 305.82 * k
* Note that because of the log function, the dependence on n is
Increase the default value of default_statistics_target from 10 to 100, and its maximum value from 1000 to 10000. ALTER TABLE SET STATISTICS similarly now allows a value up to 10000. Per discussion.
2008-12-13 20:13:44 +01:00
* quite weak; even at n = 10^12, a 300*k sample gives <= 0.66
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
* bin size error with probability 0.99. So there's no real need to
* scale for n, which is a good thing because we don't necessarily
* know it at this point.
*--------------------
*/
stats->minrows = 300 * attr->attstattarget;
}
else
{
/* Can't do much but the minimal stuff */
stats->compute_stats = compute_minimal_stats;
/* Might as well use the same minrows as above */
stats->minrows = 300 * attr->attstattarget;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
return true;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* compute_minimal_stats() -- compute minimal column statistics
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* We use this when we can find only an "=" operator for the datatype.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* We determine the fraction of non-null rows, the average width, the
* most common values, and the (estimated) number of distinct values.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* The most common values are determined by brute force: we keep a list
* of previously seen values, ordered by number of times seen, as we scan
* the samples. A newly seen value is inserted just after the last
* multiply-seen value, causing the bottommost (oldest) singly-seen value
* to drop off the list. The accuracy of this method, and also its cost,
* depend mainly on the length of the list we are willing to keep.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
static void
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
compute_minimal_stats(VacAttrStatsP stats,
AnalyzeAttrFetchFunc fetchfunc,
int samplerows,
double totalrows)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
int i;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
int null_cnt = 0;
int nonnull_cnt = 0;
int toowide_cnt = 0;
double total_width = 0;
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
bool is_varlena = (!stats->attrtype->typbyval &&
stats->attrtype->typlen == -1);
bool is_varwidth = (!stats->attrtype->typbyval &&
stats->attrtype->typlen < 0);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
FmgrInfo f_cmpeq;
typedef struct
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Datum value;
int count;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
} TrackItem;
TrackItem *track;
int track_cnt,
track_max;
int num_mcv = stats->attr->attstattarget;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
StdAnalyzeData *mystats = (StdAnalyzeData *) stats->extra_data;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* We track up to 2*n values for an n-element MCV list; but at least 10
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
track_max = 2 * num_mcv;
if (track_max < 10)
track_max = 10;
track = (TrackItem *) palloc(track_max * sizeof(TrackItem));
track_cnt = 0;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
fmgr_info(mystats->eqfunc, &f_cmpeq);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
for (i = 0; i < samplerows; i++)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
Datum value;
bool isnull;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
bool match;
int firstcount1,
j;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
Centralize implementation of delay code by creating a pg_usleep() subroutine in src/port/pgsleep.c. Remove platform dependencies from miscadmin.h and put them in port.h where they belong. Extend recent vacuum cost-based-delay patch to apply to VACUUM FULL, ANALYZE, and non-btree index vacuuming. By the way, where is the documentation for the cost-based-delay patch?
2004-02-10 04:42:45 +01:00
vacuum_delay_point();
Add CHECK_FOR_INTERRUPTS() in various strategic spots, per comments from Hiroshi.
2002-01-06 01:37:44 +01:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
value = fetchfunc(stats, i, &isnull);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Check for null/nonnull */
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
if (isnull)
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
null_cnt++;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
continue;
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
nonnull_cnt++;
Avoid repeated detoasting (and possible memory leaks) when processing a toasted datum in VACUUM ANALYZE.
2000-12-02 20:38:34 +01:00
/*
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
* If it's a variable-width field, add up widths for average width
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* calculation. Note that if the value is toasted, we use the toasted
* width. We don't bother with this calculation if it's a fixed-width
* type.
Avoid repeated detoasting (and possible memory leaks) when processing a toasted datum in VACUUM ANALYZE.
2000-12-02 20:38:34 +01:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (is_varlena)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Support varlena fields with single-byte headers and unaligned storage. This commit breaks any code that assumes that the mere act of forming a tuple (without writing it to disk) does not "toast" any fields. While all available regression tests pass, I'm not totally sure that we've fixed every nook and cranny, especially in contrib. Greg Stark with some help from Tom Lane
2007-04-06 06:21:44 +02:00
total_width += VARSIZE_ANY(DatumGetPointer(value));
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* If the value is toasted, we want to detoast it just once to
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* avoid repeated detoastings and resultant excess memory usage
* during the comparisons. Also, check to see if the value is
* excessively wide, and if so don't detoast at all --- just
* ignore the value.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (toast_raw_datum_size(value) > WIDTH_THRESHOLD)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
toowide_cnt++;
continue;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
value = PointerGetDatum(PG_DETOAST_DATUM(value));
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
}
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
else if (is_varwidth)
{
/* must be cstring */
total_width += strlen(DatumGetCString(value)) + 1;
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* See if the value matches anything we're already tracking.
*/
match = false;
firstcount1 = track_cnt;
for (j = 0; j < track_cnt; j++)
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Pass collations to functions in FunctionCallInfoData, not FmgrInfo. Since collation is effectively an argument, not a property of the function, FmgrInfo is really the wrong place for it; and this becomes critical in cases where a cached FmgrInfo is used for varying purposes that might need different collation settings. Fix by passing it in FunctionCallInfoData instead. In particular this allows a clean fix for bug #5970 (record_cmp not working). This requires touching a bit more code than the original method, but nobody ever thought that collations would not be an invasive patch...
2011-04-13 01:19:24 +02:00
/* We always use the default collation for statistics */
if (DatumGetBool(FunctionCall2Coll(&f_cmpeq,
DEFAULT_COLLATION_OID,
value, track[j].value)))
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
match = true;
break;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (j < firstcount1 && track[j].count == 1)
firstcount1 = j;
}
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (match)
{
/* Found a match */
track[j].count++;
/* This value may now need to "bubble up" in the track list */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
while (j > 0 && track[j].count > track[j - 1].count)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
swapDatum(track[j].value, track[j - 1].value);
swapInt(track[j].count, track[j - 1].count);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
j--;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
else
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* No match. Insert at head of count-1 list */
if (track_cnt < track_max)
track_cnt++;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
for (j = track_cnt - 1; j > firstcount1; j--)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
track[j].value = track[j - 1].value;
track[j].count = track[j - 1].count;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
if (firstcount1 < track_cnt)
{
track[firstcount1].value = value;
track[firstcount1].count = 1;
}
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
/* We can only compute real stats if we found some non-null values. */
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (nonnull_cnt > 0)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int nmultiple,
summultiple;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stats_valid = true;
/* Do the simple null-frac and width stats */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
stats->stanullfrac = (double) null_cnt / (double) samplerows;
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
if (is_varwidth)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stawidth = total_width / (double) nonnull_cnt;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
else
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stawidth = stats->attrtype->typlen;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Count the number of values we found multiple times */
summultiple = 0;
for (nmultiple = 0; nmultiple < track_cnt; nmultiple++)
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (track[nmultiple].count == 1)
break;
summultiple += track[nmultiple].count;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (nmultiple == 0)
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* If we found no repeated values, assume it's a unique column */
stats->stadistinct = -1.0;
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
else if (track_cnt < track_max && toowide_cnt == 0 &&
nmultiple == track_cnt)
Third round of fmgr updates: eliminate calls using fmgr() and fmgr_faddr() in favor of new-style calls. Lots of cleanup of sloppy casts to use XXXGetDatum and DatumGetXXX ...
2000-05-30 06:25:00 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Our track list includes every value in the sample, and every
* value appeared more than once. Assume the column has just
* these values.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
stats->stadistinct = track_cnt;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
else
{
/*----------
* Estimate the number of distinct values using the estimator
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
* proposed by Haas and Stokes in IBM Research Report RJ 10025:
* n*d / (n - f1 + f1*n/N)
* where f1 is the number of distinct values that occurred
* exactly once in our sample of n rows (from a total of N),
* and d is the total number of distinct values in the sample.
* This is their Duj1 estimator; the other estimators they
* recommend are considerably more complex, and are numerically
* very unstable when n is much smaller than N.
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* We assume (not very reliably!) that all the multiply-occurring
* values are reflected in the final track[] list, and the other
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
* nonnull values all appeared but once. (XXX this usually
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
* results in a drastic overestimate of ndistinct. Can we do
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
* any better?)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*----------
*/
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int f1 = nonnull_cnt - summultiple;
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
int d = f1 + nmultiple;
pgindent run.
2002-09-04 22:31:48 +02:00
double numer,
denom,
stadistinct;
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
numer = (double) samplerows *(double) d;
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
denom = (double) (samplerows - f1) +
(double) f1 *(double) samplerows / totalrows;
pgindent run.
2002-09-04 22:31:48 +02:00
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
stadistinct = numer / denom;
/* Clamp to sane range in case of roundoff error */
if (stadistinct < (double) d)
stadistinct = (double) d;
if (stadistinct > totalrows)
stadistinct = totalrows;
stats->stadistinct = floor(stadistinct + 0.5);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* If we estimated the number of distinct values at more than 10% of
* the total row count (a very arbitrary limit), then assume that
* stadistinct should scale with the row count rather than be a fixed
* value.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (stats->stadistinct > 0.1 * totalrows)
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
stats->stadistinct = -(stats->stadistinct / totalrows);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Decide how many values are worth storing as most-common values. If
* we are able to generate a complete MCV list (all the values in the
* sample will fit, and we think these are all the ones in the table),
* then do so. Otherwise, store only those values that are
* significantly more common than the (estimated) average. We set the
* threshold rather arbitrarily at 25% more than average, with at
* least 2 instances in the sample.
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
*/
if (track_cnt < track_max && toowide_cnt == 0 &&
stats->stadistinct > 0 &&
track_cnt <= num_mcv)
{
/* Track list includes all values seen, and all will fit */
num_mcv = track_cnt;
}
else
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
double ndistinct = stats->stadistinct;
double avgcount,
mincount;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
if (ndistinct < 0)
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
ndistinct = -ndistinct * totalrows;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/* estimate # of occurrences in sample of a typical value */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
avgcount = (double) samplerows / ndistinct;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/* set minimum threshold count to store a value */
mincount = avgcount * 1.25;
if (mincount < 2)
mincount = 2;
if (num_mcv > track_cnt)
num_mcv = track_cnt;
for (i = 0; i < num_mcv; i++)
{
if (track[i].count < mincount)
{
num_mcv = i;
break;
}
}
}
/* Generate MCV slot entry */
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (num_mcv > 0)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
MemoryContext old_context;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Datum *mcv_values;
float4 *mcv_freqs;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
/* Must copy the target values into anl_context */
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
old_context = MemoryContextSwitchTo(stats->anl_context);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
mcv_values = (Datum *) palloc(num_mcv * sizeof(Datum));
mcv_freqs = (float4 *) palloc(num_mcv * sizeof(float4));
for (i = 0; i < num_mcv; i++)
{
mcv_values[i] = datumCopy(track[i].value,
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
stats->attrtype->typbyval,
stats->attrtype->typlen);
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
mcv_freqs[i] = (double) track[i].count / (double) samplerows;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
MemoryContextSwitchTo(old_context);
stats->stakind[0] = STATISTIC_KIND_MCV;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
stats->staop[0] = mystats->eqopr;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stanumbers[0] = mcv_freqs;
stats->numnumbers[0] = num_mcv;
stats->stavalues[0] = mcv_values;
stats->numvalues[0] = num_mcv;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Accept the defaults for stats->statypid and others. They have
* been set before we were called (see vacuum.h)
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
*/
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
}
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
else if (null_cnt > 0)
{
/* We found only nulls; assume the column is entirely null */
stats->stats_valid = true;
stats->stanullfrac = 1.0;
if (is_varwidth)
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
stats->stawidth = 0; /* "unknown" */
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
else
stats->stawidth = stats->attrtype->typlen;
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
stats->stadistinct = 0.0; /* "unknown" */
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* We don't need to bother cleaning up any of our temporary palloc's */
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
/*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* compute_scalar_stats() -- compute column statistics
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* We use this when we can find "=" and "<" operators for the datatype.
Avoid repeated detoasting (and possible memory leaks) when processing a toasted datum in VACUUM ANALYZE.
2000-12-02 20:38:34 +01:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* We determine the fraction of non-null rows, the average width, the
* most common values, the (estimated) number of distinct values, the
* distribution histogram, and the correlation of physical to logical order.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* The desired stats can be determined fairly easily after sorting the
* data values into order.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
static void
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
compute_scalar_stats(VacAttrStatsP stats,
AnalyzeAttrFetchFunc fetchfunc,
int samplerows,
double totalrows)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
int i;
int null_cnt = 0;
int nonnull_cnt = 0;
int toowide_cnt = 0;
double total_width = 0;
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
bool is_varlena = (!stats->attrtype->typbyval &&
stats->attrtype->typlen == -1);
bool is_varwidth = (!stats->attrtype->typbyval &&
stats->attrtype->typlen < 0);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
double corr_xysum;
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
SortSupportData ssup;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
ScalarItem *values;
int values_cnt = 0;
int *tupnoLink;
ScalarMCVItem *track;
int track_cnt = 0;
int num_mcv = stats->attr->attstattarget;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
int num_bins = stats->attr->attstattarget;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
StdAnalyzeData *mystats = (StdAnalyzeData *) stats->extra_data;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
values = (ScalarItem *) palloc(samplerows * sizeof(ScalarItem));
tupnoLink = (int *) palloc(samplerows * sizeof(int));
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
track = (ScalarMCVItem *) palloc(num_mcv * sizeof(ScalarMCVItem));
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
memset(&ssup, 0, sizeof(ssup));
ssup.ssup_cxt = CurrentMemoryContext;
/* We always use the default collation for statistics */
ssup.ssup_collation = DEFAULT_COLLATION_OID;
ssup.ssup_nulls_first = false;
PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Initial scan to find sortable values */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
for (i = 0; i < samplerows; i++)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Datum value;
bool isnull;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Centralize implementation of delay code by creating a pg_usleep() subroutine in src/port/pgsleep.c. Remove platform dependencies from miscadmin.h and put them in port.h where they belong. Extend recent vacuum cost-based-delay patch to apply to VACUUM FULL, ANALYZE, and non-btree index vacuuming. By the way, where is the documentation for the cost-based-delay patch?
2004-02-10 04:42:45 +01:00
vacuum_delay_point();
Add CHECK_FOR_INTERRUPTS() in various strategic spots, per comments from Hiroshi.
2002-01-06 01:37:44 +01:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
value = fetchfunc(stats, i, &isnull);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Check for null/nonnull */
if (isnull)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
null_cnt++;
continue;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
nonnull_cnt++;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
* If it's a variable-width field, add up widths for average width
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* calculation. Note that if the value is toasted, we use the toasted
* width. We don't bother with this calculation if it's a fixed-width
* type.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (is_varlena)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Support varlena fields with single-byte headers and unaligned storage. This commit breaks any code that assumes that the mere act of forming a tuple (without writing it to disk) does not "toast" any fields. While all available regression tests pass, I'm not totally sure that we've fixed every nook and cranny, especially in contrib. Greg Stark with some help from Tom Lane
2007-04-06 06:21:44 +02:00
total_width += VARSIZE_ANY(DatumGetPointer(value));
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* If the value is toasted, we want to detoast it just once to
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* avoid repeated detoastings and resultant excess memory usage
* during the comparisons. Also, check to see if the value is
* excessively wide, and if so don't detoast at all --- just
* ignore the value.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (toast_raw_datum_size(value) > WIDTH_THRESHOLD)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
toowide_cnt++;
continue;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
value = PointerGetDatum(PG_DETOAST_DATUM(value));
}
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
else if (is_varwidth)
{
/* must be cstring */
total_width += strlen(DatumGetCString(value)) + 1;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Add it to the list to be sorted */
values[values_cnt].value = value;
values[values_cnt].tupno = values_cnt;
tupnoLink[values_cnt] = values_cnt;
values_cnt++;
}
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
/* We can only compute real stats if we found some sortable values. */
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (values_cnt > 0)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int ndistinct, /* # distinct values in sample */
nmultiple, /* # that appear multiple times */
num_hist,
dups_cnt;
int slot_idx = 0;
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
CompareScalarsContext cxt;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Sort the collected values */
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
cxt.ssup = &ssup;
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
cxt.tupnoLink = tupnoLink;
qsort_arg((void *) values, values_cnt, sizeof(ScalarItem),
compare_scalars, (void *) &cxt);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Now scan the values in order, find the most common ones, and also
* accumulate ordering-correlation statistics.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
2005-11-22 19:17:34 +01:00
* To determine which are most common, we first have to count the
* number of duplicates of each value. The duplicates are adjacent in
* the sorted list, so a brute-force approach is to compare successive
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* datum values until we find two that are not equal. However, that
* requires N-1 invocations of the datum comparison routine, which are
* completely redundant with work that was done during the sort. (The
* sort algorithm must at some point have compared each pair of items
* that are adjacent in the sorted order; otherwise it could not know
* that it's ordered the pair correctly.) We exploit this by having
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
* compare_scalars remember the highest tupno index that each
* ScalarItem has been found equal to. At the end of the sort, a
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* ScalarItem's tupnoLink will still point to itself if and only if it
* is the last item of its group of duplicates (since the group will
* be ordered by tupno).
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
corr_xysum = 0;
ndistinct = 0;
nmultiple = 0;
dups_cnt = 0;
for (i = 0; i < values_cnt; i++)
{
int tupno = values[i].tupno;
Fix a couple of places where lack of parenthesization of a cast causes pgindent to make weird formatting decisions. Easiest fix seems to be to put in the extra parens...
2001-10-25 22:37:30 +02:00
corr_xysum += ((double) i) * ((double) tupno);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
dups_cnt++;
if (tupnoLink[tupno] == tupno)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Reached end of duplicates of this value */
ndistinct++;
if (dups_cnt > 1)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
nmultiple++;
if (track_cnt < num_mcv ||
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
dups_cnt > track[track_cnt - 1].count)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
/*
* Found a new item for the mcv list; find its
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* position, bubbling down old items if needed. Loop
* invariant is that j points at an empty/ replaceable
* slot.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int j;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (track_cnt < num_mcv)
track_cnt++;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
for (j = track_cnt - 1; j > 0; j--)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
if (dups_cnt <= track[j - 1].count)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
break;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
track[j].count = track[j - 1].count;
track[j].first = track[j - 1].first;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
track[j].count = dups_cnt;
track[j].first = i + 1 - dups_cnt;
}
}
dups_cnt = 0;
}
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stats_valid = true;
/* Do the simple null-frac and width stats */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
stats->stanullfrac = (double) null_cnt / (double) samplerows;
The cstring datatype can now be copied, passed around, etc. The typlen value '-2' is used to indicate a variable-width type whose width is computed as strlen(datum)+1. Everything that looks at typlen is updated except for array support, which Joe Conway is working on; at the moment it wouldn't work to try to create an array of cstring.
2002-08-24 17:00:47 +02:00
if (is_varwidth)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stawidth = total_width / (double) nonnull_cnt;
else
stats->stawidth = stats->attrtype->typlen;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (nmultiple == 0)
{
/* If we found no repeated values, assume it's a unique column */
stats->stadistinct = -1.0;
}
else if (toowide_cnt == 0 && nmultiple == ndistinct)
{
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Every value in the sample appeared more than once. Assume the
* column has just these values.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
stats->stadistinct = ndistinct;
}
else
{
/*----------
* Estimate the number of distinct values using the estimator
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
* proposed by Haas and Stokes in IBM Research Report RJ 10025:
* n*d / (n - f1 + f1*n/N)
* where f1 is the number of distinct values that occurred
* exactly once in our sample of n rows (from a total of N),
* and d is the total number of distinct values in the sample.
* This is their Duj1 estimator; the other estimators they
* recommend are considerably more complex, and are numerically
* very unstable when n is much smaller than N.
*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* Overwidth values are assumed to have been distinct.
*----------
*/
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int f1 = ndistinct - nmultiple + toowide_cnt;
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
int d = f1 + nmultiple;
pgindent run.
2002-09-04 22:31:48 +02:00
double numer,
denom,
stadistinct;
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
numer = (double) samplerows *(double) d;
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
denom = (double) (samplerows - f1) +
(double) f1 *(double) samplerows / totalrows;
pgindent run.
2002-09-04 22:31:48 +02:00
Replace number-of-distinct-values estimator equation, per recent pghackers discussion.
2002-02-18 17:04:14 +01:00
stadistinct = numer / denom;
/* Clamp to sane range in case of roundoff error */
if (stadistinct < (double) d)
stadistinct = (double) d;
if (stadistinct > totalrows)
stadistinct = totalrows;
stats->stadistinct = floor(stadistinct + 0.5);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
}
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* If we estimated the number of distinct values at more than 10% of
* the total row count (a very arbitrary limit), then assume that
* stadistinct should scale with the row count rather than be a fixed
* value.
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
if (stats->stadistinct > 0.1 * totalrows)
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
stats->stadistinct = -(stats->stadistinct / totalrows);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Decide how many values are worth storing as most-common values. If
* we are able to generate a complete MCV list (all the values in the
* sample will fit, and we think these are all the ones in the table),
* then do so. Otherwise, store only those values that are
* significantly more common than the (estimated) average. We set the
* threshold rather arbitrarily at 25% more than average, with at
* least 2 instances in the sample. Also, we won't suppress values
* that have a frequency of at least 1/K where K is the intended
* number of histogram bins; such values might otherwise cause us to
Clarify a confusing comment about MCVs vs histogram entries. Per Nathan Boley.
2009-01-07 00:46:06 +01:00
* emit duplicate histogram bin boundaries. (We might end up with
* duplicate histogram entries anyway, if the distribution is skewed;
* but we prefer to treat such values as MCVs if at all possible.)
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
*/
if (track_cnt == ndistinct && toowide_cnt == 0 &&
stats->stadistinct > 0 &&
track_cnt <= num_mcv)
{
/* Track list includes all values seen, and all will fit */
num_mcv = track_cnt;
}
else
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
double ndistinct = stats->stadistinct;
double avgcount,
mincount,
maxmincount;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
if (ndistinct < 0)
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
ndistinct = -ndistinct * totalrows;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/* estimate # of occurrences in sample of a typical value */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
avgcount = (double) samplerows / ndistinct;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
/* set minimum threshold count to store a value */
mincount = avgcount * 1.25;
if (mincount < 2)
mincount = 2;
/* don't let threshold exceed 1/K, however */
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
maxmincount = (double) samplerows / (double) num_bins;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
if (mincount > maxmincount)
mincount = maxmincount;
if (num_mcv > track_cnt)
num_mcv = track_cnt;
for (i = 0; i < num_mcv; i++)
{
if (track[i].count < mincount)
{
num_mcv = i;
break;
}
}
}
/* Generate MCV slot entry */
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (num_mcv > 0)
{
MemoryContext old_context;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Datum *mcv_values;
float4 *mcv_freqs;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
/* Must copy the target values into anl_context */
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
old_context = MemoryContextSwitchTo(stats->anl_context);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
mcv_values = (Datum *) palloc(num_mcv * sizeof(Datum));
mcv_freqs = (float4 *) palloc(num_mcv * sizeof(float4));
for (i = 0; i < num_mcv; i++)
{
mcv_values[i] = datumCopy(values[track[i].first].value,
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
stats->attrtype->typbyval,
stats->attrtype->typlen);
After further thought about support for gathering stats on functional indexes, it seems like we ought to put another layer of indirection between the compute_stats functions and the actual data storage. This would allow us to compute the values on-the-fly, for example.
2004-02-13 07:39:49 +01:00
mcv_freqs[i] = (double) track[i].count / (double) samplerows;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
MemoryContextSwitchTo(old_context);
stats->stakind[slot_idx] = STATISTIC_KIND_MCV;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
stats->staop[slot_idx] = mystats->eqopr;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stanumbers[slot_idx] = mcv_freqs;
stats->numnumbers[slot_idx] = num_mcv;
stats->stavalues[slot_idx] = mcv_values;
stats->numvalues[slot_idx] = num_mcv;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Accept the defaults for stats->statypid and others. They have
* been set before we were called (see vacuum.h)
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
slot_idx++;
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Generate a histogram slot entry if there are at least two distinct
* values not accounted for in the MCV list. (This ensures the
* histogram won't collapse to empty or a singleton.)
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
num_hist = ndistinct - num_mcv;
Be a little smarter about deciding how many most-common values to save.
2001-06-06 23:29:17 +02:00
if (num_hist > num_bins)
num_hist = num_bins + 1;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (num_hist >= 2)
{
MemoryContext old_context;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Datum *hist_values;
int nvals;
Avoid integer overflow in the loop that extracts histogram entries from ANALYZE's total sample. The original coding is at risk of overflow for statistics targets exceeding about 2675; this was not a problem before 8.4 but it is now. Per bug #4793 from Dennis Noordsij.
2009-05-05 20:02:11 +02:00
int pos,
posfrac,
delta,
deltafrac;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* Sort the MCV items into position order to speed next loop */
qsort((void *) track, num_mcv,
sizeof(ScalarMCVItem), compare_mcvs);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
/*
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* Collapse out the MCV items from the values[] array.
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Note we destroy the values[] array here... but we don't need it
* for anything more. We do, however, still need values_cnt.
* nvals will be the number of remaining entries in values[].
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (num_mcv > 0)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int src,
dest;
int j;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
src = dest = 0;
j = 0; /* index of next interesting MCV item */
while (src < values_cnt)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int ncopy;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (j < num_mcv)
{
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
int first = track[j].first;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
if (src >= first)
{
/* advance past this MCV item */
src = first + track[j].count;
j++;
continue;
}
ncopy = first - src;
}
else
ncopy = values_cnt - src;
memmove(&values[dest], &values[src],
ncopy * sizeof(ScalarItem));
src += ncopy;
dest += ncopy;
}
nvals = dest;
}
else
nvals = values_cnt;
Assert(nvals >= num_hist);
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
/* Must copy the target values into anl_context */
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
old_context = MemoryContextSwitchTo(stats->anl_context);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
hist_values = (Datum *) palloc(num_hist * sizeof(Datum));
Avoid integer overflow in the loop that extracts histogram entries from ANALYZE's total sample. The original coding is at risk of overflow for statistics targets exceeding about 2675; this was not a problem before 8.4 but it is now. Per bug #4793 from Dennis Noordsij.
2009-05-05 20:02:11 +02:00
/*
* The object of this loop is to copy the first and last values[]
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* entries along with evenly-spaced values in between. So the
Avoid integer overflow in the loop that extracts histogram entries from ANALYZE's total sample. The original coding is at risk of overflow for statistics targets exceeding about 2675; this was not a problem before 8.4 but it is now. Per bug #4793 from Dennis Noordsij.
2009-05-05 20:02:11 +02:00
* i'th value is values[(i * (nvals - 1)) / (num_hist - 1)]. But
* computing that subscript directly risks integer overflow when
* the stats target is more than a couple thousand. Instead we
* add (nvals - 1) / (num_hist - 1) to pos at each step, tracking
* the integral and fractional parts of the sum separately.
*/
delta = (nvals - 1) / (num_hist - 1);
deltafrac = (nvals - 1) % (num_hist - 1);
pos = posfrac = 0;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
for (i = 0; i < num_hist; i++)
{
hist_values[i] = datumCopy(values[pos].value,
Fix ANALYZE's ancient deficiency of not trying to collect stats for expression indexes when the index column type (the opclass opckeytype) is different from the expression's datatype. When coded, this limitation wasn't worth worrying about because we had no intelligence to speak of in stats collection for the datatypes used by such opclasses. However, now that there's non-toy estimation capability for tsvector queries, it amounts to a bug that ANALYZE fails to do this. The fix changes struct VacAttrStats, and therefore constitutes an API break for custom typanalyze functions. Therefore we can't back-patch it into released branches, but it was agreed that 9.0 isn't yet frozen hard enough to make such a change unacceptable. Ergo, back-patch to 9.0 but no further. The API break had better be mentioned in 9.0 release notes.
2010-08-02 00:38:11 +02:00
stats->attrtype->typbyval,
stats->attrtype->typlen);
Avoid integer overflow in the loop that extracts histogram entries from ANALYZE's total sample. The original coding is at risk of overflow for statistics targets exceeding about 2675; this was not a problem before 8.4 but it is now. Per bug #4793 from Dennis Noordsij.
2009-05-05 20:02:11 +02:00
pos += delta;
posfrac += deltafrac;
if (posfrac >= (num_hist - 1))
{
/* fractional part exceeds 1, carry to integer part */
pos++;
posfrac -= (num_hist - 1);
}
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
Avoid integer overflow in the loop that extracts histogram entries from ANALYZE's total sample. The original coding is at risk of overflow for statistics targets exceeding about 2675; this was not a problem before 8.4 but it is now. Per bug #4793 from Dennis Noordsij.
2009-05-05 20:02:11 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
MemoryContextSwitchTo(old_context);
stats->stakind[slot_idx] = STATISTIC_KIND_HISTOGRAM;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
stats->staop[slot_idx] = mystats->ltopr;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stavalues[slot_idx] = hist_values;
stats->numvalues[slot_idx] = num_hist;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Accept the defaults for stats->statypid and others. They have
* been set before we were called (see vacuum.h)
Extend VacAttrStats to allow typanalyze functions to store statistic values of different types than the underlying column. The capability isn't yet used for anything, but will be required by upcoming patch to analyze tsvector columns. Jan Urbanski
2008-07-01 12:33:09 +02:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
slot_idx++;
}
/* Generate a correlation entry if there are multiple values */
if (values_cnt > 1)
{
MemoryContext old_context;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
float4 *corrs;
double corr_xsum,
corr_x2sum;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
Fix up memory leakage created by recent changes.
2002-06-16 00:25:40 +02:00
/* Must copy the target values into anl_context */
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
old_context = MemoryContextSwitchTo(stats->anl_context);
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
corrs = (float4 *) palloc(sizeof(float4));
MemoryContextSwitchTo(old_context);
/*----------
* Since we know the x and y value sets are both
* 0, 1, ..., values_cnt-1
* we have sum(x) = sum(y) =
* (values_cnt-1)*values_cnt / 2
* and sum(x^2) = sum(y^2) =
* (values_cnt-1)*values_cnt*(2*values_cnt-1) / 6.
*----------
*/
Fix a couple of places where lack of parenthesization of a cast causes pgindent to make weird formatting decisions. Easiest fix seems to be to put in the extra parens...
2001-10-25 22:37:30 +02:00
corr_xsum = ((double) (values_cnt - 1)) *
((double) values_cnt) / 2.0;
corr_x2sum = ((double) (values_cnt - 1)) *
((double) values_cnt) * (double) (2 * values_cnt - 1) / 6.0;
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* And the correlation coefficient reduces to */
corrs[0] = (values_cnt * corr_xysum - corr_xsum * corr_xsum) /
(values_cnt * corr_x2sum - corr_xsum * corr_xsum);
stats->stakind[slot_idx] = STATISTIC_KIND_CORRELATION;
Add hooks for type-specific calculation of ANALYZE statistics. Idea and coding by Mark Cave-Ayland, some kibitzing by Tom Lane. initdb forced due to new column in pg_type.
2004-02-13 00:41:04 +01:00
stats->staop[slot_idx] = mystats->ltopr;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
stats->stanumbers[slot_idx] = corrs;
stats->numnumbers[slot_idx] = 1;
slot_idx++;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
}
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
else if (nonnull_cnt == 0 && null_cnt > 0)
{
/* We found only nulls; assume the column is entirely null */
stats->stats_valid = true;
stats->stanullfrac = 1.0;
if (is_varwidth)
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
stats->stawidth = 0; /* "unknown" */
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
else
stats->stawidth = stats->attrtype->typlen;
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
stats->stadistinct = 0.0; /* "unknown" */
Fix ANALYZE to accumulate some minimal statistics for an all-null column. Per gripes from Mike Mascari and Bernd Heller.
2005-02-11 01:41:12 +01:00
}
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/* We don't need to bother cleaning up any of our temporary palloc's */
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
}
/*
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
* qsort_arg comparator for sorting ScalarItems
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
* Aside from sorting the items, we update the tupnoLink[] array
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
* whenever two ScalarItems are found to contain equal datums. The array
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
* is indexed by tupno; for each ScalarItem, it contains the highest
* tupno that that item's datum has been found to be equal to. This allows
* us to avoid additional comparisons in compute_scalar_stats().
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
*/
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
static int
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
compare_scalars(const void *a, const void *b, void *arg)
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
{
Remove many -Wcast-qual warnings This addresses only those cases that are easy to fix by adding or moving a const qualifier or removing an unnecessary cast. There are many more complicated cases remaining.
2011-09-11 20:54:32 +02:00
Datum da = ((const ScalarItem *) a)->value;
int ta = ((const ScalarItem *) a)->tupno;
Datum db = ((const ScalarItem *) b)->value;
int tb = ((const ScalarItem *) b)->tupno;
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
CompareScalarsContext *cxt = (CompareScalarsContext *) arg;
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
int compare;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Create a "sort support" interface API for faster sorting. This patch creates an API whereby a btree index opclass can optionally provide non-SQL-callable support functions for sorting. In the initial patch, we only use this to provide a directly-callable comparator function, which can be invoked with a bit less overhead than the traditional SQL-callable comparator. While that should be of value in itself, the real reason for doing this is to provide a datatype-extensible framework for more aggressive optimizations, as in Peter Geoghegan's recent work. Robert Haas and Tom Lane
2011-12-07 06:18:38 +01:00
compare = ApplySortComparator(da, false, db, false, cxt->ssup);
Tweak sorting so that nulls appear at the front of a descending sort (vs. at the end of a normal sort). This ensures that explicit sorts yield the same ordering as a btree index scan. To be really sure that that equivalence holds, we use the btree entries in pg_amop to decide whether we are looking at a '<' or '>' operator. For a sort operator that has no btree association, we put the nulls at the front if the operator is named '>' ... pretty grotty, but it does the right thing in simple ASC and DESC cases, and at least there's no possibility of getting a different answer depending on the plan type chosen.
2001-06-02 21:01:53 +02:00
if (compare != 0)
return compare;
Add analyze.c file for split.
2000-05-29 19:44:17 +02:00
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
* The two datums are equal, so update cxt->tupnoLink[].
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
*/
Make use of qsort_arg in several places that were formerly using klugy static variables. This avoids any risk of potential non-reentrancy, and in particular offers a much cleaner workaround for the Intel compiler bug that was affecting ginutil.c.
2006-10-05 19:57:40 +02:00
if (cxt->tupnoLink[ta] < tb)
cxt->tupnoLink[ta] = tb;
if (cxt->tupnoLink[tb] < ta)
cxt->tupnoLink[tb] = ta;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
/*
* For equal datums, sort by tupno
*/
return ta - tb;
}
/*
* qsort comparator for sorting ScalarMCVItems by position
*/
static int
compare_mcvs(const void *a, const void *b)
{
Remove many -Wcast-qual warnings This addresses only those cases that are easy to fix by adding or moving a const qualifier or removing an unnecessary cast. There are many more complicated cases remaining.
2011-09-11 20:54:32 +02:00
int da = ((const ScalarMCVItem *) a)->first;
int db = ((const ScalarMCVItem *) b)->first;
Rewrite of planner statistics-gathering code. ANALYZE is now available as a separate statement (though it can still be invoked as part of VACUUM, too). pg_statistic redesigned to be more flexible about what statistics are stored. ANALYZE now collects a list of several of the most common values, not just one, plus a histogram (not just the min and max values). Random sampling is used to make the process reasonably fast even on very large tables. The number of values and histogram bins collected is now user-settable via an ALTER TABLE command. There is more still to do; the new stats are not being used everywhere they could be in the planner. But the remaining changes for this project should be localized, and the behavior is already better than before. A not-very-related change is that sorting now makes use of btree comparison routines if it can find one, rather than invoking '<' twice.
2001-05-07 02:43:27 +02:00
return da - db;
}