2009-01-04 23:19:59 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* pg_stat_statements.c
|
|
|
|
* Track statement execution times across a whole database cluster.
|
|
|
|
*
|
2012-03-29 03:00:31 +02:00
|
|
|
* Execution costs are totalled for each distinct source query, and kept in
|
2014-05-06 18:12:18 +02:00
|
|
|
* a shared hashtable. (We track only as many distinct queries as will fit
|
2012-03-29 03:00:31 +02:00
|
|
|
* in the designated amount of shared memory.)
|
|
|
|
*
|
|
|
|
* As of Postgres 9.2, this module normalizes query entries. Normalization
|
|
|
|
* is a process whereby similar queries, typically differing only in their
|
|
|
|
* constants (though the exact rules are somewhat more subtle than that) are
|
|
|
|
* recognized as equivalent, and are tracked as a single entry. This is
|
|
|
|
* particularly useful for non-prepared queries.
|
|
|
|
*
|
|
|
|
* Normalization is implemented by fingerprinting queries, selectively
|
|
|
|
* serializing those fields of each query tree's nodes that are judged to be
|
2014-05-06 18:12:18 +02:00
|
|
|
* essential to the query. This is referred to as a query jumble. This is
|
2012-03-29 03:00:31 +02:00
|
|
|
* distinct from a regular serialization in that various extraneous
|
|
|
|
* information is ignored as irrelevant or not essential to the query, such
|
|
|
|
* as the collations of Vars and, most notably, the values of constants.
|
|
|
|
*
|
|
|
|
* This jumble is acquired at the end of parse analysis of each query, and
|
2017-10-12 01:52:46 +02:00
|
|
|
* a 64-bit hash of it is stored into the query's Query.queryId field.
|
2012-03-29 03:00:31 +02:00
|
|
|
* The server then copies this value around, making it available in plan
|
|
|
|
* tree(s) generated from the query. The executor can then use this value
|
|
|
|
* to blame query costs on the proper queryId.
|
|
|
|
*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* To facilitate presenting entries to users, we create "representative" query
|
2017-03-28 02:14:36 +02:00
|
|
|
* strings in which constants are replaced with parameter symbols ($n), to
|
|
|
|
* make it clearer what a normalized entry can represent. To save on shared
|
|
|
|
* memory, and to avoid having to truncate oversized query strings, we store
|
|
|
|
* these strings in a temporary external query-texts file. Offsets into this
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* file are kept in shared memory.
|
|
|
|
*
|
2009-01-04 23:19:59 +01:00
|
|
|
* Note about locking issues: to create or delete an entry in the shared
|
|
|
|
* hashtable, one must hold pgss->lock exclusively. Modifying any field
|
|
|
|
* in an entry except the counters requires the same. To look up an entry,
|
|
|
|
* one must hold the lock shared. To read or update the counters within
|
|
|
|
* an entry, one must hold the lock shared or exclusive (so the entry doesn't
|
|
|
|
* disappear!) and also take the entry's mutex spinlock.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* The shared state variable pgss->extent (the next free spot in the external
|
|
|
|
* query-text file) should be accessed only while holding either the
|
|
|
|
* pgss->mutex spinlock, or exclusive lock on pgss->lock. We use the mutex to
|
|
|
|
* allow reserving file space while holding only shared lock on pgss->lock.
|
|
|
|
* Rewriting the entire external query-text file, eg for garbage collection,
|
|
|
|
* requires holding pgss->lock exclusively; this allows individual entries
|
|
|
|
* in the file to be read or written while holding only shared lock.
|
2009-01-04 23:19:59 +01:00
|
|
|
*
|
|
|
|
*
|
2018-01-03 05:30:12 +01:00
|
|
|
* Copyright (c) 2008-2018, PostgreSQL Global Development Group
|
2009-01-04 23:19:59 +01:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* contrib/pg_stat_statements/pg_stat_statements.c
|
2009-01-04 23:19:59 +01:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
#include "postgres.h"
|
|
|
|
|
2015-03-28 14:22:51 +01:00
|
|
|
#include <math.h>
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
#include <sys/stat.h>
|
2009-01-04 23:19:59 +01:00
|
|
|
#include <unistd.h>
|
|
|
|
|
|
|
|
#include "access/hash.h"
|
2017-03-30 20:18:53 +02:00
|
|
|
#include "catalog/pg_authid.h"
|
2009-01-04 23:19:59 +01:00
|
|
|
#include "executor/instrument.h"
|
2010-01-08 01:38:20 +01:00
|
|
|
#include "funcapi.h"
|
2009-01-04 23:19:59 +01:00
|
|
|
#include "mb/pg_wchar.h"
|
|
|
|
#include "miscadmin.h"
|
2012-03-29 03:00:31 +02:00
|
|
|
#include "parser/analyze.h"
|
|
|
|
#include "parser/parsetree.h"
|
|
|
|
#include "parser/scanner.h"
|
2017-01-14 22:17:30 +01:00
|
|
|
#include "parser/scansup.h"
|
2014-02-03 15:19:49 +01:00
|
|
|
#include "pgstat.h"
|
2009-01-04 23:19:59 +01:00
|
|
|
#include "storage/fd.h"
|
|
|
|
#include "storage/ipc.h"
|
2011-09-04 07:13:16 +02:00
|
|
|
#include "storage/spin.h"
|
2009-12-15 21:04:49 +01:00
|
|
|
#include "tcop/utility.h"
|
2018-04-15 02:12:14 +02:00
|
|
|
#include "utils/acl.h"
|
2009-01-04 23:19:59 +01:00
|
|
|
#include "utils/builtins.h"
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
#include "utils/memutils.h"
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
PG_MODULE_MAGIC;
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Location of permanent stats file (valid when database is shut down) */
|
2014-06-04 05:09:45 +02:00
|
|
|
#define PGSS_DUMP_FILE PGSTAT_STAT_PERMANENT_DIRECTORY "/pg_stat_statements.stat"
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* Location of external query text file. We don't keep it in the core
|
|
|
|
* system's stats_temp_directory. The core system can safely use that GUC
|
|
|
|
* setting, because the statistics collector temp file paths are set only once
|
|
|
|
* as part of changing the GUC, but pg_stat_statements has no way of avoiding
|
|
|
|
* race conditions. Besides, we only expect modest, infrequent I/O for query
|
|
|
|
* strings, so placing the file on a faster filesystem is not compelling.
|
|
|
|
*/
|
2014-02-03 15:19:49 +01:00
|
|
|
#define PGSS_TEXT_FILE PG_STAT_TMP_DIR "/pgss_query_texts.stat"
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
/* Magic number identifying the stats file format */
|
2017-10-12 01:52:46 +02:00
|
|
|
static const uint32 PGSS_FILE_HEADER = 0x20171004;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
2013-12-07 18:06:02 +01:00
|
|
|
/* PostgreSQL major version number, changes in which invalidate all entries */
|
|
|
|
static const uint32 PGSS_PG_MAJOR_VERSION = PG_VERSION_NUM / 100;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* XXX: Should USAGE_EXEC reflect execution time and/or buffer usage? */
|
|
|
|
#define USAGE_EXEC(duration) (1.0)
|
|
|
|
#define USAGE_INIT (1.0) /* including initial planning */
|
2012-04-08 21:49:47 +02:00
|
|
|
#define ASSUMED_MEDIAN_INIT (10.0) /* initial assumed median usage */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
#define ASSUMED_LENGTH_INIT 1024 /* initial assumed mean query length */
|
2009-01-04 23:19:59 +01:00
|
|
|
#define USAGE_DECREASE_FACTOR (0.99) /* decreased every entry_dealloc */
|
2012-04-08 21:49:47 +02:00
|
|
|
#define STICKY_DECREASE_FACTOR (0.50) /* factor for sticky entries */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
#define USAGE_DEALLOC_PERCENT 5 /* free this % of entries at once */
|
2009-01-04 23:19:59 +01:00
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
#define JUMBLE_SIZE 1024 /* query serialization buffer size */
|
|
|
|
|
2013-12-07 18:06:02 +01:00
|
|
|
/*
|
|
|
|
* Extension version number, for supporting older extension versions' objects
|
|
|
|
*/
|
|
|
|
typedef enum pgssVersion
|
|
|
|
{
|
|
|
|
PGSS_V1_0 = 0,
|
|
|
|
PGSS_V1_1,
|
2015-03-27 20:43:22 +01:00
|
|
|
PGSS_V1_2,
|
|
|
|
PGSS_V1_3
|
2013-12-07 18:06:02 +01:00
|
|
|
} pgssVersion;
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
2012-03-29 03:00:31 +02:00
|
|
|
* Hashtable key that defines the identity of a hashtable entry. We separate
|
|
|
|
* queries by user and by database even if they are otherwise identical.
|
2017-10-20 15:40:17 +02:00
|
|
|
*
|
|
|
|
* Right now, this structure contains no padding. If you add any, make sure
|
|
|
|
* to teach pgss_store() to zero the padding bytes. Otherwise, things will
|
|
|
|
* break, because pgss_hash is created using HASH_BLOBS, and thus tag_hash
|
|
|
|
* is used to hash this.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
typedef struct pgssHashKey
|
|
|
|
{
|
|
|
|
Oid userid; /* user OID */
|
|
|
|
Oid dbid; /* database OID */
|
2017-10-12 01:52:46 +02:00
|
|
|
uint64 queryid; /* query identifier */
|
2009-01-04 23:19:59 +01:00
|
|
|
} pgssHashKey;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The actual stats counters kept within pgssEntry.
|
|
|
|
*/
|
|
|
|
typedef struct Counters
|
|
|
|
{
|
2010-02-26 03:01:40 +01:00
|
|
|
int64 calls; /* # of times executed */
|
2012-04-28 22:03:57 +02:00
|
|
|
double total_time; /* total execution time, in msec */
|
2017-02-06 10:33:58 +01:00
|
|
|
double min_time; /* minimum execution time in msec */
|
2015-05-24 03:35:49 +02:00
|
|
|
double max_time; /* maximum execution time in msec */
|
|
|
|
double mean_time; /* mean execution time in msec */
|
|
|
|
double sum_var_time; /* sum of variances in execution time in msec */
|
2010-02-26 03:01:40 +01:00
|
|
|
int64 rows; /* total # of retrieved or affected rows */
|
2012-06-10 21:20:04 +02:00
|
|
|
int64 shared_blks_hit; /* # of shared buffer hits */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
int64 shared_blks_read; /* # of shared disk blocks read */
|
2012-02-23 02:33:05 +01:00
|
|
|
int64 shared_blks_dirtied; /* # of shared disk blocks dirtied */
|
2010-02-26 03:01:40 +01:00
|
|
|
int64 shared_blks_written; /* # of shared disk blocks written */
|
2012-06-10 21:20:04 +02:00
|
|
|
int64 local_blks_hit; /* # of local buffer hits */
|
|
|
|
int64 local_blks_read; /* # of local disk blocks read */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
int64 local_blks_dirtied; /* # of local disk blocks dirtied */
|
|
|
|
int64 local_blks_written; /* # of local disk blocks written */
|
2012-06-10 21:20:04 +02:00
|
|
|
int64 temp_blks_read; /* # of temp blocks read */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
int64 temp_blks_written; /* # of temp blocks written */
|
2012-06-10 21:20:04 +02:00
|
|
|
double blk_read_time; /* time spent reading, in msec */
|
|
|
|
double blk_write_time; /* time spent writing, in msec */
|
|
|
|
double usage; /* usage factor */
|
2009-01-04 23:19:59 +01:00
|
|
|
} Counters;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Statistics per statement
|
|
|
|
*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Note: in event of a failure in garbage collection of the query text file,
|
|
|
|
* we reset query_offset to zero and query_len to -1. This will be seen as
|
|
|
|
* an invalid state by qtext_fetch().
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
typedef struct pgssEntry
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
pgssHashKey key; /* hash key of entry - MUST BE FIRST */
|
2009-01-04 23:19:59 +01:00
|
|
|
Counters counters; /* the statistics for this query */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Size query_offset; /* query text offset in external file */
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
int query_len; /* # of valid bytes in query string, or -1 */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
int encoding; /* query text encoding */
|
2009-01-04 23:19:59 +01:00
|
|
|
slock_t mutex; /* protects the counters only */
|
|
|
|
} pgssEntry;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Global shared state
|
|
|
|
*/
|
|
|
|
typedef struct pgssSharedState
|
|
|
|
{
|
2014-01-27 17:07:44 +01:00
|
|
|
LWLock *lock; /* protects hashtable search/modification */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
double cur_median_usage; /* current median usage in hashtable */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Size mean_query_len; /* current mean entry text length */
|
|
|
|
slock_t mutex; /* protects following fields only: */
|
|
|
|
Size extent; /* current extent of query file */
|
|
|
|
int n_writers; /* number of active writers to query file */
|
|
|
|
int gc_count; /* query file garbage collection cycle count */
|
2009-01-04 23:19:59 +01:00
|
|
|
} pgssSharedState;
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/*
|
|
|
|
* Struct for tracking locations/lengths of constants during normalization
|
|
|
|
*/
|
|
|
|
typedef struct pgssLocationLen
|
|
|
|
{
|
|
|
|
int location; /* start offset in query text */
|
|
|
|
int length; /* length in bytes, or -1 to ignore */
|
2012-06-10 21:20:04 +02:00
|
|
|
} pgssLocationLen;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Working state for computing a query jumble and producing a normalized
|
|
|
|
* query string
|
|
|
|
*/
|
|
|
|
typedef struct pgssJumbleState
|
|
|
|
{
|
|
|
|
/* Jumble of current query tree */
|
|
|
|
unsigned char *jumble;
|
|
|
|
|
|
|
|
/* Number of bytes used in jumble[] */
|
|
|
|
Size jumble_len;
|
|
|
|
|
|
|
|
/* Array of locations of constants that should be removed */
|
|
|
|
pgssLocationLen *clocations;
|
|
|
|
|
|
|
|
/* Allocated length of clocations array */
|
|
|
|
int clocations_buf_size;
|
|
|
|
|
|
|
|
/* Current number of valid entries in clocations array */
|
|
|
|
int clocations_count;
|
2017-03-28 02:14:36 +02:00
|
|
|
|
|
|
|
/* highest Param id we've seen, in order to start normalization correctly */
|
|
|
|
int highest_extern_param_id;
|
2012-06-10 21:20:04 +02:00
|
|
|
} pgssJumbleState;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*---- Local variables ----*/
|
|
|
|
|
2012-03-29 22:42:09 +02:00
|
|
|
/* Current nesting depth of ExecutorRun+ProcessUtility calls */
|
2009-06-11 16:49:15 +02:00
|
|
|
static int nested_level = 0;
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/* Saved hook values in case of unload */
|
2009-06-11 16:49:15 +02:00
|
|
|
static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
|
2012-03-29 03:00:31 +02:00
|
|
|
static post_parse_analyze_hook_type prev_post_parse_analyze_hook = NULL;
|
2009-06-11 16:49:15 +02:00
|
|
|
static ExecutorStart_hook_type prev_ExecutorStart = NULL;
|
|
|
|
static ExecutorRun_hook_type prev_ExecutorRun = NULL;
|
2011-02-27 19:43:29 +01:00
|
|
|
static ExecutorFinish_hook_type prev_ExecutorFinish = NULL;
|
2009-06-11 16:49:15 +02:00
|
|
|
static ExecutorEnd_hook_type prev_ExecutorEnd = NULL;
|
2009-12-15 21:04:49 +01:00
|
|
|
static ProcessUtility_hook_type prev_ProcessUtility = NULL;
|
2009-06-11 16:49:15 +02:00
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/* Links to shared memory state */
|
2009-06-11 16:49:15 +02:00
|
|
|
static pgssSharedState *pgss = NULL;
|
|
|
|
static HTAB *pgss_hash = NULL;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/*---- GUC variables ----*/
|
|
|
|
|
|
|
|
typedef enum
|
|
|
|
{
|
|
|
|
PGSS_TRACK_NONE, /* track no statements */
|
|
|
|
PGSS_TRACK_TOP, /* only top level statements */
|
2009-12-15 21:04:49 +01:00
|
|
|
PGSS_TRACK_ALL /* all statements, including nested ones */
|
2017-06-21 20:39:04 +02:00
|
|
|
} PGSSTrackLevel;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
2010-01-08 01:38:20 +01:00
|
|
|
static const struct config_enum_entry track_options[] =
|
|
|
|
{
|
2009-01-04 23:19:59 +01:00
|
|
|
{"none", PGSS_TRACK_NONE, false},
|
|
|
|
{"top", PGSS_TRACK_TOP, false},
|
|
|
|
{"all", PGSS_TRACK_ALL, false},
|
|
|
|
{NULL, 0, false}
|
|
|
|
};
|
|
|
|
|
|
|
|
static int pgss_max; /* max # statements to track */
|
|
|
|
static int pgss_track; /* tracking level */
|
2010-02-26 03:01:40 +01:00
|
|
|
static bool pgss_track_utility; /* whether to track utility commands */
|
2009-01-04 23:19:59 +01:00
|
|
|
static bool pgss_save; /* whether to save stats across shutdown */
|
|
|
|
|
|
|
|
|
|
|
|
#define pgss_enabled() \
|
|
|
|
(pgss_track == PGSS_TRACK_ALL || \
|
|
|
|
(pgss_track == PGSS_TRACK_TOP && nested_level == 0))
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
#define record_gc_qtexts() \
|
|
|
|
do { \
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss; \
|
|
|
|
SpinLockAcquire(&s->mutex); \
|
|
|
|
s->gc_count++; \
|
|
|
|
SpinLockRelease(&s->mutex); \
|
|
|
|
} while(0)
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*---- Function declarations ----*/
|
|
|
|
|
2009-06-11 16:49:15 +02:00
|
|
|
void _PG_init(void);
|
|
|
|
void _PG_fini(void);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
PG_FUNCTION_INFO_V1(pg_stat_statements_reset);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
PG_FUNCTION_INFO_V1(pg_stat_statements_1_2);
|
2015-03-27 20:43:22 +01:00
|
|
|
PG_FUNCTION_INFO_V1(pg_stat_statements_1_3);
|
2009-01-04 23:19:59 +01:00
|
|
|
PG_FUNCTION_INFO_V1(pg_stat_statements);
|
|
|
|
|
|
|
|
static void pgss_shmem_startup(void);
|
|
|
|
static void pgss_shmem_shutdown(int code, Datum arg);
|
2012-03-29 03:00:31 +02:00
|
|
|
static void pgss_post_parse_analyze(ParseState *pstate, Query *query);
|
2009-01-04 23:19:59 +01:00
|
|
|
static void pgss_ExecutorStart(QueryDesc *queryDesc, int eflags);
|
|
|
|
static void pgss_ExecutorRun(QueryDesc *queryDesc,
|
2009-06-11 16:49:15 +02:00
|
|
|
ScanDirection direction,
|
2017-03-23 18:05:48 +01:00
|
|
|
uint64 count, bool execute_once);
|
2011-02-27 19:43:29 +01:00
|
|
|
static void pgss_ExecutorFinish(QueryDesc *queryDesc);
|
2009-01-04 23:19:59 +01:00
|
|
|
static void pgss_ExecutorEnd(QueryDesc *queryDesc);
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
static void pgss_ProcessUtility(PlannedStmt *pstmt, const char *queryString,
|
2013-04-28 06:18:45 +02:00
|
|
|
ProcessUtilityContext context, ParamListInfo params,
|
2017-04-01 06:17:18 +02:00
|
|
|
QueryEnvironment *queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
DestReceiver *dest, char *completionTag);
|
2017-10-12 01:52:46 +02:00
|
|
|
static uint64 pgss_hash_string(const char *str, int len);
|
|
|
|
static void pgss_store(const char *query, uint64 queryId,
|
2017-01-14 22:17:30 +01:00
|
|
|
int query_location, int query_len,
|
2012-03-29 03:00:31 +02:00
|
|
|
double total_time, uint64 rows,
|
|
|
|
const BufferUsage *bufusage,
|
2012-06-10 21:20:04 +02:00
|
|
|
pgssJumbleState *jstate);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
static void pg_stat_statements_internal(FunctionCallInfo fcinfo,
|
|
|
|
pgssVersion api_version,
|
|
|
|
bool showtext);
|
2009-06-11 16:49:15 +02:00
|
|
|
static Size pgss_memsize(void);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
static pgssEntry *entry_alloc(pgssHashKey *key, Size query_offset, int query_len,
|
|
|
|
int encoding, bool sticky);
|
2009-01-04 23:19:59 +01:00
|
|
|
static void entry_dealloc(void);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
static bool qtext_store(const char *query, int query_len,
|
|
|
|
Size *query_offset, int *gc_count);
|
|
|
|
static char *qtext_load_file(Size *buffer_size);
|
|
|
|
static char *qtext_fetch(Size query_offset, int query_len,
|
|
|
|
char *buffer, Size buffer_size);
|
|
|
|
static bool need_gc_qtexts(void);
|
|
|
|
static void gc_qtexts(void);
|
2009-01-04 23:19:59 +01:00
|
|
|
static void entry_reset(void);
|
2012-06-10 21:20:04 +02:00
|
|
|
static void AppendJumble(pgssJumbleState *jstate,
|
2012-03-29 03:00:31 +02:00
|
|
|
const unsigned char *item, Size size);
|
2012-06-10 21:20:04 +02:00
|
|
|
static void JumbleQuery(pgssJumbleState *jstate, Query *query);
|
|
|
|
static void JumbleRangeTable(pgssJumbleState *jstate, List *rtable);
|
|
|
|
static void JumbleExpr(pgssJumbleState *jstate, Node *node);
|
|
|
|
static void RecordConstLocation(pgssJumbleState *jstate, int location);
|
|
|
|
static char *generate_normalized_query(pgssJumbleState *jstate, const char *query,
|
2017-01-14 22:17:30 +01:00
|
|
|
int query_loc, int *query_len_p, int encoding);
|
|
|
|
static void fill_in_constant_lengths(pgssJumbleState *jstate, const char *query,
|
|
|
|
int query_loc);
|
2012-03-29 03:00:31 +02:00
|
|
|
static int comp_location(const void *a, const void *b);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Module load callback
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
_PG_init(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* In order to create our shared memory area, we have to be loaded via
|
2009-06-11 16:49:15 +02:00
|
|
|
* shared_preload_libraries. If not, fall out without hooking into any of
|
|
|
|
* the main system. (We don't throw error here because it seems useful to
|
|
|
|
* allow the pg_stat_statements functions to be created even when the
|
|
|
|
* module isn't active. The functions must protect themselves against
|
|
|
|
* being called then, however.)
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
if (!process_shared_preload_libraries_in_progress)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Define (or redefine) custom GUC variables.
|
|
|
|
*/
|
|
|
|
DefineCustomIntVariable("pg_stat_statements.max",
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
"Sets the maximum number of statements tracked by pg_stat_statements.",
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL,
|
|
|
|
&pgss_max,
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
5000,
|
2009-01-04 23:19:59 +01:00
|
|
|
100,
|
Be more wary about 32-bit integer overflow in pg_stat_statements.
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files. It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:
1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits. We still need overflow guards on its use, but
this helps.
2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize. If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.
3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms. It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.
Per report from Bruno da Silva. I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place. But it did get that big, so #2 is a reasonable defense,
and #3 could explain the reported performance difficulties.
(See also commit 8bbe4cbd9, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)
This issue is old, and is primarily a problem for old platforms,
so back-patch.
Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
2022-08-03 00:05:34 +02:00
|
|
|
INT_MAX / 2,
|
2009-01-04 23:19:59 +01:00
|
|
|
PGC_POSTMASTER,
|
|
|
|
0,
|
|
|
|
NULL,
|
2011-04-07 06:11:01 +02:00
|
|
|
NULL,
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL);
|
|
|
|
|
|
|
|
DefineCustomEnumVariable("pg_stat_statements.track",
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
"Selects which statements are tracked by pg_stat_statements.",
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL,
|
|
|
|
&pgss_track,
|
|
|
|
PGSS_TRACK_TOP,
|
|
|
|
track_options,
|
|
|
|
PGC_SUSET,
|
|
|
|
0,
|
|
|
|
NULL,
|
2011-04-07 06:11:01 +02:00
|
|
|
NULL,
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL);
|
|
|
|
|
2009-12-15 21:04:49 +01:00
|
|
|
DefineCustomBoolVariable("pg_stat_statements.track_utility",
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
"Selects whether utility commands are tracked by pg_stat_statements.",
|
2009-12-15 21:04:49 +01:00
|
|
|
NULL,
|
|
|
|
&pgss_track_utility,
|
|
|
|
true,
|
|
|
|
PGC_SUSET,
|
|
|
|
0,
|
|
|
|
NULL,
|
2011-04-07 06:11:01 +02:00
|
|
|
NULL,
|
2009-12-15 21:04:49 +01:00
|
|
|
NULL);
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
DefineCustomBoolVariable("pg_stat_statements.save",
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
"Save pg_stat_statements statistics across server shutdowns.",
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL,
|
|
|
|
&pgss_save,
|
|
|
|
true,
|
|
|
|
PGC_SIGHUP,
|
|
|
|
0,
|
|
|
|
NULL,
|
2011-04-07 06:11:01 +02:00
|
|
|
NULL,
|
2009-01-04 23:19:59 +01:00
|
|
|
NULL);
|
|
|
|
|
2009-01-05 14:35:38 +01:00
|
|
|
EmitWarningsOnPlaceholders("pg_stat_statements");
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
|
|
|
* Request additional shared resources. (These are no-ops if we're not in
|
|
|
|
* the postmaster process.) We'll allocate or attach to the shared
|
|
|
|
* resources in pgss_shmem_startup().
|
|
|
|
*/
|
|
|
|
RequestAddinShmemSpace(pgss_memsize());
|
2016-02-04 22:43:04 +01:00
|
|
|
RequestNamedLWLockTranche("pg_stat_statements", 1);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Install hooks.
|
|
|
|
*/
|
|
|
|
prev_shmem_startup_hook = shmem_startup_hook;
|
|
|
|
shmem_startup_hook = pgss_shmem_startup;
|
2012-03-29 03:00:31 +02:00
|
|
|
prev_post_parse_analyze_hook = post_parse_analyze_hook;
|
|
|
|
post_parse_analyze_hook = pgss_post_parse_analyze;
|
2009-01-04 23:19:59 +01:00
|
|
|
prev_ExecutorStart = ExecutorStart_hook;
|
|
|
|
ExecutorStart_hook = pgss_ExecutorStart;
|
|
|
|
prev_ExecutorRun = ExecutorRun_hook;
|
|
|
|
ExecutorRun_hook = pgss_ExecutorRun;
|
2011-02-27 19:43:29 +01:00
|
|
|
prev_ExecutorFinish = ExecutorFinish_hook;
|
|
|
|
ExecutorFinish_hook = pgss_ExecutorFinish;
|
2009-01-04 23:19:59 +01:00
|
|
|
prev_ExecutorEnd = ExecutorEnd_hook;
|
|
|
|
ExecutorEnd_hook = pgss_ExecutorEnd;
|
2009-12-15 21:04:49 +01:00
|
|
|
prev_ProcessUtility = ProcessUtility_hook;
|
|
|
|
ProcessUtility_hook = pgss_ProcessUtility;
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Module unload callback
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
_PG_fini(void)
|
|
|
|
{
|
|
|
|
/* Uninstall hooks. */
|
2009-12-15 21:04:49 +01:00
|
|
|
shmem_startup_hook = prev_shmem_startup_hook;
|
2012-03-29 03:00:31 +02:00
|
|
|
post_parse_analyze_hook = prev_post_parse_analyze_hook;
|
2009-01-04 23:19:59 +01:00
|
|
|
ExecutorStart_hook = prev_ExecutorStart;
|
|
|
|
ExecutorRun_hook = prev_ExecutorRun;
|
2011-02-27 19:43:29 +01:00
|
|
|
ExecutorFinish_hook = prev_ExecutorFinish;
|
2009-01-04 23:19:59 +01:00
|
|
|
ExecutorEnd_hook = prev_ExecutorEnd;
|
2009-12-15 21:04:49 +01:00
|
|
|
ProcessUtility_hook = prev_ProcessUtility;
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* shmem_startup hook: allocate or attach to shared memory,
|
|
|
|
* then load any pre-existing statistics from file.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Also create and load the query-texts file, which is expected to exist
|
|
|
|
* (even if empty) while the module is enabled.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_shmem_startup(void)
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
bool found;
|
2009-01-04 23:19:59 +01:00
|
|
|
HASHCTL info;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
FILE *file = NULL;
|
|
|
|
FILE *qfile = NULL;
|
2009-01-04 23:19:59 +01:00
|
|
|
uint32 header;
|
|
|
|
int32 num;
|
2013-12-07 18:06:02 +01:00
|
|
|
int32 pgver;
|
2009-01-04 23:19:59 +01:00
|
|
|
int32 i;
|
|
|
|
int buffer_size;
|
|
|
|
char *buffer = NULL;
|
|
|
|
|
|
|
|
if (prev_shmem_startup_hook)
|
|
|
|
prev_shmem_startup_hook();
|
|
|
|
|
|
|
|
/* reset in case this is a restart within the postmaster */
|
|
|
|
pgss = NULL;
|
|
|
|
pgss_hash = NULL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create or attach to the shared memory state, including hash table
|
|
|
|
*/
|
|
|
|
LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
|
|
|
|
|
|
|
|
pgss = ShmemInitStruct("pg_stat_statements",
|
|
|
|
sizeof(pgssSharedState),
|
|
|
|
&found);
|
|
|
|
|
|
|
|
if (!found)
|
2009-06-11 16:49:15 +02:00
|
|
|
{
|
2009-01-04 23:19:59 +01:00
|
|
|
/* First time through ... */
|
2016-02-04 22:43:04 +01:00
|
|
|
pgss->lock = &(GetNamedLWLockTranche("pg_stat_statements"))->lock;
|
2012-04-08 21:49:47 +02:00
|
|
|
pgss->cur_median_usage = ASSUMED_MEDIAN_INIT;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
pgss->mean_query_len = ASSUMED_LENGTH_INIT;
|
|
|
|
SpinLockInit(&pgss->mutex);
|
|
|
|
pgss->extent = 0;
|
|
|
|
pgss->n_writers = 0;
|
|
|
|
pgss->gc_count = 0;
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
memset(&info, 0, sizeof(info));
|
|
|
|
info.keysize = sizeof(pgssHashKey);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
info.entrysize = sizeof(pgssEntry);
|
2009-01-04 23:19:59 +01:00
|
|
|
pgss_hash = ShmemInitHash("pg_stat_statements hash",
|
|
|
|
pgss_max, pgss_max,
|
|
|
|
&info,
|
2017-10-12 01:52:46 +02:00
|
|
|
HASH_ELEM | HASH_BLOBS);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
LWLockRelease(AddinShmemInitLock);
|
|
|
|
|
|
|
|
/*
|
2009-06-11 16:49:15 +02:00
|
|
|
* If we're in the postmaster (or a standalone backend...), set up a shmem
|
|
|
|
* exit hook to dump the statistics to disk.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
if (!IsUnderPostmaster)
|
|
|
|
on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
|
|
|
|
|
|
|
|
/*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Done if some other process already completed our initialization.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (found)
|
2009-01-04 23:19:59 +01:00
|
|
|
return;
|
|
|
|
|
2009-07-27 06:09:55 +02:00
|
|
|
/*
|
|
|
|
* Note: we don't bother with locks here, because there should be no other
|
|
|
|
* processes running when this code is reached.
|
|
|
|
*/
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
/* Unlink query text file possibly left over from crash */
|
|
|
|
unlink(PGSS_TEXT_FILE);
|
|
|
|
|
|
|
|
/* Allocate new query text temp file */
|
|
|
|
qfile = AllocateFile(PGSS_TEXT_FILE, PG_BINARY_W);
|
|
|
|
if (qfile == NULL)
|
|
|
|
goto write_error;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we were told not to load old statistics, we're done. (Note we do
|
|
|
|
* not try to unlink any old dump file in this case. This seems a bit
|
|
|
|
* questionable but it's the historical behavior.)
|
|
|
|
*/
|
|
|
|
if (!pgss_save)
|
|
|
|
{
|
|
|
|
FreeFile(qfile);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Attempt to load old statistics from the dump file.
|
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
file = AllocateFile(PGSS_DUMP_FILE, PG_BINARY_R);
|
|
|
|
if (file == NULL)
|
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (errno != ENOENT)
|
|
|
|
goto read_error;
|
|
|
|
/* No existing persisted stats file, so we're done */
|
|
|
|
FreeFile(qfile);
|
|
|
|
return;
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
buffer_size = 2048;
|
2009-01-04 23:19:59 +01:00
|
|
|
buffer = (char *) palloc(buffer_size);
|
|
|
|
|
|
|
|
if (fread(&header, sizeof(uint32), 1, file) != 1 ||
|
2013-12-07 18:06:02 +01:00
|
|
|
fread(&pgver, sizeof(uint32), 1, file) != 1 ||
|
2009-01-04 23:19:59 +01:00
|
|
|
fread(&num, sizeof(int32), 1, file) != 1)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
goto read_error;
|
|
|
|
|
|
|
|
if (header != PGSS_FILE_HEADER ||
|
|
|
|
pgver != PGSS_PG_MAJOR_VERSION)
|
|
|
|
goto data_error;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
for (i = 0; i < num; i++)
|
|
|
|
{
|
|
|
|
pgssEntry temp;
|
2009-06-11 16:49:15 +02:00
|
|
|
pgssEntry *entry;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Size query_offset;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (fread(&temp, sizeof(pgssEntry), 1, file) != 1)
|
|
|
|
goto read_error;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* Encoding is the only field we can easily sanity-check */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (!PG_VALID_BE_ENCODING(temp.encoding))
|
|
|
|
goto data_error;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Resize buffer as needed */
|
2012-03-29 03:00:31 +02:00
|
|
|
if (temp.query_len >= buffer_size)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
buffer_size = Max(buffer_size * 2, temp.query_len + 1);
|
|
|
|
buffer = repalloc(buffer, buffer_size);
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (fread(buffer, 1, temp.query_len + 1, file) != temp.query_len + 1)
|
|
|
|
goto read_error;
|
|
|
|
|
|
|
|
/* Should have a trailing null, but let's make sure */
|
2012-03-29 03:00:31 +02:00
|
|
|
buffer[temp.query_len] = '\0';
|
|
|
|
|
|
|
|
/* Skip loading "sticky" entries */
|
|
|
|
if (temp.counters.calls == 0)
|
|
|
|
continue;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Store the query text */
|
|
|
|
query_offset = pgss->extent;
|
|
|
|
if (fwrite(buffer, 1, temp.query_len + 1, qfile) != temp.query_len + 1)
|
|
|
|
goto write_error;
|
|
|
|
pgss->extent += temp.query_len + 1;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* make the hashtable entry (discards old entries if too many) */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
entry = entry_alloc(&temp.key, query_offset, temp.query_len,
|
|
|
|
temp.encoding,
|
|
|
|
false);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* copy in the actual stats */
|
|
|
|
entry->counters = temp.counters;
|
|
|
|
}
|
|
|
|
|
|
|
|
pfree(buffer);
|
|
|
|
FreeFile(file);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
FreeFile(qfile);
|
2012-05-27 10:54:31 +02:00
|
|
|
|
|
|
|
/*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Remove the persisted stats file so it's not included in
|
|
|
|
* backups/replication slaves, etc. A new file will be written on next
|
|
|
|
* shutdown.
|
|
|
|
*
|
|
|
|
* Note: it's okay if the PGSS_TEXT_FILE is included in a basebackup,
|
|
|
|
* because we remove that file on startup; it acts inversely to
|
|
|
|
* PGSS_DUMP_FILE, in that it is only supposed to be around when the
|
|
|
|
* server is running, whereas PGSS_DUMP_FILE is only supposed to be around
|
2014-05-06 18:12:18 +02:00
|
|
|
* when the server is not running. Leaving the file creates no danger of
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* a newly restored database having a spurious record of execution costs,
|
|
|
|
* which is what we're really concerned about here.
|
2012-05-27 10:54:31 +02:00
|
|
|
*/
|
|
|
|
unlink(PGSS_DUMP_FILE);
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
return;
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
read_error:
|
2009-01-04 23:19:59 +01:00
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not read pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_DUMP_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
goto fail;
|
|
|
|
data_error:
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("ignoring invalid data in pg_stat_statement file \"%s\"",
|
|
|
|
PGSS_DUMP_FILE)));
|
|
|
|
goto fail;
|
|
|
|
write_error:
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
fail:
|
2009-01-04 23:19:59 +01:00
|
|
|
if (buffer)
|
|
|
|
pfree(buffer);
|
|
|
|
if (file)
|
|
|
|
FreeFile(file);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (qfile)
|
|
|
|
FreeFile(qfile);
|
2009-01-04 23:19:59 +01:00
|
|
|
/* If possible, throw away the bogus file; ignore any error */
|
|
|
|
unlink(PGSS_DUMP_FILE);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Don't unlink PGSS_TEXT_FILE here; it should always be around while the
|
|
|
|
* server is running with pg_stat_statements enabled
|
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* shmem_shutdown hook: Dump statistics into file.
|
|
|
|
*
|
|
|
|
* Note: we don't bother with acquiring lock, because there should be no
|
|
|
|
* other processes running when this is called.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_shmem_shutdown(int code, Datum arg)
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
FILE *file;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
char *qbuffer = NULL;
|
|
|
|
Size qbuffer_size = 0;
|
2009-06-11 16:49:15 +02:00
|
|
|
HASH_SEQ_STATUS hash_seq;
|
|
|
|
int32 num_entries;
|
|
|
|
pgssEntry *entry;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* Don't try to dump during a crash. */
|
|
|
|
if (code)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Safety check ... shouldn't get here unless shmem is set up. */
|
|
|
|
if (!pgss || !pgss_hash)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Don't dump if told not to. */
|
|
|
|
if (!pgss_save)
|
|
|
|
return;
|
|
|
|
|
2012-05-27 10:54:31 +02:00
|
|
|
file = AllocateFile(PGSS_DUMP_FILE ".tmp", PG_BINARY_W);
|
2009-01-04 23:19:59 +01:00
|
|
|
if (file == NULL)
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
if (fwrite(&PGSS_FILE_HEADER, sizeof(uint32), 1, file) != 1)
|
|
|
|
goto error;
|
2013-12-07 18:06:02 +01:00
|
|
|
if (fwrite(&PGSS_PG_MAJOR_VERSION, sizeof(uint32), 1, file) != 1)
|
|
|
|
goto error;
|
2009-01-04 23:19:59 +01:00
|
|
|
num_entries = hash_get_num_entries(pgss_hash);
|
|
|
|
if (fwrite(&num_entries, sizeof(int32), 1, file) != 1)
|
|
|
|
goto error;
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
qbuffer = qtext_load_file(&qbuffer_size);
|
|
|
|
if (qbuffer == NULL)
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* When serializing to disk, we store query texts immediately after their
|
2014-05-06 18:12:18 +02:00
|
|
|
* entry data. Any orphaned query texts are thereby excluded.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
2012-03-29 03:00:31 +02:00
|
|
|
int len = entry->query_len;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
char *qstr = qtext_fetch(entry->query_offset, len,
|
|
|
|
qbuffer, qbuffer_size);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (qstr == NULL)
|
|
|
|
continue; /* Ignore any entries with bogus texts */
|
|
|
|
|
|
|
|
if (fwrite(entry, sizeof(pgssEntry), 1, file) != 1 ||
|
|
|
|
fwrite(qstr, 1, len + 1, file) != len + 1)
|
|
|
|
{
|
|
|
|
/* note: we assume hash_seq_term won't change errno */
|
|
|
|
hash_seq_term(&hash_seq);
|
2009-01-04 23:19:59 +01:00
|
|
|
goto error;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
free(qbuffer);
|
|
|
|
qbuffer = NULL;
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
if (FreeFile(file))
|
|
|
|
{
|
|
|
|
file = NULL;
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
|
2012-05-27 10:54:31 +02:00
|
|
|
/*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Rename file into place, so we atomically replace any old one.
|
2012-05-27 10:54:31 +02:00
|
|
|
*/
|
2016-03-10 03:53:53 +01:00
|
|
|
(void) durable_rename(PGSS_DUMP_FILE ".tmp", PGSS_DUMP_FILE, LOG);
|
2012-05-27 10:54:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Unlink query-texts file; it's not needed while shutdown */
|
|
|
|
unlink(PGSS_TEXT_FILE);
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
return;
|
|
|
|
|
|
|
|
error:
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
2012-06-10 21:20:04 +02:00
|
|
|
PGSS_DUMP_FILE ".tmp")));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (qbuffer)
|
|
|
|
free(qbuffer);
|
2009-01-04 23:19:59 +01:00
|
|
|
if (file)
|
|
|
|
FreeFile(file);
|
2012-05-27 10:54:31 +02:00
|
|
|
unlink(PGSS_DUMP_FILE ".tmp");
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
unlink(PGSS_TEXT_FILE);
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/*
|
|
|
|
* Post-parse-analysis hook: mark query with a queryId
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_post_parse_analyze(ParseState *pstate, Query *query)
|
|
|
|
{
|
|
|
|
pgssJumbleState jstate;
|
|
|
|
|
2014-04-21 19:28:07 +02:00
|
|
|
if (prev_post_parse_analyze_hook)
|
|
|
|
prev_post_parse_analyze_hook(pstate, query);
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/* Assert we didn't do this already */
|
2017-10-12 01:52:46 +02:00
|
|
|
Assert(query->queryId == UINT64CONST(0));
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/* Safety check... */
|
|
|
|
if (!pgss || !pgss_hash)
|
|
|
|
return;
|
|
|
|
|
2012-03-29 21:32:50 +02:00
|
|
|
/*
|
|
|
|
* Utility statements get queryId zero. We do this even in cases where
|
|
|
|
* the statement contains an optimizable statement for which a queryId
|
|
|
|
* could be derived (such as EXPLAIN or DECLARE CURSOR). For such cases,
|
|
|
|
* runtime control will first go through ProcessUtility and then the
|
2012-06-10 21:20:04 +02:00
|
|
|
* executor, and we don't want the executor hooks to do anything, since we
|
|
|
|
* are already measuring the statement's costs at the utility level.
|
2012-03-29 21:32:50 +02:00
|
|
|
*/
|
2012-03-29 03:00:31 +02:00
|
|
|
if (query->utilityStmt)
|
2012-03-29 21:32:50 +02:00
|
|
|
{
|
2017-10-12 01:52:46 +02:00
|
|
|
query->queryId = UINT64CONST(0);
|
2012-03-29 03:00:31 +02:00
|
|
|
return;
|
2012-03-29 21:32:50 +02:00
|
|
|
}
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/* Set up workspace for query jumbling */
|
|
|
|
jstate.jumble = (unsigned char *) palloc(JUMBLE_SIZE);
|
|
|
|
jstate.jumble_len = 0;
|
|
|
|
jstate.clocations_buf_size = 32;
|
|
|
|
jstate.clocations = (pgssLocationLen *)
|
|
|
|
palloc(jstate.clocations_buf_size * sizeof(pgssLocationLen));
|
|
|
|
jstate.clocations_count = 0;
|
2017-03-28 02:14:36 +02:00
|
|
|
jstate.highest_extern_param_id = 0;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/* Compute query ID and mark the Query node with it */
|
|
|
|
JumbleQuery(&jstate, query);
|
2017-10-12 01:52:46 +02:00
|
|
|
query->queryId =
|
|
|
|
DatumGetUInt64(hash_any_extended(jstate.jumble, jstate.jumble_len, 0));
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2012-03-29 21:32:50 +02:00
|
|
|
/*
|
|
|
|
* If we are unlucky enough to get a hash of zero, use 1 instead, to
|
|
|
|
* prevent confusion with the utility-statement case.
|
|
|
|
*/
|
2017-10-12 01:52:46 +02:00
|
|
|
if (query->queryId == UINT64CONST(0))
|
|
|
|
query->queryId = UINT64CONST(1);
|
2012-03-29 21:32:50 +02:00
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/*
|
|
|
|
* If we were able to identify any ignorable constants, we immediately
|
|
|
|
* create a hash table entry for the query, so that we can record the
|
|
|
|
* normalized form of the query string. If there were no such constants,
|
|
|
|
* the normalized string would be the same as the query text anyway, so
|
|
|
|
* there's no need for an early entry.
|
|
|
|
*/
|
|
|
|
if (jstate.clocations_count > 0)
|
|
|
|
pgss_store(pstate->p_sourcetext,
|
|
|
|
query->queryId,
|
2017-01-14 22:17:30 +01:00
|
|
|
query->stmt_location,
|
|
|
|
query->stmt_len,
|
2012-03-29 03:00:31 +02:00
|
|
|
0,
|
|
|
|
0,
|
2012-04-09 17:16:04 +02:00
|
|
|
NULL,
|
2012-03-29 03:00:31 +02:00
|
|
|
&jstate);
|
|
|
|
}
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
|
|
|
* ExecutorStart hook: start up tracking if needed
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_ExecutorStart(QueryDesc *queryDesc, int eflags)
|
|
|
|
{
|
|
|
|
if (prev_ExecutorStart)
|
|
|
|
prev_ExecutorStart(queryDesc, eflags);
|
|
|
|
else
|
|
|
|
standard_ExecutorStart(queryDesc, eflags);
|
|
|
|
|
2012-03-29 21:32:50 +02:00
|
|
|
/*
|
|
|
|
* If query has queryId zero, don't track it. This prevents double
|
|
|
|
* counting of optimizable statements that are directly contained in
|
|
|
|
* utility statements.
|
|
|
|
*/
|
2017-10-12 01:52:46 +02:00
|
|
|
if (pgss_enabled() && queryDesc->plannedstmt->queryId != UINT64CONST(0))
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
|
|
|
/*
|
2009-06-11 16:49:15 +02:00
|
|
|
* Set up to track total elapsed time in ExecutorRun. Make sure the
|
|
|
|
* space is allocated in the per-query context so it will go away at
|
|
|
|
* ExecutorEnd.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
if (queryDesc->totaltime == NULL)
|
|
|
|
{
|
|
|
|
MemoryContext oldcxt;
|
|
|
|
|
|
|
|
oldcxt = MemoryContextSwitchTo(queryDesc->estate->es_query_cxt);
|
2009-12-15 05:57:48 +01:00
|
|
|
queryDesc->totaltime = InstrAlloc(1, INSTRUMENT_ALL);
|
2009-01-04 23:19:59 +01:00
|
|
|
MemoryContextSwitchTo(oldcxt);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* ExecutorRun hook: all we need do is track nesting depth
|
|
|
|
*/
|
|
|
|
static void
|
2017-03-23 18:05:48 +01:00
|
|
|
pgss_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, uint64 count,
|
|
|
|
bool execute_once)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
|
|
|
nested_level++;
|
|
|
|
PG_TRY();
|
|
|
|
{
|
|
|
|
if (prev_ExecutorRun)
|
2017-03-23 18:05:48 +01:00
|
|
|
prev_ExecutorRun(queryDesc, direction, count, execute_once);
|
2009-01-04 23:19:59 +01:00
|
|
|
else
|
2017-03-23 18:05:48 +01:00
|
|
|
standard_ExecutorRun(queryDesc, direction, count, execute_once);
|
2009-01-04 23:19:59 +01:00
|
|
|
nested_level--;
|
|
|
|
}
|
|
|
|
PG_CATCH();
|
|
|
|
{
|
|
|
|
nested_level--;
|
|
|
|
PG_RE_THROW();
|
|
|
|
}
|
|
|
|
PG_END_TRY();
|
2011-02-27 19:43:29 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* ExecutorFinish hook: all we need do is track nesting depth
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_ExecutorFinish(QueryDesc *queryDesc)
|
|
|
|
{
|
|
|
|
nested_level++;
|
|
|
|
PG_TRY();
|
|
|
|
{
|
|
|
|
if (prev_ExecutorFinish)
|
|
|
|
prev_ExecutorFinish(queryDesc);
|
|
|
|
else
|
|
|
|
standard_ExecutorFinish(queryDesc);
|
|
|
|
nested_level--;
|
|
|
|
}
|
|
|
|
PG_CATCH();
|
|
|
|
{
|
|
|
|
nested_level--;
|
|
|
|
PG_RE_THROW();
|
|
|
|
}
|
|
|
|
PG_END_TRY();
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* ExecutorEnd hook: store results if needed
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
pgss_ExecutorEnd(QueryDesc *queryDesc)
|
|
|
|
{
|
2017-10-12 01:52:46 +02:00
|
|
|
uint64 queryId = queryDesc->plannedstmt->queryId;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2017-10-12 01:52:46 +02:00
|
|
|
if (queryId != UINT64CONST(0) && queryDesc->totaltime && pgss_enabled())
|
2012-03-29 21:32:50 +02:00
|
|
|
{
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
2009-06-11 16:49:15 +02:00
|
|
|
* Make sure stats accumulation is done. (Note: it's okay if several
|
|
|
|
* levels of hook all do this.)
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
InstrEndLoop(queryDesc->totaltime);
|
|
|
|
|
|
|
|
pgss_store(queryDesc->sourceText,
|
2012-03-29 03:00:31 +02:00
|
|
|
queryId,
|
2017-01-14 22:17:30 +01:00
|
|
|
queryDesc->plannedstmt->stmt_location,
|
|
|
|
queryDesc->plannedstmt->stmt_len,
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
queryDesc->totaltime->total * 1000.0, /* convert to msec */
|
2010-01-08 01:38:20 +01:00
|
|
|
queryDesc->estate->es_processed,
|
2012-03-29 03:00:31 +02:00
|
|
|
&queryDesc->totaltime->bufusage,
|
|
|
|
NULL);
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
if (prev_ExecutorEnd)
|
|
|
|
prev_ExecutorEnd(queryDesc);
|
|
|
|
else
|
|
|
|
standard_ExecutorEnd(queryDesc);
|
|
|
|
}
|
|
|
|
|
2009-12-15 21:04:49 +01:00
|
|
|
/*
|
|
|
|
* ProcessUtility hook
|
|
|
|
*/
|
|
|
|
static void
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
pgss_ProcessUtility(PlannedStmt *pstmt, const char *queryString,
|
2017-04-01 06:17:18 +02:00
|
|
|
ProcessUtilityContext context,
|
|
|
|
ParamListInfo params, QueryEnvironment *queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
DestReceiver *dest, char *completionTag)
|
2009-12-15 21:04:49 +01:00
|
|
|
{
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
Node *parsetree = pstmt->utilityStmt;
|
|
|
|
|
2012-03-29 22:42:09 +02:00
|
|
|
/*
|
2012-06-10 21:20:04 +02:00
|
|
|
* If it's an EXECUTE statement, we don't track it and don't increment the
|
|
|
|
* nesting level. This allows the cycles to be charged to the underlying
|
|
|
|
* PREPARE instead (by the Executor hooks), which is much more useful.
|
2012-03-29 22:42:09 +02:00
|
|
|
*
|
|
|
|
* We also don't track execution of PREPARE. If we did, we would get one
|
|
|
|
* hash table entry for the PREPARE (with hash calculated from the query
|
|
|
|
* string), and then a different one with the same query string (but hash
|
|
|
|
* calculated from the query tree) would be used to accumulate costs of
|
|
|
|
* ensuing EXECUTEs. This would be confusing, and inconsistent with other
|
|
|
|
* cases where planning time is not included at all.
|
2014-08-25 18:13:24 +02:00
|
|
|
*
|
|
|
|
* Likewise, we don't track execution of DEALLOCATE.
|
2012-03-29 22:42:09 +02:00
|
|
|
*/
|
|
|
|
if (pgss_track_utility && pgss_enabled() &&
|
|
|
|
!IsA(parsetree, ExecuteStmt) &&
|
2014-08-25 18:13:24 +02:00
|
|
|
!IsA(parsetree, PrepareStmt) &&
|
|
|
|
!IsA(parsetree, DeallocateStmt))
|
2009-12-15 21:04:49 +01:00
|
|
|
{
|
|
|
|
instr_time start;
|
|
|
|
instr_time duration;
|
2014-05-27 04:23:29 +02:00
|
|
|
uint64 rows;
|
2012-03-29 03:00:31 +02:00
|
|
|
BufferUsage bufusage_start,
|
|
|
|
bufusage;
|
2009-12-15 21:04:49 +01:00
|
|
|
|
2012-03-27 21:17:22 +02:00
|
|
|
bufusage_start = pgBufferUsage;
|
2009-12-15 21:04:49 +01:00
|
|
|
INSTR_TIME_SET_CURRENT(start);
|
|
|
|
|
|
|
|
nested_level++;
|
|
|
|
PG_TRY();
|
|
|
|
{
|
|
|
|
if (prev_ProcessUtility)
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
prev_ProcessUtility(pstmt, queryString,
|
2017-04-01 06:17:18 +02:00
|
|
|
context, params, queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
dest, completionTag);
|
2009-12-15 21:04:49 +01:00
|
|
|
else
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
standard_ProcessUtility(pstmt, queryString,
|
2017-04-01 06:17:18 +02:00
|
|
|
context, params, queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
dest, completionTag);
|
2009-12-15 21:04:49 +01:00
|
|
|
nested_level--;
|
|
|
|
}
|
|
|
|
PG_CATCH();
|
|
|
|
{
|
|
|
|
nested_level--;
|
|
|
|
PG_RE_THROW();
|
|
|
|
}
|
|
|
|
PG_END_TRY();
|
|
|
|
|
|
|
|
INSTR_TIME_SET_CURRENT(duration);
|
|
|
|
INSTR_TIME_SUBTRACT(duration, start);
|
|
|
|
|
|
|
|
/* parse command tag to retrieve the number of affected rows. */
|
|
|
|
if (completionTag &&
|
2014-05-27 04:23:29 +02:00
|
|
|
strncmp(completionTag, "COPY ", 5) == 0)
|
Widen query numbers-of-tuples-processed counters to uint64.
This patch widens SPI_processed, EState's es_processed field, PortalData's
portalPos field, FuncCallContext's call_cntr and max_calls fields,
ExecutorRun's count argument, PortalRunFetch's result, and the max number
of rows in a SPITupleTable to uint64, and deals with (I hope) all the
ensuing fallout. Some of these values were declared uint32 before, and
others "long".
I also removed PortalData's posOverflow field, since that logic seems
pretty useless given that portalPos is now always 64 bits.
The user-visible results are that command tags for SELECT etc will
correctly report tuple counts larger than 4G, as will plpgsql's GET
GET DIAGNOSTICS ... ROW_COUNT command. Queries processing more tuples
than that are still not exactly the norm, but they're becoming more
common.
Most values associated with FETCH/MOVE distances, such as PortalRun's count
argument and the count argument of most SPI functions that have one, remain
declared as "long". It's not clear whether it would be worth promoting
those to int64; but it would definitely be a large dollop of additional
API churn on top of this, and it would only help 32-bit platforms which
seem relatively less likely to see any benefit.
Andreas Scherbaum, reviewed by Christian Ullrich, additional hacking by me
2016-03-12 22:05:10 +01:00
|
|
|
rows = pg_strtouint64(completionTag + 5, NULL, 10);
|
2014-05-27 04:23:29 +02:00
|
|
|
else
|
2009-12-15 21:04:49 +01:00
|
|
|
rows = 0;
|
|
|
|
|
2010-01-08 01:38:20 +01:00
|
|
|
/* calc differences of buffer counters. */
|
|
|
|
bufusage.shared_blks_hit =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.shared_blks_hit - bufusage_start.shared_blks_hit;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.shared_blks_read =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.shared_blks_read - bufusage_start.shared_blks_read;
|
2012-02-23 02:33:05 +01:00
|
|
|
bufusage.shared_blks_dirtied =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.shared_blks_dirtied - bufusage_start.shared_blks_dirtied;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.shared_blks_written =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.shared_blks_written - bufusage_start.shared_blks_written;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.local_blks_hit =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.local_blks_hit - bufusage_start.local_blks_hit;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.local_blks_read =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.local_blks_read - bufusage_start.local_blks_read;
|
2012-02-23 02:33:05 +01:00
|
|
|
bufusage.local_blks_dirtied =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.local_blks_dirtied - bufusage_start.local_blks_dirtied;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.local_blks_written =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.local_blks_written - bufusage_start.local_blks_written;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.temp_blks_read =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.temp_blks_read - bufusage_start.temp_blks_read;
|
2010-01-08 01:38:20 +01:00
|
|
|
bufusage.temp_blks_written =
|
2012-03-27 21:17:22 +02:00
|
|
|
pgBufferUsage.temp_blks_written - bufusage_start.temp_blks_written;
|
2012-04-30 00:13:33 +02:00
|
|
|
bufusage.blk_read_time = pgBufferUsage.blk_read_time;
|
|
|
|
INSTR_TIME_SUBTRACT(bufusage.blk_read_time, bufusage_start.blk_read_time);
|
|
|
|
bufusage.blk_write_time = pgBufferUsage.blk_write_time;
|
|
|
|
INSTR_TIME_SUBTRACT(bufusage.blk_write_time, bufusage_start.blk_write_time);
|
2010-01-08 01:38:20 +01:00
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
pgss_store(queryString,
|
2017-01-14 22:17:30 +01:00
|
|
|
0, /* signal that it's a utility stmt */
|
|
|
|
pstmt->stmt_location,
|
|
|
|
pstmt->stmt_len,
|
2012-04-28 22:03:57 +02:00
|
|
|
INSTR_TIME_GET_MILLISEC(duration),
|
2012-03-29 03:00:31 +02:00
|
|
|
rows,
|
|
|
|
&bufusage,
|
|
|
|
NULL);
|
2009-12-15 21:04:49 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
if (prev_ProcessUtility)
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
prev_ProcessUtility(pstmt, queryString,
|
2017-04-01 06:17:18 +02:00
|
|
|
context, params, queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
dest, completionTag);
|
2009-12-15 21:04:49 +01:00
|
|
|
else
|
Change representation of statement lists, and add statement location info.
This patch makes several changes that improve the consistency of
representation of lists of statements. It's always been the case
that the output of parse analysis is a list of Query nodes, whatever
the types of the individual statements in the list. This patch brings
similar consistency to the outputs of raw parsing and planning steps:
* The output of raw parsing is now always a list of RawStmt nodes;
the statement-type-dependent nodes are one level down from that.
* The output of pg_plan_queries() is now always a list of PlannedStmt
nodes, even for utility statements. In the case of a utility statement,
"planning" just consists of wrapping a CMD_UTILITY PlannedStmt around
the utility node. This list representation is now used in Portal and
CachedPlan plan lists, replacing the former convention of intermixing
PlannedStmts with bare utility-statement nodes.
Now, every list of statements has a consistent head-node type depending
on how far along it is in processing. This allows changing many places
that formerly used generic "Node *" pointers to use a more specific
pointer type, thus reducing the number of IsA() tests and casts needed,
as well as improving code clarity.
Also, the post-parse-analysis representation of DECLARE CURSOR is changed
so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained
SELECT remains a child of the DeclareCursorStmt rather than getting flipped
around to be the other way. It's now true for both Query and PlannedStmt
that utilityStmt is non-null if and only if commandType is CMD_UTILITY.
That allows simplifying a lot of places that were testing both fields.
(I think some of those were just defensive programming, but in many places,
it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.)
Because PlannedStmt carries a canSetTag field, we're also able to get rid
of some ad-hoc rules about how to reconstruct canSetTag for a bare utility
statement; specifically, the assumption that a utility is canSetTag if and
only if it's the only one in its list. While I see no near-term need for
relaxing that restriction, it's nice to get rid of the ad-hocery.
The API of ProcessUtility() is changed so that what it's passed is the
wrapper PlannedStmt not just the bare utility statement. This will affect
all users of ProcessUtility_hook, but the changes are pretty trivial; see
the affected contrib modules for examples of the minimum change needed.
(Most compilers should give pointer-type-mismatch warnings for uncorrected
code.)
There's also a change in the API of ExplainOneQuery_hook, to pass through
cursorOptions instead of expecting hook functions to know what to pick.
This is needed because of the DECLARE CURSOR changes, but really should
have been done in 9.6; it's unlikely that any extant hook functions
know about using CURSOR_OPT_PARALLEL_OK.
Finally, teach gram.y to save statement boundary locations in RawStmt
nodes, and pass those through to Query and PlannedStmt nodes. This allows
more intelligent handling of cases where a source query string contains
multiple statements. This patch doesn't actually do anything with the
information, but a follow-on patch will. (Passing this information through
cleanly is the true motivation for these changes; while I think this is all
good cleanup, it's unlikely we'd have bothered without this end goal.)
catversion bump because addition of location fields to struct Query
affects stored rules.
This patch is by me, but it owes a good deal to Fabien Coelho who did
a lot of preliminary work on the problem, and also reviewed the patch.
Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre
2017-01-14 22:02:35 +01:00
|
|
|
standard_ProcessUtility(pstmt, queryString,
|
2017-04-01 06:17:18 +02:00
|
|
|
context, params, queryEnv,
|
2013-04-28 06:18:45 +02:00
|
|
|
dest, completionTag);
|
2009-12-15 21:04:49 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/*
|
|
|
|
* Given an arbitrarily long query string, produce a hash for the purposes of
|
|
|
|
* identifying the query, without normalizing constants. Used when hashing
|
2012-04-08 21:49:47 +02:00
|
|
|
* utility statements.
|
2012-03-29 03:00:31 +02:00
|
|
|
*/
|
2017-10-12 01:52:46 +02:00
|
|
|
static uint64
|
2017-01-14 22:17:30 +01:00
|
|
|
pgss_hash_string(const char *str, int len)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
2017-10-12 01:52:46 +02:00
|
|
|
return DatumGetUInt64(hash_any_extended((const unsigned char *) str,
|
|
|
|
len, 0));
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
|
|
|
* Store some statistics for a statement.
|
2012-03-29 03:00:31 +02:00
|
|
|
*
|
2017-01-14 22:17:30 +01:00
|
|
|
* If queryId is 0 then this is a utility statement and we should compute
|
|
|
|
* a suitable queryId internally.
|
|
|
|
*
|
2012-03-29 03:00:31 +02:00
|
|
|
* If jstate is not NULL then we're trying to create an entry for which
|
|
|
|
* we have no statistics as yet; we just want to record the normalized
|
2012-04-09 17:16:04 +02:00
|
|
|
* query string. total_time, rows, bufusage are ignored in this case.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
static void
|
2017-10-12 01:52:46 +02:00
|
|
|
pgss_store(const char *query, uint64 queryId,
|
2017-01-14 22:17:30 +01:00
|
|
|
int query_location, int query_len,
|
2012-03-29 03:00:31 +02:00
|
|
|
double total_time, uint64 rows,
|
|
|
|
const BufferUsage *bufusage,
|
2012-06-10 21:20:04 +02:00
|
|
|
pgssJumbleState *jstate)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
pgssHashKey key;
|
|
|
|
pgssEntry *entry;
|
2012-03-29 03:00:31 +02:00
|
|
|
char *norm_query = NULL;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
int encoding = GetDatabaseEncoding();
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
Assert(query != NULL);
|
|
|
|
|
|
|
|
/* Safety check... */
|
|
|
|
if (!pgss || !pgss_hash)
|
|
|
|
return;
|
|
|
|
|
2017-01-14 22:17:30 +01:00
|
|
|
/*
|
|
|
|
* Confine our attention to the relevant part of the string, if the query
|
|
|
|
* is a portion of a multi-statement source string.
|
|
|
|
*
|
|
|
|
* First apply starting offset, unless it's -1 (unknown).
|
|
|
|
*/
|
|
|
|
if (query_location >= 0)
|
|
|
|
{
|
|
|
|
Assert(query_location <= strlen(query));
|
|
|
|
query += query_location;
|
|
|
|
/* Length of 0 (or -1) means "rest of string" */
|
|
|
|
if (query_len <= 0)
|
|
|
|
query_len = strlen(query);
|
|
|
|
else
|
|
|
|
Assert(query_len <= strlen(query));
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* If query location is unknown, distrust query_len as well */
|
|
|
|
query_location = 0;
|
|
|
|
query_len = strlen(query);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Discard leading and trailing whitespace, too. Use scanner_isspace()
|
|
|
|
* not libc's isspace(), because we want to match the lexer's behavior.
|
|
|
|
*/
|
|
|
|
while (query_len > 0 && scanner_isspace(query[0]))
|
|
|
|
query++, query_location++, query_len--;
|
|
|
|
while (query_len > 0 && scanner_isspace(query[query_len - 1]))
|
|
|
|
query_len--;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For utility statements, we just hash the query string to get an ID.
|
|
|
|
*/
|
2017-10-12 01:52:46 +02:00
|
|
|
if (queryId == UINT64CONST(0))
|
2017-01-14 22:17:30 +01:00
|
|
|
queryId = pgss_hash_string(query, query_len);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/* Set up key for hashtable search */
|
|
|
|
key.userid = GetUserId();
|
|
|
|
key.dbid = MyDatabaseId;
|
2012-03-29 03:00:31 +02:00
|
|
|
key.queryid = queryId;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* Lookup the hash table entry with shared lock. */
|
|
|
|
LWLockAcquire(pgss->lock, LW_SHARED);
|
|
|
|
|
|
|
|
entry = (pgssEntry *) hash_search(pgss_hash, &key, HASH_FIND, NULL);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2012-04-09 17:16:04 +02:00
|
|
|
/* Create new entry, if not present */
|
2009-01-04 23:19:59 +01:00
|
|
|
if (!entry)
|
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Size query_offset;
|
|
|
|
int gc_count;
|
|
|
|
bool stored;
|
|
|
|
bool do_gc;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Create a new, normalized query string if caller asked. We don't
|
|
|
|
* need to hold the lock while doing this work. (Note: in any case,
|
|
|
|
* it's possible that someone else creates a duplicate hashtable entry
|
|
|
|
* in the interval where we don't hold the lock below. That case is
|
|
|
|
* handled by entry_alloc.)
|
2012-03-29 03:00:31 +02:00
|
|
|
*/
|
|
|
|
if (jstate)
|
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
LWLockRelease(pgss->lock);
|
2012-03-29 03:00:31 +02:00
|
|
|
norm_query = generate_normalized_query(jstate, query,
|
2017-01-14 22:17:30 +01:00
|
|
|
query_location,
|
2012-03-29 03:00:31 +02:00
|
|
|
&query_len,
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
encoding);
|
|
|
|
LWLockAcquire(pgss->lock, LW_SHARED);
|
|
|
|
}
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Append new query text to file with only shared lock held */
|
|
|
|
stored = qtext_store(norm_query ? norm_query : query, query_len,
|
|
|
|
&query_offset, &gc_count);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* Determine whether we need to garbage collect external query texts
|
|
|
|
* while the shared lock is still held. This micro-optimization
|
|
|
|
* avoids taking the time to decide this while holding exclusive lock.
|
|
|
|
*/
|
|
|
|
do_gc = need_gc_qtexts();
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Need exclusive lock to make a new hashtable entry - promote */
|
|
|
|
LWLockRelease(pgss->lock);
|
|
|
|
LWLockAcquire(pgss->lock, LW_EXCLUSIVE);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* A garbage collection may have occurred while we weren't holding the
|
|
|
|
* lock. In the unlikely event that this happens, the query text we
|
|
|
|
* stored above will have been garbage collected, so write it again.
|
|
|
|
* This should be infrequent enough that doing it while holding
|
|
|
|
* exclusive lock isn't a performance problem.
|
|
|
|
*/
|
|
|
|
if (!stored || pgss->gc_count != gc_count)
|
|
|
|
stored = qtext_store(norm_query ? norm_query : query, query_len,
|
|
|
|
&query_offset, NULL);
|
|
|
|
|
|
|
|
/* If we failed to write to the text file, give up */
|
|
|
|
if (!stored)
|
|
|
|
goto done;
|
|
|
|
|
|
|
|
/* OK to create a new hashtable entry */
|
|
|
|
entry = entry_alloc(&key, query_offset, query_len, encoding,
|
|
|
|
jstate != NULL);
|
|
|
|
|
|
|
|
/* If needed, perform garbage collection while exclusive lock held */
|
|
|
|
if (do_gc)
|
|
|
|
gc_qtexts();
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
2012-04-09 17:16:04 +02:00
|
|
|
/* Increment the counts, except when jstate is not NULL */
|
|
|
|
if (!jstate)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
2012-04-09 17:16:04 +02:00
|
|
|
/*
|
|
|
|
* Grab the spinlock while updating the counters (see comment about
|
|
|
|
* locking rules at the head of the file)
|
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
volatile pgssEntry *e = (volatile pgssEntry *) entry;
|
|
|
|
|
|
|
|
SpinLockAcquire(&e->mutex);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2012-04-09 17:16:04 +02:00
|
|
|
/* "Unstick" entry if it was previously sticky */
|
|
|
|
if (e->counters.calls == 0)
|
|
|
|
e->counters.usage = USAGE_INIT;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2012-04-09 17:16:04 +02:00
|
|
|
e->counters.calls += 1;
|
2009-12-15 21:04:49 +01:00
|
|
|
e->counters.total_time += total_time;
|
2015-03-27 20:43:22 +01:00
|
|
|
if (e->counters.calls == 1)
|
|
|
|
{
|
|
|
|
e->counters.min_time = total_time;
|
|
|
|
e->counters.max_time = total_time;
|
|
|
|
e->counters.mean_time = total_time;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
2015-05-24 03:35:49 +02:00
|
|
|
* Welford's method for accurately computing variance. See
|
|
|
|
* <http://www.johndcook.com/blog/standard_deviation/>
|
2015-03-27 20:43:22 +01:00
|
|
|
*/
|
2015-05-24 03:35:49 +02:00
|
|
|
double old_mean = e->counters.mean_time;
|
2015-03-27 20:43:22 +01:00
|
|
|
|
|
|
|
e->counters.mean_time +=
|
|
|
|
(total_time - old_mean) / e->counters.calls;
|
|
|
|
e->counters.sum_var_time +=
|
|
|
|
(total_time - old_mean) * (total_time - e->counters.mean_time);
|
|
|
|
|
|
|
|
/* calculate min and max time */
|
|
|
|
if (e->counters.min_time > total_time)
|
|
|
|
e->counters.min_time = total_time;
|
|
|
|
if (e->counters.max_time < total_time)
|
|
|
|
e->counters.max_time = total_time;
|
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
e->counters.rows += rows;
|
2010-01-08 01:38:20 +01:00
|
|
|
e->counters.shared_blks_hit += bufusage->shared_blks_hit;
|
|
|
|
e->counters.shared_blks_read += bufusage->shared_blks_read;
|
2012-02-23 02:33:05 +01:00
|
|
|
e->counters.shared_blks_dirtied += bufusage->shared_blks_dirtied;
|
2010-01-08 01:38:20 +01:00
|
|
|
e->counters.shared_blks_written += bufusage->shared_blks_written;
|
|
|
|
e->counters.local_blks_hit += bufusage->local_blks_hit;
|
|
|
|
e->counters.local_blks_read += bufusage->local_blks_read;
|
2012-02-23 02:33:05 +01:00
|
|
|
e->counters.local_blks_dirtied += bufusage->local_blks_dirtied;
|
2010-01-08 01:38:20 +01:00
|
|
|
e->counters.local_blks_written += bufusage->local_blks_written;
|
|
|
|
e->counters.temp_blks_read += bufusage->temp_blks_read;
|
|
|
|
e->counters.temp_blks_written += bufusage->temp_blks_written;
|
2012-04-30 00:13:33 +02:00
|
|
|
e->counters.blk_read_time += INSTR_TIME_GET_MILLISEC(bufusage->blk_read_time);
|
|
|
|
e->counters.blk_write_time += INSTR_TIME_GET_MILLISEC(bufusage->blk_write_time);
|
2012-04-28 22:03:57 +02:00
|
|
|
e->counters.usage += USAGE_EXEC(total_time);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
SpinLockRelease(&e->mutex);
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
done:
|
2009-01-04 23:19:59 +01:00
|
|
|
LWLockRelease(pgss->lock);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* We postpone this clean-up until we're out of the lock */
|
2012-03-29 03:00:31 +02:00
|
|
|
if (norm_query)
|
|
|
|
pfree(norm_query);
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reset all statement statistics.
|
|
|
|
*/
|
|
|
|
Datum
|
|
|
|
pg_stat_statements_reset(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
if (!pgss || !pgss_hash)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("pg_stat_statements must be loaded via shared_preload_libraries")));
|
|
|
|
entry_reset();
|
|
|
|
PG_RETURN_VOID();
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Number of output arguments (columns) for various API versions */
|
2012-02-23 02:33:05 +01:00
|
|
|
#define PG_STAT_STATEMENTS_COLS_V1_0 14
|
2013-12-07 18:06:02 +01:00
|
|
|
#define PG_STAT_STATEMENTS_COLS_V1_1 18
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
#define PG_STAT_STATEMENTS_COLS_V1_2 19
|
2015-03-27 20:43:22 +01:00
|
|
|
#define PG_STAT_STATEMENTS_COLS_V1_3 23
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
#define PG_STAT_STATEMENTS_COLS 23 /* maximum of above */
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Retrieve statement statistics.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*
|
|
|
|
* The SQL API of this function has changed multiple times, and will likely
|
|
|
|
* do so again in future. To support the case where a newer version of this
|
|
|
|
* loadable module is being used with an old SQL declaration of the function,
|
|
|
|
* we continue to support the older API versions. For 1.2 and later, the
|
|
|
|
* expected API version is identified by embedding it in the C name of the
|
|
|
|
* function. Unfortunately we weren't bright enough to do that for 1.1.
|
|
|
|
*/
|
2015-03-27 20:43:22 +01:00
|
|
|
Datum
|
|
|
|
pg_stat_statements_1_3(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
bool showtext = PG_GETARG_BOOL(0);
|
|
|
|
|
|
|
|
pg_stat_statements_internal(fcinfo, PGSS_V1_3, showtext);
|
|
|
|
|
|
|
|
return (Datum) 0;
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Datum
|
|
|
|
pg_stat_statements_1_2(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
bool showtext = PG_GETARG_BOOL(0);
|
|
|
|
|
|
|
|
pg_stat_statements_internal(fcinfo, PGSS_V1_2, showtext);
|
|
|
|
|
|
|
|
return (Datum) 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Legacy entry point for pg_stat_statements() API versions 1.0 and 1.1.
|
|
|
|
* This can be removed someday, perhaps.
|
2009-01-04 23:19:59 +01:00
|
|
|
*/
|
|
|
|
Datum
|
|
|
|
pg_stat_statements(PG_FUNCTION_ARGS)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
{
|
|
|
|
/* If it's really API 1.1, we'll figure that out below */
|
|
|
|
pg_stat_statements_internal(fcinfo, PGSS_V1_0, true);
|
|
|
|
|
|
|
|
return (Datum) 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Common code for all versions of pg_stat_statements() */
|
|
|
|
static void
|
|
|
|
pg_stat_statements_internal(FunctionCallInfo fcinfo,
|
|
|
|
pgssVersion api_version,
|
|
|
|
bool showtext)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
|
|
|
|
TupleDesc tupdesc;
|
|
|
|
Tuplestorestate *tupstore;
|
|
|
|
MemoryContext per_query_ctx;
|
|
|
|
MemoryContext oldcontext;
|
|
|
|
Oid userid = GetUserId();
|
2017-03-30 20:18:53 +02:00
|
|
|
bool is_allowed_role = false;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
char *qbuffer = NULL;
|
|
|
|
Size qbuffer_size = 0;
|
|
|
|
Size extent = 0;
|
|
|
|
int gc_count = 0;
|
2009-06-11 16:49:15 +02:00
|
|
|
HASH_SEQ_STATUS hash_seq;
|
|
|
|
pgssEntry *entry;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
2017-03-30 20:18:53 +02:00
|
|
|
/* Superusers or members of pg_read_all_stats members are allowed */
|
|
|
|
is_allowed_role = is_member_of_role(GetUserId(), DEFAULT_ROLE_READ_ALL_STATS);
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* hash table must exist already */
|
2009-01-04 23:19:59 +01:00
|
|
|
if (!pgss || !pgss_hash)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
|
|
|
|
errmsg("pg_stat_statements must be loaded via shared_preload_libraries")));
|
|
|
|
|
|
|
|
/* check to see if caller supports us returning a tuplestore */
|
|
|
|
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
|
|
|
|
errmsg("set-valued function called in context that cannot accept a set")));
|
|
|
|
if (!(rsinfo->allowedModes & SFRM_Materialize))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
|
|
|
|
errmsg("materialize mode required, but it is not " \
|
|
|
|
"allowed in this context")));
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Switch into long-lived context to construct returned data structures */
|
|
|
|
per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
|
|
|
|
oldcontext = MemoryContextSwitchTo(per_query_ctx);
|
|
|
|
|
2010-01-08 01:38:20 +01:00
|
|
|
/* Build a tuple descriptor for our result type */
|
|
|
|
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
|
|
|
|
elog(ERROR, "return type must be a row type");
|
2013-12-07 18:06:02 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* Check we have the expected number of output arguments. Aside from
|
|
|
|
* being a good safety check, we need a kluge here to detect API version
|
|
|
|
* 1.1, which was wedged into the code in an ill-considered way.
|
|
|
|
*/
|
2013-12-07 18:06:02 +01:00
|
|
|
switch (tupdesc->natts)
|
|
|
|
{
|
|
|
|
case PG_STAT_STATEMENTS_COLS_V1_0:
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (api_version != PGSS_V1_0)
|
|
|
|
elog(ERROR, "incorrect number of output arguments");
|
2013-12-07 18:06:02 +01:00
|
|
|
break;
|
|
|
|
case PG_STAT_STATEMENTS_COLS_V1_1:
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* pg_stat_statements() should have told us 1.0 */
|
|
|
|
if (api_version != PGSS_V1_0)
|
|
|
|
elog(ERROR, "incorrect number of output arguments");
|
|
|
|
api_version = PGSS_V1_1;
|
2013-12-07 18:06:02 +01:00
|
|
|
break;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
case PG_STAT_STATEMENTS_COLS_V1_2:
|
|
|
|
if (api_version != PGSS_V1_2)
|
|
|
|
elog(ERROR, "incorrect number of output arguments");
|
2013-12-07 18:06:02 +01:00
|
|
|
break;
|
2015-03-27 20:43:22 +01:00
|
|
|
case PG_STAT_STATEMENTS_COLS_V1_3:
|
|
|
|
if (api_version != PGSS_V1_3)
|
|
|
|
elog(ERROR, "incorrect number of output arguments");
|
|
|
|
break;
|
2013-12-07 18:06:02 +01:00
|
|
|
default:
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
elog(ERROR, "incorrect number of output arguments");
|
2013-12-07 18:06:02 +01:00
|
|
|
}
|
2010-01-08 01:38:20 +01:00
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
tupstore = tuplestore_begin_heap(true, false, work_mem);
|
|
|
|
rsinfo->returnMode = SFRM_Materialize;
|
|
|
|
rsinfo->setResult = tupstore;
|
|
|
|
rsinfo->setDesc = tupdesc;
|
|
|
|
|
2009-12-29 18:40:59 +01:00
|
|
|
MemoryContextSwitchTo(oldcontext);
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* We'd like to load the query text file (if needed) while not holding any
|
2014-05-06 18:12:18 +02:00
|
|
|
* lock on pgss->lock. In the worst case we'll have to do this again
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* after we have the lock, but it's unlikely enough to make this a win
|
2014-05-06 18:12:18 +02:00
|
|
|
* despite occasional duplicated work. We need to reload if anybody
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* writes to the file (either a retail qtext_store(), or a garbage
|
|
|
|
* collection) between this point and where we've gotten shared lock. If
|
|
|
|
* a qtext_store is actually in progress when we look, we might as well
|
|
|
|
* skip the speculative load entirely.
|
|
|
|
*/
|
|
|
|
if (showtext)
|
|
|
|
{
|
|
|
|
int n_writers;
|
|
|
|
|
|
|
|
/* Take the mutex so we can examine variables */
|
|
|
|
{
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss;
|
|
|
|
|
|
|
|
SpinLockAcquire(&s->mutex);
|
|
|
|
extent = s->extent;
|
|
|
|
n_writers = s->n_writers;
|
|
|
|
gc_count = s->gc_count;
|
|
|
|
SpinLockRelease(&s->mutex);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No point in loading file now if there are active writers */
|
|
|
|
if (n_writers == 0)
|
|
|
|
qbuffer = qtext_load_file(&qbuffer_size);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get shared lock, load or reload the query text file if we must, and
|
|
|
|
* iterate over the hashtable entries.
|
|
|
|
*
|
|
|
|
* With a large hash table, we might be holding the lock rather longer
|
|
|
|
* than one could wish. However, this only blocks creation of new hash
|
|
|
|
* table entries, and the larger the hash table the less likely that is to
|
|
|
|
* be needed. So we can hope this is okay. Perhaps someday we'll decide
|
|
|
|
* we need to partition the hash table to limit the time spent holding any
|
|
|
|
* one lock.
|
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
LWLockAcquire(pgss->lock, LW_SHARED);
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (showtext)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Here it is safe to examine extent and gc_count without taking the
|
|
|
|
* mutex. Note that although other processes might change
|
|
|
|
* pgss->extent just after we look at it, the strings they then write
|
|
|
|
* into the file cannot yet be referenced in the hashtable, so we
|
|
|
|
* don't care whether we see them or not.
|
|
|
|
*
|
|
|
|
* If qtext_load_file fails, we just press on; we'll return NULL for
|
|
|
|
* every query text.
|
|
|
|
*/
|
|
|
|
if (qbuffer == NULL ||
|
|
|
|
pgss->extent != extent ||
|
|
|
|
pgss->gc_count != gc_count)
|
|
|
|
{
|
|
|
|
if (qbuffer)
|
|
|
|
free(qbuffer);
|
|
|
|
qbuffer = qtext_load_file(&qbuffer_size);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
|
|
|
Datum values[PG_STAT_STATEMENTS_COLS];
|
|
|
|
bool nulls[PG_STAT_STATEMENTS_COLS];
|
|
|
|
int i = 0;
|
|
|
|
Counters tmp;
|
2015-03-28 18:56:37 +01:00
|
|
|
double stddev;
|
2013-12-08 11:59:07 +01:00
|
|
|
int64 queryid = entry->key.queryid;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
memset(values, 0, sizeof(values));
|
|
|
|
memset(nulls, 0, sizeof(nulls));
|
|
|
|
|
|
|
|
values[i++] = ObjectIdGetDatum(entry->key.userid);
|
|
|
|
values[i++] = ObjectIdGetDatum(entry->key.dbid);
|
|
|
|
|
2017-03-30 20:18:53 +02:00
|
|
|
if (is_allowed_role || entry->key.userid == userid)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (api_version >= PGSS_V1_2)
|
2013-12-08 11:59:07 +01:00
|
|
|
values[i++] = Int64GetDatumFast(queryid);
|
2013-12-07 18:06:02 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (showtext)
|
|
|
|
{
|
|
|
|
char *qstr = qtext_fetch(entry->query_offset,
|
|
|
|
entry->query_len,
|
|
|
|
qbuffer,
|
|
|
|
qbuffer_size);
|
|
|
|
|
|
|
|
if (qstr)
|
|
|
|
{
|
|
|
|
char *enc;
|
|
|
|
|
2014-02-23 22:59:05 +01:00
|
|
|
enc = pg_any_to_server(qstr,
|
|
|
|
entry->query_len,
|
|
|
|
entry->encoding);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
values[i++] = CStringGetTextDatum(enc);
|
|
|
|
|
|
|
|
if (enc != qstr)
|
|
|
|
pfree(enc);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Just return a null if we fail to find the text */
|
|
|
|
nulls[i++] = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Query text not requested */
|
|
|
|
nulls[i++] = true;
|
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
else
|
2013-12-07 18:06:02 +01:00
|
|
|
{
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Don't show queryid */
|
|
|
|
if (api_version >= PGSS_V1_2)
|
2013-12-07 18:06:02 +01:00
|
|
|
nulls[i++] = true;
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* Don't show query text, but hint as to the reason for not doing
|
|
|
|
* so if it was requested
|
|
|
|
*/
|
|
|
|
if (showtext)
|
|
|
|
values[i++] = CStringGetTextDatum("<insufficient privilege>");
|
|
|
|
else
|
|
|
|
nulls[i++] = true;
|
2013-12-07 18:06:02 +01:00
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* copy counters to a local variable to keep locking time short */
|
|
|
|
{
|
|
|
|
volatile pgssEntry *e = (volatile pgssEntry *) entry;
|
|
|
|
|
|
|
|
SpinLockAcquire(&e->mutex);
|
|
|
|
tmp = e->counters;
|
|
|
|
SpinLockRelease(&e->mutex);
|
|
|
|
}
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/* Skip entry if unexecuted (ie, it's a pending "sticky" entry) */
|
|
|
|
if (tmp.calls == 0)
|
|
|
|
continue;
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.calls);
|
|
|
|
values[i++] = Float8GetDatumFast(tmp.total_time);
|
2015-03-27 20:43:22 +01:00
|
|
|
if (api_version >= PGSS_V1_3)
|
|
|
|
{
|
|
|
|
values[i++] = Float8GetDatumFast(tmp.min_time);
|
|
|
|
values[i++] = Float8GetDatumFast(tmp.max_time);
|
|
|
|
values[i++] = Float8GetDatumFast(tmp.mean_time);
|
2015-05-24 03:35:49 +02:00
|
|
|
|
2015-03-27 20:43:22 +01:00
|
|
|
/*
|
|
|
|
* Note we are calculating the population variance here, not the
|
2015-05-24 03:35:49 +02:00
|
|
|
* sample variance, as we have data for the whole population, so
|
|
|
|
* Bessel's correction is not used, and we don't divide by
|
2015-03-27 20:43:22 +01:00
|
|
|
* tmp.calls - 1.
|
|
|
|
*/
|
|
|
|
if (tmp.calls > 1)
|
2015-03-28 18:56:37 +01:00
|
|
|
stddev = sqrt(tmp.sum_var_time / tmp.calls);
|
2015-03-27 20:43:22 +01:00
|
|
|
else
|
2015-03-28 18:56:37 +01:00
|
|
|
stddev = 0.0;
|
|
|
|
values[i++] = Float8GetDatumFast(stddev);
|
2015-03-27 20:43:22 +01:00
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.rows);
|
2010-01-08 01:38:20 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.shared_blks_hit);
|
|
|
|
values[i++] = Int64GetDatumFast(tmp.shared_blks_read);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (api_version >= PGSS_V1_1)
|
2012-02-23 02:33:05 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.shared_blks_dirtied);
|
2010-01-08 01:38:20 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.shared_blks_written);
|
|
|
|
values[i++] = Int64GetDatumFast(tmp.local_blks_hit);
|
|
|
|
values[i++] = Int64GetDatumFast(tmp.local_blks_read);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (api_version >= PGSS_V1_1)
|
2012-02-23 02:33:05 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.local_blks_dirtied);
|
2010-01-08 01:38:20 +01:00
|
|
|
values[i++] = Int64GetDatumFast(tmp.local_blks_written);
|
|
|
|
values[i++] = Int64GetDatumFast(tmp.temp_blks_read);
|
|
|
|
values[i++] = Int64GetDatumFast(tmp.temp_blks_written);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (api_version >= PGSS_V1_1)
|
2012-03-27 21:17:22 +02:00
|
|
|
{
|
2012-04-30 00:13:33 +02:00
|
|
|
values[i++] = Float8GetDatumFast(tmp.blk_read_time);
|
|
|
|
values[i++] = Float8GetDatumFast(tmp.blk_write_time);
|
2012-03-27 21:17:22 +02:00
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
Assert(i == (api_version == PGSS_V1_0 ? PG_STAT_STATEMENTS_COLS_V1_0 :
|
|
|
|
api_version == PGSS_V1_1 ? PG_STAT_STATEMENTS_COLS_V1_1 :
|
|
|
|
api_version == PGSS_V1_2 ? PG_STAT_STATEMENTS_COLS_V1_2 :
|
2015-03-27 20:43:22 +01:00
|
|
|
api_version == PGSS_V1_3 ? PG_STAT_STATEMENTS_COLS_V1_3 :
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
-1 /* fail if you forget to update this assert */ ));
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* clean up and return the tuplestore */
|
2009-01-04 23:19:59 +01:00
|
|
|
LWLockRelease(pgss->lock);
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (qbuffer)
|
|
|
|
free(qbuffer);
|
2009-01-04 23:19:59 +01:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
tuplestore_donestoring(tupstore);
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Estimate shared memory space needed.
|
|
|
|
*/
|
|
|
|
static Size
|
|
|
|
pgss_memsize(void)
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
Size size;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
size = MAXALIGN(sizeof(pgssSharedState));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
size = add_size(size, hash_estimate_size(pgss_max, sizeof(pgssEntry)));
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
return size;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate a new hashtable entry.
|
|
|
|
* caller must hold an exclusive lock on pgss->lock
|
|
|
|
*
|
2012-03-29 03:00:31 +02:00
|
|
|
* "query" need not be null-terminated; we rely on query_len instead
|
|
|
|
*
|
2012-04-09 17:16:04 +02:00
|
|
|
* If "sticky" is true, make the new entry artificially sticky so that it will
|
|
|
|
* probably still be there when the query finishes execution. We do this by
|
|
|
|
* giving it a median usage value rather than the normal value. (Strictly
|
|
|
|
* speaking, query strings are normalized on a best effort basis, though it
|
|
|
|
* would be difficult to demonstrate this even under artificial conditions.)
|
|
|
|
*
|
2009-01-04 23:19:59 +01:00
|
|
|
* Note: despite needing exclusive lock, it's not an error for the target
|
2014-05-06 18:12:18 +02:00
|
|
|
* entry to already exist. This is because pgss_store releases and
|
2009-01-04 23:19:59 +01:00
|
|
|
* reacquires lock after failing to find a match; so someone else could
|
|
|
|
* have made the entry while we waited to get exclusive lock.
|
|
|
|
*/
|
|
|
|
static pgssEntry *
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
entry_alloc(pgssHashKey *key, Size query_offset, int query_len, int encoding,
|
|
|
|
bool sticky)
|
2009-01-04 23:19:59 +01:00
|
|
|
{
|
|
|
|
pgssEntry *entry;
|
2009-06-11 16:49:15 +02:00
|
|
|
bool found;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
/* Make space if needed */
|
|
|
|
while (hash_get_num_entries(pgss_hash) >= pgss_max)
|
|
|
|
entry_dealloc();
|
|
|
|
|
|
|
|
/* Find or create an entry with desired hash code */
|
|
|
|
entry = (pgssEntry *) hash_search(pgss_hash, key, HASH_ENTER, &found);
|
|
|
|
|
|
|
|
if (!found)
|
|
|
|
{
|
|
|
|
/* New entry, initialize it */
|
|
|
|
|
|
|
|
/* reset the statistics */
|
|
|
|
memset(&entry->counters, 0, sizeof(Counters));
|
2012-04-09 17:16:04 +02:00
|
|
|
/* set the appropriate initial usage count */
|
|
|
|
entry->counters.usage = sticky ? pgss->cur_median_usage : USAGE_INIT;
|
2009-01-04 23:19:59 +01:00
|
|
|
/* re-initialize the mutex each time ... we assume no one using it */
|
|
|
|
SpinLockInit(&entry->mutex);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* ... and don't forget the query text metadata */
|
|
|
|
Assert(query_len >= 0);
|
|
|
|
entry->query_offset = query_offset;
|
2012-03-29 03:00:31 +02:00
|
|
|
entry->query_len = query_len;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
entry->encoding = encoding;
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
return entry;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* qsort comparator for sorting into increasing usage order
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
entry_cmp(const void *lhs, const void *rhs)
|
|
|
|
{
|
2012-03-29 03:00:31 +02:00
|
|
|
double l_usage = (*(pgssEntry *const *) lhs)->counters.usage;
|
|
|
|
double r_usage = (*(pgssEntry *const *) rhs)->counters.usage;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
if (l_usage < r_usage)
|
|
|
|
return -1;
|
|
|
|
else if (l_usage > r_usage)
|
|
|
|
return +1;
|
|
|
|
else
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
* Deallocate least-used entries.
|
|
|
|
*
|
2009-01-04 23:19:59 +01:00
|
|
|
* Caller must hold an exclusive lock on pgss->lock.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
entry_dealloc(void)
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
HASH_SEQ_STATUS hash_seq;
|
|
|
|
pgssEntry **entries;
|
|
|
|
pgssEntry *entry;
|
|
|
|
int nvictims;
|
|
|
|
int i;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
Size tottextlen;
|
|
|
|
int nvalidtexts;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
2012-04-08 21:49:47 +02:00
|
|
|
/*
|
|
|
|
* Sort entries by usage and deallocate USAGE_DEALLOC_PERCENT of them.
|
|
|
|
* While we're scanning the table, apply the decay factor to the usage
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
* values, and update the mean query length.
|
|
|
|
*
|
|
|
|
* Note that the mean query length is almost immediately obsolete, since
|
|
|
|
* we compute it before not after discarding the least-used entries.
|
|
|
|
* Hopefully, that doesn't affect the mean too much; it doesn't seem worth
|
|
|
|
* making two passes to get a more current result. Likewise, the new
|
|
|
|
* cur_median_usage includes the entries we're about to zap.
|
2012-04-08 21:49:47 +02:00
|
|
|
*/
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
entries = palloc(hash_get_num_entries(pgss_hash) * sizeof(pgssEntry *));
|
|
|
|
|
|
|
|
i = 0;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
tottextlen = 0;
|
|
|
|
nvalidtexts = 0;
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
|
|
|
entries[i++] = entry;
|
2012-04-08 21:49:47 +02:00
|
|
|
/* "Sticky" entries get a different usage decay rate. */
|
|
|
|
if (entry->counters.calls == 0)
|
|
|
|
entry->counters.usage *= STICKY_DECREASE_FACTOR;
|
|
|
|
else
|
|
|
|
entry->counters.usage *= USAGE_DECREASE_FACTOR;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* In the mean length computation, ignore dropped texts. */
|
|
|
|
if (entry->query_len >= 0)
|
|
|
|
{
|
|
|
|
tottextlen += entry->query_len + 1;
|
|
|
|
nvalidtexts++;
|
|
|
|
}
|
2009-01-04 23:19:59 +01:00
|
|
|
}
|
|
|
|
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* Sort into increasing order by usage */
|
2009-01-04 23:19:59 +01:00
|
|
|
qsort(entries, i, sizeof(pgssEntry *), entry_cmp);
|
2012-04-08 21:49:47 +02:00
|
|
|
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* Record the (approximate) median usage */
|
2012-04-08 21:49:47 +02:00
|
|
|
if (i > 0)
|
|
|
|
pgss->cur_median_usage = entries[i / 2]->counters.usage;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* Record the mean query length */
|
|
|
|
if (nvalidtexts > 0)
|
|
|
|
pgss->mean_query_len = tottextlen / nvalidtexts;
|
|
|
|
else
|
|
|
|
pgss->mean_query_len = ASSUMED_LENGTH_INIT;
|
2012-04-08 21:49:47 +02:00
|
|
|
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* Now zap an appropriate fraction of lowest-usage entries */
|
2009-01-04 23:19:59 +01:00
|
|
|
nvictims = Max(10, i * USAGE_DEALLOC_PERCENT / 100);
|
|
|
|
nvictims = Min(nvictims, i);
|
|
|
|
|
|
|
|
for (i = 0; i < nvictims; i++)
|
|
|
|
{
|
|
|
|
hash_search(pgss_hash, &entries[i]->key, HASH_REMOVE, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
pfree(entries);
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
2017-01-14 22:17:30 +01:00
|
|
|
* Given a query string (not necessarily null-terminated), allocate a new
|
|
|
|
* entry in the external query text file and store the string there.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*
|
|
|
|
* If successful, returns true, and stores the new entry's offset in the file
|
2014-05-06 18:12:18 +02:00
|
|
|
* into *query_offset. Also, if gc_count isn't NULL, *gc_count is set to the
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* number of garbage collections that have occurred so far.
|
|
|
|
*
|
|
|
|
* On failure, returns false.
|
|
|
|
*
|
|
|
|
* At least a shared lock on pgss->lock must be held by the caller, so as
|
2014-05-06 18:12:18 +02:00
|
|
|
* to prevent a concurrent garbage collection. Share-lock-holding callers
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* should pass a gc_count pointer to obtain the number of garbage collections,
|
|
|
|
* so that they can recheck the count after obtaining exclusive lock to
|
|
|
|
* detect whether a garbage collection occurred (and removed this entry).
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
qtext_store(const char *query, int query_len,
|
|
|
|
Size *query_offset, int *gc_count)
|
|
|
|
{
|
|
|
|
Size off;
|
|
|
|
int fd;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We use a spinlock to protect extent/n_writers/gc_count, so that
|
|
|
|
* multiple processes may execute this function concurrently.
|
|
|
|
*/
|
|
|
|
{
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss;
|
|
|
|
|
|
|
|
SpinLockAcquire(&s->mutex);
|
|
|
|
off = s->extent;
|
|
|
|
s->extent += query_len + 1;
|
|
|
|
s->n_writers++;
|
|
|
|
if (gc_count)
|
|
|
|
*gc_count = s->gc_count;
|
|
|
|
SpinLockRelease(&s->mutex);
|
|
|
|
}
|
|
|
|
|
|
|
|
*query_offset = off;
|
|
|
|
|
Be more wary about 32-bit integer overflow in pg_stat_statements.
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files. It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:
1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits. We still need overflow guards on its use, but
this helps.
2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize. If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.
3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms. It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.
Per report from Bruno da Silva. I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place. But it did get that big, so #2 is a reasonable defense,
and #3 could explain the reported performance difficulties.
(See also commit 8bbe4cbd9, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)
This issue is old, and is primarily a problem for old platforms,
so back-patch.
Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
2022-08-03 00:05:34 +02:00
|
|
|
/*
|
|
|
|
* Don't allow the file to grow larger than what qtext_load_file can
|
|
|
|
* (theoretically) handle. This has been seen to be reachable on 32-bit
|
|
|
|
* platforms.
|
|
|
|
*/
|
|
|
|
if (unlikely(query_len >= MaxAllocHugeSize - off))
|
|
|
|
{
|
|
|
|
errno = EFBIG; /* not quite right, but it'll do */
|
|
|
|
fd = -1;
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Now write the data into the successfully-reserved part of the file */
|
2017-09-23 15:49:22 +02:00
|
|
|
fd = OpenTransientFile(PGSS_TEXT_FILE, O_RDWR | O_CREAT | PG_BINARY);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (fd < 0)
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
if (lseek(fd, off, SEEK_SET) != off)
|
|
|
|
goto error;
|
|
|
|
|
2017-01-14 22:17:30 +01:00
|
|
|
if (write(fd, query, query_len) != query_len)
|
|
|
|
goto error;
|
|
|
|
if (write(fd, "\0", 1) != 1)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
goto error;
|
|
|
|
|
|
|
|
CloseTransientFile(fd);
|
|
|
|
|
|
|
|
/* Mark our write complete */
|
|
|
|
{
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss;
|
|
|
|
|
|
|
|
SpinLockAcquire(&s->mutex);
|
|
|
|
s->n_writers--;
|
|
|
|
SpinLockRelease(&s->mutex);
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
|
|
|
error:
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
|
|
|
|
if (fd >= 0)
|
|
|
|
CloseTransientFile(fd);
|
|
|
|
|
|
|
|
/* Mark our write complete */
|
|
|
|
{
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss;
|
|
|
|
|
|
|
|
SpinLockAcquire(&s->mutex);
|
|
|
|
s->n_writers--;
|
|
|
|
SpinLockRelease(&s->mutex);
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Read the external query text file into a malloc'd buffer.
|
|
|
|
*
|
|
|
|
* Returns NULL (without throwing an error) if unable to read, eg
|
|
|
|
* file not there or insufficient memory.
|
|
|
|
*
|
|
|
|
* On success, the buffer size is also returned into *buffer_size.
|
|
|
|
*
|
|
|
|
* This can be called without any lock on pgss->lock, but in that case
|
|
|
|
* the caller is responsible for verifying that the result is sane.
|
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
qtext_load_file(Size *buffer_size)
|
|
|
|
{
|
|
|
|
char *buf;
|
|
|
|
int fd;
|
|
|
|
struct stat stat;
|
2021-11-01 00:13:48 +01:00
|
|
|
Size nread;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
2017-09-23 15:49:22 +02:00
|
|
|
fd = OpenTransientFile(PGSS_TEXT_FILE, O_RDONLY | PG_BINARY);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
if (fd < 0)
|
|
|
|
{
|
|
|
|
if (errno != ENOENT)
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
errmsg("could not read pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Get file length */
|
|
|
|
if (fstat(fd, &stat))
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not stat pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
CloseTransientFile(fd);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Allocate buffer; beware that off_t might be wider than size_t */
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
if (stat.st_size <= MaxAllocHugeSize)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
buf = (char *) malloc(stat.st_size);
|
|
|
|
else
|
|
|
|
buf = NULL;
|
|
|
|
if (buf == NULL)
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode(ERRCODE_OUT_OF_MEMORY),
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
errmsg("out of memory"),
|
|
|
|
errdetail("Could not allocate enough memory to read pg_stat_statement file \"%s\".",
|
|
|
|
PGSS_TEXT_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
CloseTransientFile(fd);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2021-11-01 00:13:48 +01:00
|
|
|
* OK, slurp in the file. Windows fails if we try to read more than
|
|
|
|
* INT_MAX bytes at once, and other platforms might not like that either,
|
|
|
|
* so read a very large file in 1GB segments.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*/
|
2021-11-01 00:13:48 +01:00
|
|
|
nread = 0;
|
|
|
|
while (nread < stat.st_size)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
{
|
2021-11-01 00:13:48 +01:00
|
|
|
int toread = Min(1024 * 1024 * 1024, stat.st_size - nread);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we get a short read and errno doesn't get set, the reason is
|
|
|
|
* probably that garbage collection truncated the file since we did
|
|
|
|
* the fstat(), so we don't log a complaint --- but we don't return
|
|
|
|
* the data, either, since it's most likely corrupt due to concurrent
|
|
|
|
* writes from garbage collection.
|
|
|
|
*/
|
|
|
|
errno = 0;
|
|
|
|
if (read(fd, buf + nread, toread) != toread)
|
|
|
|
{
|
|
|
|
if (errno)
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not read pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
free(buf);
|
|
|
|
CloseTransientFile(fd);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
nread += toread;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
CloseTransientFile(fd);
|
|
|
|
|
2021-11-01 00:13:48 +01:00
|
|
|
*buffer_size = nread;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Locate a query text in the file image previously read by qtext_load_file().
|
|
|
|
*
|
|
|
|
* We validate the given offset/length, and return NULL if bogus. Otherwise,
|
|
|
|
* the result points to a null-terminated string within the buffer.
|
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
qtext_fetch(Size query_offset, int query_len,
|
|
|
|
char *buffer, Size buffer_size)
|
|
|
|
{
|
|
|
|
/* File read failed? */
|
|
|
|
if (buffer == NULL)
|
|
|
|
return NULL;
|
|
|
|
/* Bogus offset/length? */
|
|
|
|
if (query_len < 0 ||
|
|
|
|
query_offset + query_len >= buffer_size)
|
|
|
|
return NULL;
|
|
|
|
/* As a further sanity check, make sure there's a trailing null */
|
|
|
|
if (buffer[query_offset + query_len] != '\0')
|
|
|
|
return NULL;
|
|
|
|
/* Looks OK */
|
|
|
|
return buffer + query_offset;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Do we need to garbage-collect the external query text file?
|
|
|
|
*
|
|
|
|
* Caller should hold at least a shared lock on pgss->lock.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
need_gc_qtexts(void)
|
|
|
|
{
|
|
|
|
Size extent;
|
|
|
|
|
|
|
|
/* Read shared extent pointer */
|
|
|
|
{
|
|
|
|
volatile pgssSharedState *s = (volatile pgssSharedState *) pgss;
|
|
|
|
|
|
|
|
SpinLockAcquire(&s->mutex);
|
|
|
|
extent = s->extent;
|
|
|
|
SpinLockRelease(&s->mutex);
|
|
|
|
}
|
|
|
|
|
Be more wary about 32-bit integer overflow in pg_stat_statements.
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files. It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:
1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits. We still need overflow guards on its use, but
this helps.
2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize. If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.
3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms. It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.
Per report from Bruno da Silva. I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place. But it did get that big, so #2 is a reasonable defense,
and #3 could explain the reported performance difficulties.
(See also commit 8bbe4cbd9, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)
This issue is old, and is primarily a problem for old platforms,
so back-patch.
Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
2022-08-03 00:05:34 +02:00
|
|
|
/*
|
|
|
|
* Don't proceed if file does not exceed 512 bytes per possible entry.
|
|
|
|
*
|
|
|
|
* Here and in the next test, 32-bit machines have overflow hazards if
|
|
|
|
* pgss_max and/or mean_query_len are large. Force the multiplications
|
|
|
|
* and comparisons to be done in uint64 arithmetic to forestall trouble.
|
|
|
|
*/
|
|
|
|
if ((uint64) extent < (uint64) 512 * pgss_max)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
return false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Don't proceed if file is less than about 50% bloat. Nothing can or
|
|
|
|
* should be done in the event of unusually large query texts accounting
|
|
|
|
* for file's large size. We go to the trouble of maintaining the mean
|
|
|
|
* query length in order to prevent garbage collection from thrashing
|
|
|
|
* uselessly.
|
|
|
|
*/
|
Be more wary about 32-bit integer overflow in pg_stat_statements.
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files. It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:
1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits. We still need overflow guards on its use, but
this helps.
2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize. If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.
3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms. It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.
Per report from Bruno da Silva. I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place. But it did get that big, so #2 is a reasonable defense,
and #3 could explain the reported performance difficulties.
(See also commit 8bbe4cbd9, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)
This issue is old, and is primarily a problem for old platforms,
so back-patch.
Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
2022-08-03 00:05:34 +02:00
|
|
|
if ((uint64) extent < (uint64) pgss->mean_query_len * pgss_max * 2)
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
return false;
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Garbage-collect orphaned query texts in external file.
|
|
|
|
*
|
|
|
|
* This won't be called often in the typical case, since it's likely that
|
|
|
|
* there won't be too much churn, and besides, a similar compaction process
|
|
|
|
* occurs when serializing to disk at shutdown or as part of resetting.
|
|
|
|
* Despite this, it seems prudent to plan for the edge case where the file
|
|
|
|
* becomes unreasonably large, with no other method of compaction likely to
|
|
|
|
* occur in the foreseeable future.
|
|
|
|
*
|
|
|
|
* The caller must hold an exclusive lock on pgss->lock.
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
*
|
|
|
|
* At the first sign of trouble we unlink the query text file to get a clean
|
|
|
|
* slate (although existing statistics are retained), rather than risk
|
|
|
|
* thrashing by allowing the same problem case to recur indefinitely.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
gc_qtexts(void)
|
|
|
|
{
|
|
|
|
char *qbuffer;
|
|
|
|
Size qbuffer_size;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
FILE *qfile = NULL;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
HASH_SEQ_STATUS hash_seq;
|
|
|
|
pgssEntry *entry;
|
|
|
|
Size extent;
|
|
|
|
int nentries;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* When called from pgss_store, some other session might have proceeded
|
|
|
|
* with garbage collection in the no-lock-held interim of lock strength
|
2014-05-06 18:12:18 +02:00
|
|
|
* escalation. Check once more that this is actually necessary.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*/
|
|
|
|
if (!need_gc_qtexts())
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
* Load the old texts file. If we fail (out of memory, for instance),
|
|
|
|
* invalidate query texts. Hopefully this is rare. It might seem better
|
|
|
|
* to leave things alone on an OOM failure, but the problem is that the
|
|
|
|
* file is only going to get bigger; hoping for a future non-OOM result is
|
|
|
|
* risky and can easily lead to complete denial of service.
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*/
|
|
|
|
qbuffer = qtext_load_file(&qbuffer_size);
|
|
|
|
if (qbuffer == NULL)
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
goto gc_fail;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We overwrite the query texts file in place, so as to reduce the risk of
|
|
|
|
* an out-of-disk-space failure. Since the file is guaranteed not to get
|
|
|
|
* larger, this should always work on traditional filesystems; though we
|
|
|
|
* could still lose on copy-on-write filesystems.
|
|
|
|
*/
|
|
|
|
qfile = AllocateFile(PGSS_TEXT_FILE, PG_BINARY_W);
|
|
|
|
if (qfile == NULL)
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
goto gc_fail;
|
|
|
|
}
|
|
|
|
|
|
|
|
extent = 0;
|
|
|
|
nentries = 0;
|
|
|
|
|
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
|
|
|
int query_len = entry->query_len;
|
|
|
|
char *qry = qtext_fetch(entry->query_offset,
|
|
|
|
query_len,
|
|
|
|
qbuffer,
|
|
|
|
qbuffer_size);
|
|
|
|
|
|
|
|
if (qry == NULL)
|
|
|
|
{
|
|
|
|
/* Trouble ... drop the text */
|
|
|
|
entry->query_offset = 0;
|
|
|
|
entry->query_len = -1;
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/* entry will not be counted in mean query length computation */
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (fwrite(qry, 1, query_len + 1, qfile) != query_len + 1)
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
hash_seq_term(&hash_seq);
|
|
|
|
goto gc_fail;
|
|
|
|
}
|
|
|
|
|
|
|
|
entry->query_offset = extent;
|
|
|
|
extent += query_len + 1;
|
|
|
|
nentries++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2014-05-06 18:12:18 +02:00
|
|
|
* Truncate away any now-unused space. If this fails for some odd reason,
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* we log it, but there's no need to fail.
|
|
|
|
*/
|
|
|
|
if (ftruncate(fileno(qfile), extent) != 0)
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
errmsg("could not truncate pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
if (FreeFile(qfile))
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
qfile = NULL;
|
|
|
|
goto gc_fail;
|
|
|
|
}
|
|
|
|
|
|
|
|
elog(DEBUG1, "pgss gc of queries file shrunk size from %zu to %zu",
|
|
|
|
pgss->extent, extent);
|
|
|
|
|
|
|
|
/* Reset the shared extent pointer */
|
|
|
|
pgss->extent = extent;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Also update the mean query length, to be sure that need_gc_qtexts()
|
|
|
|
* won't still think we have a problem.
|
|
|
|
*/
|
|
|
|
if (nentries > 0)
|
|
|
|
pgss->mean_query_len = extent / nentries;
|
|
|
|
else
|
|
|
|
pgss->mean_query_len = ASSUMED_LENGTH_INIT;
|
|
|
|
|
|
|
|
free(qbuffer);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* OK, count a garbage collection cycle. (Note: even though we have
|
|
|
|
* exclusive lock on pgss->lock, we must take pgss->mutex for this, since
|
|
|
|
* other processes may examine gc_count while holding only the mutex.
|
|
|
|
* Also, we have to advance the count *after* we've rewritten the file,
|
|
|
|
* else other processes might not realize they read a stale file.)
|
|
|
|
*/
|
|
|
|
record_gc_qtexts();
|
|
|
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
gc_fail:
|
|
|
|
/* clean up resources */
|
|
|
|
if (qfile)
|
|
|
|
FreeFile(qfile);
|
|
|
|
if (qbuffer)
|
|
|
|
free(qbuffer);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Since the contents of the external file are now uncertain, mark all
|
|
|
|
* hashtable entries as having invalid texts.
|
|
|
|
*/
|
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
|
|
|
entry->query_offset = 0;
|
|
|
|
entry->query_len = -1;
|
|
|
|
}
|
|
|
|
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
/*
|
|
|
|
* Destroy the query text file and create a new, empty one
|
|
|
|
*/
|
|
|
|
(void) unlink(PGSS_TEXT_FILE);
|
|
|
|
qfile = AllocateFile(PGSS_TEXT_FILE, PG_BINARY_W);
|
|
|
|
if (qfile == NULL)
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
errmsg("could not write new pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
Improve contrib/pg_stat_statements' handling of garbage collection failure.
If we can't read the query texts file (whether because out-of-memory, or
for some other reason), give up and reset the file to empty, discarding all
stored query texts, though not the statistics per se. We used to leave
things alone and hope for better luck next time, but the problem is that
the file is only going to get bigger and even harder to slurp into memory.
Better to do something that will get us out of trouble.
Likewise reset the file to empty for any other failure within gc_qtexts().
The previous behavior after a write error was to discard query texts but
not do anything to truncate the file, which is just weird.
Also, increase the maximum supported file size from MaxAllocSize to
MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage
collection successfully.
Also, fix recalculation of mean_query_len within entry_dealloc() to match
the calculation in gc_qtexts(). The previous coding overlooked the
possibility of dropped texts (query_len == -1) and would underestimate the
mean of the remaining entries in such cases, thus possibly causing excess
garbage collection cycles.
In passing, add some errdetail to the log entry that complains about
insufficient memory to read the query texts file, which after all was
Jim Nasby's original complaint.
Back-patch to 9.4 where the current handling of query texts was
introduced.
Peter Geoghegan, rather editorialized upon by me
2015-10-04 23:58:29 +02:00
|
|
|
else
|
|
|
|
FreeFile(qfile);
|
|
|
|
|
|
|
|
/* Reset the shared extent pointer */
|
|
|
|
pgss->extent = 0;
|
|
|
|
|
|
|
|
/* Reset mean_query_len to match the new state */
|
|
|
|
pgss->mean_query_len = ASSUMED_LENGTH_INIT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Bump the GC count even though we failed.
|
|
|
|
*
|
|
|
|
* This is needed to make concurrent readers of file without any lock on
|
|
|
|
* pgss->lock notice existence of new version of file. Once readers
|
|
|
|
* subsequently observe a change in GC count with pgss->lock held, that
|
|
|
|
* forces a safe reopen of file. Writers also require that we bump here,
|
|
|
|
* of course. (As required by locking protocol, readers and writers don't
|
|
|
|
* trust earlier file contents until gc_count is found unchanged after
|
|
|
|
* pgss->lock acquired in shared or exclusive mode respectively.)
|
|
|
|
*/
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
record_gc_qtexts();
|
|
|
|
}
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
/*
|
|
|
|
* Release all entries.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
entry_reset(void)
|
|
|
|
{
|
2009-06-11 16:49:15 +02:00
|
|
|
HASH_SEQ_STATUS hash_seq;
|
|
|
|
pgssEntry *entry;
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
FILE *qfile;
|
2009-01-04 23:19:59 +01:00
|
|
|
|
|
|
|
LWLockAcquire(pgss->lock, LW_EXCLUSIVE);
|
|
|
|
|
|
|
|
hash_seq_init(&hash_seq, pgss_hash);
|
|
|
|
while ((entry = hash_seq_search(&hash_seq)) != NULL)
|
|
|
|
{
|
|
|
|
hash_search(pgss_hash, &entry->key, HASH_REMOVE, NULL);
|
|
|
|
}
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/*
|
|
|
|
* Write new empty query file, perhaps even creating a new one to recover
|
|
|
|
* if the file was missing.
|
|
|
|
*/
|
|
|
|
qfile = AllocateFile(PGSS_TEXT_FILE, PG_BINARY_W);
|
|
|
|
if (qfile == NULL)
|
|
|
|
{
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not create pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If ftruncate fails, log it, but it's not a fatal problem */
|
|
|
|
if (ftruncate(fileno(qfile), 0) != 0)
|
|
|
|
ereport(LOG,
|
|
|
|
(errcode_for_file_access(),
|
Phase 3 of pgindent updates.
Don't move parenthesized lines to the left, even if that means they
flow past the right margin.
By default, BSD indent lines up statement continuation lines that are
within parentheses so that they start just to the right of the preceding
left parenthesis. However, traditionally, if that resulted in the
continuation line extending to the right of the desired right margin,
then indent would push it left just far enough to not overrun the margin,
if it could do so without making the continuation line start to the left of
the current statement indent. That makes for a weird mix of indentations
unless one has been completely rigid about never violating the 80-column
limit.
This behavior has been pretty universally panned by Postgres developers.
Hence, disable it with indent's new -lpl switch, so that parenthesized
lines are always lined up with the preceding left paren.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:35:54 +02:00
|
|
|
errmsg("could not truncate pg_stat_statement file \"%s\": %m",
|
|
|
|
PGSS_TEXT_FILE)));
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
|
|
|
|
FreeFile(qfile);
|
|
|
|
|
|
|
|
done:
|
|
|
|
pgss->extent = 0;
|
|
|
|
/* This counts as a query text garbage collection for our purposes */
|
|
|
|
record_gc_qtexts();
|
|
|
|
|
2009-01-04 23:19:59 +01:00
|
|
|
LWLockRelease(pgss->lock);
|
|
|
|
}
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* AppendJumble: Append a value that is substantive in a given query to
|
|
|
|
* the current jumble.
|
|
|
|
*/
|
|
|
|
static void
|
2012-06-10 21:20:04 +02:00
|
|
|
AppendJumble(pgssJumbleState *jstate, const unsigned char *item, Size size)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
unsigned char *jumble = jstate->jumble;
|
|
|
|
Size jumble_len = jstate->jumble_len;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Whenever the jumble buffer is full, we hash the current contents and
|
|
|
|
* reset the buffer to contain just that hash value, thus relying on the
|
|
|
|
* hash to summarize everything so far.
|
|
|
|
*/
|
|
|
|
while (size > 0)
|
|
|
|
{
|
|
|
|
Size part_size;
|
|
|
|
|
|
|
|
if (jumble_len >= JUMBLE_SIZE)
|
|
|
|
{
|
2017-10-12 01:52:46 +02:00
|
|
|
uint64 start_hash;
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2017-10-12 01:52:46 +02:00
|
|
|
start_hash = DatumGetUInt64(hash_any_extended(jumble,
|
|
|
|
JUMBLE_SIZE, 0));
|
2012-03-29 03:00:31 +02:00
|
|
|
memcpy(jumble, &start_hash, sizeof(start_hash));
|
|
|
|
jumble_len = sizeof(start_hash);
|
|
|
|
}
|
|
|
|
part_size = Min(size, JUMBLE_SIZE - jumble_len);
|
|
|
|
memcpy(jumble + jumble_len, item, part_size);
|
|
|
|
jumble_len += part_size;
|
|
|
|
item += part_size;
|
|
|
|
size -= part_size;
|
|
|
|
}
|
|
|
|
jstate->jumble_len = jumble_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Wrappers around AppendJumble to encapsulate details of serialization
|
|
|
|
* of individual local variable elements.
|
|
|
|
*/
|
|
|
|
#define APP_JUMB(item) \
|
|
|
|
AppendJumble(jstate, (const unsigned char *) &(item), sizeof(item))
|
|
|
|
#define APP_JUMB_STRING(str) \
|
|
|
|
AppendJumble(jstate, (const unsigned char *) (str), strlen(str) + 1)
|
|
|
|
|
|
|
|
/*
|
|
|
|
* JumbleQuery: Selectively serialize the query tree, appending significant
|
|
|
|
* data to the "query jumble" while ignoring nonsignificant data.
|
|
|
|
*
|
|
|
|
* Rule of thumb for what to include is that we should ignore anything not
|
|
|
|
* semantically significant (such as alias names) as well as anything that can
|
|
|
|
* be deduced from child nodes (else we'd just be double-hashing that piece
|
|
|
|
* of information).
|
|
|
|
*/
|
|
|
|
static void
|
2012-06-10 21:20:04 +02:00
|
|
|
JumbleQuery(pgssJumbleState *jstate, Query *query)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
Assert(IsA(query, Query));
|
|
|
|
Assert(query->utilityStmt == NULL);
|
|
|
|
|
|
|
|
APP_JUMB(query->commandType);
|
|
|
|
/* resultRelation is usually predictable from commandType */
|
|
|
|
JumbleExpr(jstate, (Node *) query->cteList);
|
|
|
|
JumbleRangeTable(jstate, query->rtable);
|
|
|
|
JumbleExpr(jstate, (Node *) query->jointree);
|
|
|
|
JumbleExpr(jstate, (Node *) query->targetList);
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
JumbleExpr(jstate, (Node *) query->onConflict);
|
2012-03-29 03:00:31 +02:00
|
|
|
JumbleExpr(jstate, (Node *) query->returningList);
|
|
|
|
JumbleExpr(jstate, (Node *) query->groupClause);
|
Support GROUPING SETS, CUBE and ROLLUP.
This SQL standard functionality allows to aggregate data by different
GROUP BY clauses at once. Each grouping set returns rows with columns
grouped by in other sets set to NULL.
This could previously be achieved by doing each grouping as a separate
query, conjoined by UNION ALLs. Besides being considerably more concise,
grouping sets will in many cases be faster, requiring only one scan over
the underlying data.
The current implementation of grouping sets only supports using sorting
for input. Individual sets that share a sort order are computed in one
pass. If there are sets that don't share a sort order, additional sort &
aggregation steps are performed. These additional passes are sourced by
the previous sort step; thus avoiding repeated scans of the source data.
The code is structured in a way that adding support for purely using
hash aggregation or a mix of hashing and sorting is possible. Sorting
was chosen to be supported first, as it is the most generic method of
implementation.
Instead of, as in an earlier versions of the patch, representing the
chain of sort and aggregation steps as full blown planner and executor
nodes, all but the first sort are performed inside the aggregation node
itself. This avoids the need to do some unusual gymnastics to handle
having to return aggregated and non-aggregated tuples from underlying
nodes, as well as having to shut down underlying nodes early to limit
memory usage. The optimizer still builds Sort/Agg node to describe each
phase, but they're not part of the plan tree, but instead additional
data for the aggregation node. They're a convenient and preexisting way
to describe aggregation and sorting. The first (and possibly only) sort
step is still performed as a separate execution step. That retains
similarity with existing group by plans, makes rescans fairly simple,
avoids very deep plans (leading to slow explains) and easily allows to
avoid the sorting step if the underlying data is sorted by other means.
A somewhat ugly side of this patch is having to deal with a grammar
ambiguity between the new CUBE keyword and the cube extension/functions
named cube (and rollup). To avoid breaking existing deployments of the
cube extension it has not been renamed, neither has cube been made a
reserved keyword. Instead precedence hacking is used to make GROUP BY
cube(..) refer to the CUBE grouping sets feature, and not the function
cube(). To actually group by a function cube(), unlikely as that might
be, the function name has to be quoted.
Needs a catversion bump because stored rules may change.
Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund
Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas
Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule
Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com
2015-05-16 03:40:59 +02:00
|
|
|
JumbleExpr(jstate, (Node *) query->groupingSets);
|
2012-03-29 03:00:31 +02:00
|
|
|
JumbleExpr(jstate, query->havingQual);
|
|
|
|
JumbleExpr(jstate, (Node *) query->windowClause);
|
|
|
|
JumbleExpr(jstate, (Node *) query->distinctClause);
|
|
|
|
JumbleExpr(jstate, (Node *) query->sortClause);
|
|
|
|
JumbleExpr(jstate, query->limitOffset);
|
|
|
|
JumbleExpr(jstate, query->limitCount);
|
|
|
|
/* we ignore rowMarks */
|
|
|
|
JumbleExpr(jstate, query->setOperations);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Jumble a range table
|
|
|
|
*/
|
|
|
|
static void
|
2012-06-10 21:20:04 +02:00
|
|
|
JumbleRangeTable(pgssJumbleState *jstate, List *rtable)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
ListCell *lc;
|
|
|
|
|
|
|
|
foreach(lc, rtable)
|
|
|
|
{
|
Improve castNode notation by introducing list-extraction-specific variants.
This extends the castNode() notation introduced by commit 5bcab1114 to
provide, in one step, extraction of a list cell's pointer and coercion to
a concrete node type. For example, "lfirst_node(Foo, lc)" is the same
as "castNode(Foo, lfirst(lc))". Almost half of the uses of castNode
that have appeared so far include a list extraction call, so this is
pretty widely useful, and it saves a few more keystrokes compared to the
old way.
As with the previous patch, back-patch the addition of these macros to
pg_list.h, so that the notation will be available when back-patching.
Patch by me, after an idea of Andrew Gierth's.
Discussion: https://postgr.es/m/14197.1491841216@sss.pgh.pa.us
2017-04-10 19:51:29 +02:00
|
|
|
RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
APP_JUMB(rte->rtekind);
|
|
|
|
switch (rte->rtekind)
|
|
|
|
{
|
|
|
|
case RTE_RELATION:
|
|
|
|
APP_JUMB(rte->relid);
|
Redesign tablesample method API, and do extensive code review.
The original implementation of TABLESAMPLE modeled the tablesample method
API on index access methods, which wasn't a good choice because, without
specialized DDL commands, there's no way to build an extension that can
implement a TSM. (Raw inserts into system catalogs are not an acceptable
thing to do, because we can't undo them during DROP EXTENSION, nor will
pg_upgrade behave sanely.) Instead adopt an API more like procedural
language handlers or foreign data wrappers, wherein the only SQL-level
support object needed is a single handler function identified by having
a special return type. This lets us get rid of the supporting catalog
altogether, so that no custom DDL support is needed for the feature.
Adjust the API so that it can support non-constant tablesample arguments
(the original coding assumed we could evaluate the argument expressions at
ExecInitSampleScan time, which is undesirable even if it weren't outright
unsafe), and discourage sampling methods from looking at invisible tuples.
Make sure that the BERNOULLI and SYSTEM methods are genuinely repeatable
within and across queries, as required by the SQL standard, and deal more
honestly with methods that can't support that requirement.
Make a full code-review pass over the tablesample additions, and fix
assorted bugs, omissions, infelicities, and cosmetic issues (such as
failure to put the added code stanzas in a consistent ordering).
Improve EXPLAIN's output of tablesample plans, too.
Back-patch to 9.5 so that we don't have to support the original API
in production.
2015-07-25 20:39:00 +02:00
|
|
|
JumbleExpr(jstate, (Node *) rte->tablesample);
|
2012-03-29 03:00:31 +02:00
|
|
|
break;
|
|
|
|
case RTE_SUBQUERY:
|
|
|
|
JumbleQuery(jstate, rte->subquery);
|
|
|
|
break;
|
|
|
|
case RTE_JOIN:
|
|
|
|
APP_JUMB(rte->jointype);
|
|
|
|
break;
|
|
|
|
case RTE_FUNCTION:
|
Support multi-argument UNNEST(), and TABLE() syntax for multiple functions.
This patch adds the ability to write TABLE( function1(), function2(), ...)
as a single FROM-clause entry. The result is the concatenation of the
first row from each function, followed by the second row from each
function, etc; with NULLs inserted if any function produces fewer rows than
others. This is believed to be a much more useful behavior than what
Postgres currently does with multiple SRFs in a SELECT list.
This syntax also provides a reasonable way to combine use of column
definition lists with WITH ORDINALITY: put the column definition list
inside TABLE(), where it's clear that it doesn't control the ordinality
column as well.
Also implement SQL-compliant multiple-argument UNNEST(), by turning
UNNEST(a,b,c) into TABLE(unnest(a), unnest(b), unnest(c)).
The SQL standard specifies TABLE() with only a single function, not
multiple functions, and it seems to require an implicit UNNEST() which is
not what this patch does. There may be something wrong with that reading
of the spec, though, because if it's right then the spec's TABLE() is just
a pointless alternative spelling of UNNEST(). After further review of
that, we might choose to adopt a different syntax for what this patch does,
but in any case this functionality seems clearly worthwhile.
Andrew Gierth, reviewed by Zoltán Böszörményi and Heikki Linnakangas, and
significantly revised by me
2013-11-22 01:37:02 +01:00
|
|
|
JumbleExpr(jstate, (Node *) rte->functions);
|
2012-03-29 03:00:31 +02:00
|
|
|
break;
|
2017-03-08 16:39:37 +01:00
|
|
|
case RTE_TABLEFUNC:
|
|
|
|
JumbleExpr(jstate, (Node *) rte->tablefunc);
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case RTE_VALUES:
|
|
|
|
JumbleExpr(jstate, (Node *) rte->values_lists);
|
|
|
|
break;
|
|
|
|
case RTE_CTE:
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Depending on the CTE name here isn't ideal, but it's the
|
|
|
|
* only info we have to identify the referenced WITH item.
|
|
|
|
*/
|
|
|
|
APP_JUMB_STRING(rte->ctename);
|
|
|
|
APP_JUMB(rte->ctelevelsup);
|
|
|
|
break;
|
2017-04-01 06:17:18 +02:00
|
|
|
case RTE_NAMEDTUPLESTORE:
|
|
|
|
APP_JUMB_STRING(rte->enrname);
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
default:
|
|
|
|
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Jumble an expression tree
|
|
|
|
*
|
|
|
|
* In general this function should handle all the same node types that
|
|
|
|
* expression_tree_walker() does, and therefore it's coded to be as parallel
|
|
|
|
* to that function as possible. However, since we are only invoked on
|
|
|
|
* queries immediately post-parse-analysis, we need not handle node types
|
|
|
|
* that only appear in planning.
|
|
|
|
*
|
|
|
|
* Note: the reason we don't simply use expression_tree_walker() is that the
|
|
|
|
* point of that function is to support tree walkers that don't care about
|
2014-05-06 18:12:18 +02:00
|
|
|
* most tree node types, but here we care about all types. We should complain
|
2012-03-29 03:00:31 +02:00
|
|
|
* about any unrecognized node type.
|
|
|
|
*/
|
|
|
|
static void
|
2012-06-10 21:20:04 +02:00
|
|
|
JumbleExpr(pgssJumbleState *jstate, Node *node)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
ListCell *temp;
|
|
|
|
|
|
|
|
if (node == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Guard against stack overflow due to overly complex expressions */
|
|
|
|
check_stack_depth();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We always emit the node's NodeTag, then any additional fields that are
|
|
|
|
* considered significant, and then we recurse to any child nodes.
|
|
|
|
*/
|
|
|
|
APP_JUMB(node->type);
|
|
|
|
|
|
|
|
switch (nodeTag(node))
|
|
|
|
{
|
|
|
|
case T_Var:
|
|
|
|
{
|
|
|
|
Var *var = (Var *) node;
|
|
|
|
|
|
|
|
APP_JUMB(var->varno);
|
|
|
|
APP_JUMB(var->varattno);
|
|
|
|
APP_JUMB(var->varlevelsup);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_Const:
|
|
|
|
{
|
|
|
|
Const *c = (Const *) node;
|
|
|
|
|
|
|
|
/* We jumble only the constant's type, not its value */
|
|
|
|
APP_JUMB(c->consttype);
|
|
|
|
/* Also, record its parse location for query normalization */
|
|
|
|
RecordConstLocation(jstate, c->location);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_Param:
|
|
|
|
{
|
|
|
|
Param *p = (Param *) node;
|
|
|
|
|
|
|
|
APP_JUMB(p->paramkind);
|
|
|
|
APP_JUMB(p->paramid);
|
|
|
|
APP_JUMB(p->paramtype);
|
2017-03-28 02:14:36 +02:00
|
|
|
/* Also, track the highest external Param id */
|
|
|
|
if (p->paramkind == PARAM_EXTERN &&
|
|
|
|
p->paramid > jstate->highest_extern_param_id)
|
|
|
|
jstate->highest_extern_param_id = p->paramid;
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_Aggref:
|
|
|
|
{
|
|
|
|
Aggref *expr = (Aggref *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->aggfnoid);
|
Support ordered-set (WITHIN GROUP) aggregates.
This patch introduces generic support for ordered-set and hypothetical-set
aggregate functions, as well as implementations of the instances defined in
SQL:2008 (percentile_cont(), percentile_disc(), rank(), dense_rank(),
percent_rank(), cume_dist()). We also added mode() though it is not in the
spec, as well as versions of percentile_cont() and percentile_disc() that
can compute multiple percentile values in one pass over the data.
Unlike the original submission, this patch puts full control of the sorting
process in the hands of the aggregate's support functions. To allow the
support functions to find out how they're supposed to sort, a new API
function AggGetAggref() is added to nodeAgg.c. This allows retrieval of
the aggregate call's Aggref node, which may have other uses beyond the
immediate need. There is also support for ordered-set aggregates to
install cleanup callback functions, so that they can be sure that
infrastructure such as tuplesort objects gets cleaned up.
In passing, make some fixes in the recently-added support for variadic
aggregates, and make some editorial adjustments in the recent FILTER
additions for aggregates. Also, simplify use of IsBinaryCoercible() by
allowing it to succeed whenever the target type is ANY or ANYELEMENT.
It was inconsistent that it dealt with other polymorphic target types
but not these.
Atri Sharma and Andrew Gierth; reviewed by Pavel Stehule and Vik Fearing,
and rather heavily editorialized upon by Tom Lane
2013-12-23 22:11:35 +01:00
|
|
|
JumbleExpr(jstate, (Node *) expr->aggdirectargs);
|
2012-03-29 03:00:31 +02:00
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->aggorder);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->aggdistinct);
|
2013-07-17 02:15:36 +02:00
|
|
|
JumbleExpr(jstate, (Node *) expr->aggfilter);
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
Support GROUPING SETS, CUBE and ROLLUP.
This SQL standard functionality allows to aggregate data by different
GROUP BY clauses at once. Each grouping set returns rows with columns
grouped by in other sets set to NULL.
This could previously be achieved by doing each grouping as a separate
query, conjoined by UNION ALLs. Besides being considerably more concise,
grouping sets will in many cases be faster, requiring only one scan over
the underlying data.
The current implementation of grouping sets only supports using sorting
for input. Individual sets that share a sort order are computed in one
pass. If there are sets that don't share a sort order, additional sort &
aggregation steps are performed. These additional passes are sourced by
the previous sort step; thus avoiding repeated scans of the source data.
The code is structured in a way that adding support for purely using
hash aggregation or a mix of hashing and sorting is possible. Sorting
was chosen to be supported first, as it is the most generic method of
implementation.
Instead of, as in an earlier versions of the patch, representing the
chain of sort and aggregation steps as full blown planner and executor
nodes, all but the first sort are performed inside the aggregation node
itself. This avoids the need to do some unusual gymnastics to handle
having to return aggregated and non-aggregated tuples from underlying
nodes, as well as having to shut down underlying nodes early to limit
memory usage. The optimizer still builds Sort/Agg node to describe each
phase, but they're not part of the plan tree, but instead additional
data for the aggregation node. They're a convenient and preexisting way
to describe aggregation and sorting. The first (and possibly only) sort
step is still performed as a separate execution step. That retains
similarity with existing group by plans, makes rescans fairly simple,
avoids very deep plans (leading to slow explains) and easily allows to
avoid the sorting step if the underlying data is sorted by other means.
A somewhat ugly side of this patch is having to deal with a grammar
ambiguity between the new CUBE keyword and the cube extension/functions
named cube (and rollup). To avoid breaking existing deployments of the
cube extension it has not been renamed, neither has cube been made a
reserved keyword. Instead precedence hacking is used to make GROUP BY
cube(..) refer to the CUBE grouping sets feature, and not the function
cube(). To actually group by a function cube(), unlikely as that might
be, the function name has to be quoted.
Needs a catversion bump because stored rules may change.
Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund
Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas
Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule
Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com
2015-05-16 03:40:59 +02:00
|
|
|
case T_GroupingFunc:
|
|
|
|
{
|
|
|
|
GroupingFunc *grpnode = (GroupingFunc *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) grpnode->refs);
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_WindowFunc:
|
|
|
|
{
|
|
|
|
WindowFunc *expr = (WindowFunc *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->winfnoid);
|
|
|
|
APP_JUMB(expr->winref);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
2013-07-17 02:15:36 +02:00
|
|
|
JumbleExpr(jstate, (Node *) expr->aggfilter);
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_ArrayRef:
|
|
|
|
{
|
|
|
|
ArrayRef *aref = (ArrayRef *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) aref->refupperindexpr);
|
|
|
|
JumbleExpr(jstate, (Node *) aref->reflowerindexpr);
|
|
|
|
JumbleExpr(jstate, (Node *) aref->refexpr);
|
|
|
|
JumbleExpr(jstate, (Node *) aref->refassgnexpr);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_FuncExpr:
|
|
|
|
{
|
|
|
|
FuncExpr *expr = (FuncExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->funcid);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_NamedArgExpr:
|
|
|
|
{
|
|
|
|
NamedArgExpr *nae = (NamedArgExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(nae->argnumber);
|
|
|
|
JumbleExpr(jstate, (Node *) nae->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_OpExpr:
|
|
|
|
case T_DistinctExpr: /* struct-equivalent to OpExpr */
|
|
|
|
case T_NullIfExpr: /* struct-equivalent to OpExpr */
|
|
|
|
{
|
|
|
|
OpExpr *expr = (OpExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->opno);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_ScalarArrayOpExpr:
|
|
|
|
{
|
|
|
|
ScalarArrayOpExpr *expr = (ScalarArrayOpExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->opno);
|
|
|
|
APP_JUMB(expr->useOr);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_BoolExpr:
|
|
|
|
{
|
|
|
|
BoolExpr *expr = (BoolExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(expr->boolop);
|
|
|
|
JumbleExpr(jstate, (Node *) expr->args);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_SubLink:
|
|
|
|
{
|
|
|
|
SubLink *sublink = (SubLink *) node;
|
|
|
|
|
|
|
|
APP_JUMB(sublink->subLinkType);
|
Implement UPDATE tab SET (col1,col2,...) = (SELECT ...), ...
This SQL-standard feature allows a sub-SELECT yielding multiple columns
(but only one row) to be used to compute the new values of several columns
to be updated. While the same results can be had with an independent
sub-SELECT per column, such a workaround can require a great deal of
duplicated computation.
The standard actually says that the source for a multi-column assignment
could be any row-valued expression. The implementation used here is
tightly tied to our existing sub-SELECT support and can't handle other
cases; the Bison grammar would have some issues with them too. However,
I don't feel too bad about this since other cases can be converted into
sub-SELECTs. For instance, "SET (a,b,c) = row_valued_function(x)" could
be written "SET (a,b,c) = (SELECT * FROM row_valued_function(x))".
2014-06-18 19:22:25 +02:00
|
|
|
APP_JUMB(sublink->subLinkId);
|
2012-03-29 03:00:31 +02:00
|
|
|
JumbleExpr(jstate, (Node *) sublink->testexpr);
|
2017-01-27 01:47:03 +01:00
|
|
|
JumbleQuery(jstate, castNode(Query, sublink->subselect));
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_FieldSelect:
|
|
|
|
{
|
|
|
|
FieldSelect *fs = (FieldSelect *) node;
|
|
|
|
|
|
|
|
APP_JUMB(fs->fieldnum);
|
|
|
|
JumbleExpr(jstate, (Node *) fs->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_FieldStore:
|
|
|
|
{
|
|
|
|
FieldStore *fstore = (FieldStore *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) fstore->arg);
|
|
|
|
JumbleExpr(jstate, (Node *) fstore->newvals);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_RelabelType:
|
|
|
|
{
|
|
|
|
RelabelType *rt = (RelabelType *) node;
|
|
|
|
|
|
|
|
APP_JUMB(rt->resulttype);
|
|
|
|
JumbleExpr(jstate, (Node *) rt->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CoerceViaIO:
|
|
|
|
{
|
|
|
|
CoerceViaIO *cio = (CoerceViaIO *) node;
|
|
|
|
|
|
|
|
APP_JUMB(cio->resulttype);
|
|
|
|
JumbleExpr(jstate, (Node *) cio->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_ArrayCoerceExpr:
|
|
|
|
{
|
|
|
|
ArrayCoerceExpr *acexpr = (ArrayCoerceExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(acexpr->resulttype);
|
|
|
|
JumbleExpr(jstate, (Node *) acexpr->arg);
|
Support arrays over domains.
Allowing arrays with a domain type as their element type was left un-done
in the original domain patch, but not for any very good reason. This
omission leads to such surprising results as array_agg() not working on
a domain column, because the parser can't identify a suitable output type
for the polymorphic aggregate.
In order to fix this, first clean up the APIs of coerce_to_domain() and
some internal functions in parse_coerce.c so that we consistently pass
around a CoercionContext along with CoercionForm. Previously, we sometimes
passed an "isExplicit" boolean flag instead, which is strictly less
information; and coerce_to_domain() didn't even get that, but instead had
to reverse-engineer isExplicit from CoercionForm. That's contrary to the
documentation in primnodes.h that says that CoercionForm only affects
display and not semantics. I don't think this change fixes any live bugs,
but it makes things more consistent. The main reason for doing it though
is that now build_coercion_expression() receives ccontext, which it needs
in order to be able to recursively invoke coerce_to_target_type().
Next, reimplement ArrayCoerceExpr so that the node does not directly know
any details of what has to be done to the individual array elements while
performing the array coercion. Instead, the per-element processing is
represented by a sub-expression whose input is a source array element and
whose output is a target array element. This simplifies life in
parse_coerce.c, because it can build that sub-expression by a recursive
invocation of coerce_to_target_type(). The executor now handles the
per-element processing as a compiled expression instead of hard-wired code.
The main advantage of this is that we can use a single ArrayCoerceExpr to
handle as many as three successive steps per element: base type conversion,
typmod coercion, and domain constraint checking. The old code used two
stacked ArrayCoerceExprs to handle type + typmod coercion, which was pretty
inefficient, and adding yet another array deconstruction to do domain
constraint checking seemed very unappetizing.
In the case where we just need a single, very simple coercion function,
doing this straightforwardly leads to a noticeable increase in the
per-array-element runtime cost. Hence, add an additional shortcut evalfunc
in execExprInterp.c that skips unnecessary overhead for that specific form
of expression. The runtime speed of simple cases is within 1% or so of
where it was before, while cases that previously required two levels of
array processing are significantly faster.
Finally, create an implicit array type for every domain type, as we do for
base types, enums, etc. Everything except the array-coercion case seems
to just work without further effort.
Tom Lane, reviewed by Andrew Dunstan
Discussion: https://postgr.es/m/9852.1499791473@sss.pgh.pa.us
2017-09-30 19:40:56 +02:00
|
|
|
JumbleExpr(jstate, (Node *) acexpr->elemexpr);
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_ConvertRowtypeExpr:
|
|
|
|
{
|
|
|
|
ConvertRowtypeExpr *crexpr = (ConvertRowtypeExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(crexpr->resulttype);
|
|
|
|
JumbleExpr(jstate, (Node *) crexpr->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CollateExpr:
|
|
|
|
{
|
|
|
|
CollateExpr *ce = (CollateExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(ce->collOid);
|
|
|
|
JumbleExpr(jstate, (Node *) ce->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CaseExpr:
|
|
|
|
{
|
|
|
|
CaseExpr *caseexpr = (CaseExpr *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) caseexpr->arg);
|
|
|
|
foreach(temp, caseexpr->args)
|
|
|
|
{
|
Improve castNode notation by introducing list-extraction-specific variants.
This extends the castNode() notation introduced by commit 5bcab1114 to
provide, in one step, extraction of a list cell's pointer and coercion to
a concrete node type. For example, "lfirst_node(Foo, lc)" is the same
as "castNode(Foo, lfirst(lc))". Almost half of the uses of castNode
that have appeared so far include a list extraction call, so this is
pretty widely useful, and it saves a few more keystrokes compared to the
old way.
As with the previous patch, back-patch the addition of these macros to
pg_list.h, so that the notation will be available when back-patching.
Patch by me, after an idea of Andrew Gierth's.
Discussion: https://postgr.es/m/14197.1491841216@sss.pgh.pa.us
2017-04-10 19:51:29 +02:00
|
|
|
CaseWhen *when = lfirst_node(CaseWhen, temp);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) when->expr);
|
|
|
|
JumbleExpr(jstate, (Node *) when->result);
|
|
|
|
}
|
|
|
|
JumbleExpr(jstate, (Node *) caseexpr->defresult);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CaseTestExpr:
|
|
|
|
{
|
|
|
|
CaseTestExpr *ct = (CaseTestExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(ct->typeId);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_ArrayExpr:
|
|
|
|
JumbleExpr(jstate, (Node *) ((ArrayExpr *) node)->elements);
|
|
|
|
break;
|
|
|
|
case T_RowExpr:
|
|
|
|
JumbleExpr(jstate, (Node *) ((RowExpr *) node)->args);
|
|
|
|
break;
|
|
|
|
case T_RowCompareExpr:
|
|
|
|
{
|
|
|
|
RowCompareExpr *rcexpr = (RowCompareExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(rcexpr->rctype);
|
|
|
|
JumbleExpr(jstate, (Node *) rcexpr->largs);
|
|
|
|
JumbleExpr(jstate, (Node *) rcexpr->rargs);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CoalesceExpr:
|
|
|
|
JumbleExpr(jstate, (Node *) ((CoalesceExpr *) node)->args);
|
|
|
|
break;
|
|
|
|
case T_MinMaxExpr:
|
|
|
|
{
|
|
|
|
MinMaxExpr *mmexpr = (MinMaxExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(mmexpr->op);
|
|
|
|
JumbleExpr(jstate, (Node *) mmexpr->args);
|
|
|
|
}
|
|
|
|
break;
|
2016-08-17 02:33:01 +02:00
|
|
|
case T_SQLValueFunction:
|
|
|
|
{
|
|
|
|
SQLValueFunction *svf = (SQLValueFunction *) node;
|
|
|
|
|
|
|
|
APP_JUMB(svf->op);
|
|
|
|
/* type is fully determined by op */
|
|
|
|
APP_JUMB(svf->typmod);
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_XmlExpr:
|
|
|
|
{
|
|
|
|
XmlExpr *xexpr = (XmlExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(xexpr->op);
|
|
|
|
JumbleExpr(jstate, (Node *) xexpr->named_args);
|
|
|
|
JumbleExpr(jstate, (Node *) xexpr->args);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_NullTest:
|
|
|
|
{
|
|
|
|
NullTest *nt = (NullTest *) node;
|
|
|
|
|
|
|
|
APP_JUMB(nt->nulltesttype);
|
|
|
|
JumbleExpr(jstate, (Node *) nt->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_BooleanTest:
|
|
|
|
{
|
|
|
|
BooleanTest *bt = (BooleanTest *) node;
|
|
|
|
|
|
|
|
APP_JUMB(bt->booltesttype);
|
|
|
|
JumbleExpr(jstate, (Node *) bt->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CoerceToDomain:
|
|
|
|
{
|
|
|
|
CoerceToDomain *cd = (CoerceToDomain *) node;
|
|
|
|
|
|
|
|
APP_JUMB(cd->resulttype);
|
|
|
|
JumbleExpr(jstate, (Node *) cd->arg);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CoerceToDomainValue:
|
|
|
|
{
|
|
|
|
CoerceToDomainValue *cdv = (CoerceToDomainValue *) node;
|
|
|
|
|
|
|
|
APP_JUMB(cdv->typeId);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_SetToDefault:
|
|
|
|
{
|
|
|
|
SetToDefault *sd = (SetToDefault *) node;
|
|
|
|
|
|
|
|
APP_JUMB(sd->typeId);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CurrentOfExpr:
|
|
|
|
{
|
|
|
|
CurrentOfExpr *ce = (CurrentOfExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(ce->cvarno);
|
|
|
|
if (ce->cursor_name)
|
|
|
|
APP_JUMB_STRING(ce->cursor_name);
|
|
|
|
APP_JUMB(ce->cursor_param);
|
|
|
|
}
|
|
|
|
break;
|
Code review for NextValueExpr expression node type.
Add missing infrastructure for this node type, notably in ruleutils.c where
its lack could demonstrably cause EXPLAIN to fail. Add outfuncs/readfuncs
support. (outfuncs support is useful today for debugging purposes. The
readfuncs support may never be needed, since at present it would only
matter for parallel query and NextValueExpr should never appear in a
parallelizable query; but it seems like a bad idea to have a primnode type
that isn't fully supported here.) Teach planner infrastructure that
NextValueExpr is a volatile, parallel-unsafe, non-leaky expression node
with cost cpu_operator_cost. Given its limited scope of usage, there
*might* be no live bug today from the lack of that knowledge, but it's
certainly going to bite us on the rear someday. Teach pg_stat_statements
about the new node type, too.
While at it, also teach cost_qual_eval() that MinMaxExpr, SQLValueFunction,
XmlExpr, and CoerceToDomain should be charged as cpu_operator_cost.
Failing to do this for SQLValueFunction was an oversight in my commit
0bb51aa96. The others are longer-standing oversights, but no time like the
present to fix them. (In principle, CoerceToDomain could have cost much
higher than this, but it doesn't presently seem worth trying to examine the
domain's constraints here.)
Modify execExprInterp.c to execute NextValueExpr as an out-of-line
function; it seems quite unlikely to me that it's worth insisting that
it be inlined in all expression eval methods. Besides, providing the
out-of-line function doesn't stop anyone from inlining if they want to.
Adjust some places where NextValueExpr support had been inserted with the
aid of a dartboard rather than keeping it in the same order as elsewhere.
Discussion: https://postgr.es/m/23862.1499981661@sss.pgh.pa.us
2017-07-14 21:25:43 +02:00
|
|
|
case T_NextValueExpr:
|
|
|
|
{
|
|
|
|
NextValueExpr *nve = (NextValueExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(nve->seqid);
|
|
|
|
APP_JUMB(nve->typeId);
|
|
|
|
}
|
|
|
|
break;
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
case T_InferenceElem:
|
|
|
|
{
|
|
|
|
InferenceElem *ie = (InferenceElem *) node;
|
|
|
|
|
|
|
|
APP_JUMB(ie->infercollid);
|
2015-05-19 21:17:52 +02:00
|
|
|
APP_JUMB(ie->inferopclass);
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
JumbleExpr(jstate, ie->expr);
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_TargetEntry:
|
|
|
|
{
|
|
|
|
TargetEntry *tle = (TargetEntry *) node;
|
|
|
|
|
|
|
|
APP_JUMB(tle->resno);
|
|
|
|
APP_JUMB(tle->ressortgroupref);
|
|
|
|
JumbleExpr(jstate, (Node *) tle->expr);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_RangeTblRef:
|
|
|
|
{
|
|
|
|
RangeTblRef *rtr = (RangeTblRef *) node;
|
|
|
|
|
|
|
|
APP_JUMB(rtr->rtindex);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_JoinExpr:
|
|
|
|
{
|
|
|
|
JoinExpr *join = (JoinExpr *) node;
|
|
|
|
|
|
|
|
APP_JUMB(join->jointype);
|
|
|
|
APP_JUMB(join->isNatural);
|
|
|
|
APP_JUMB(join->rtindex);
|
|
|
|
JumbleExpr(jstate, join->larg);
|
|
|
|
JumbleExpr(jstate, join->rarg);
|
|
|
|
JumbleExpr(jstate, join->quals);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_FromExpr:
|
|
|
|
{
|
|
|
|
FromExpr *from = (FromExpr *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) from->fromlist);
|
|
|
|
JumbleExpr(jstate, from->quals);
|
|
|
|
}
|
|
|
|
break;
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
case T_OnConflictExpr:
|
|
|
|
{
|
2015-05-24 03:35:49 +02:00
|
|
|
OnConflictExpr *conf = (OnConflictExpr *) node;
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
|
|
|
|
APP_JUMB(conf->action);
|
|
|
|
JumbleExpr(jstate, (Node *) conf->arbiterElems);
|
|
|
|
JumbleExpr(jstate, conf->arbiterWhere);
|
2015-05-24 03:35:49 +02:00
|
|
|
JumbleExpr(jstate, (Node *) conf->onConflictSet);
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
JumbleExpr(jstate, conf->onConflictWhere);
|
|
|
|
APP_JUMB(conf->constraint);
|
|
|
|
APP_JUMB(conf->exclRelIndex);
|
2015-05-24 03:35:49 +02:00
|
|
|
JumbleExpr(jstate, (Node *) conf->exclRelTlist);
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_List:
|
|
|
|
foreach(temp, (List *) node)
|
|
|
|
{
|
|
|
|
JumbleExpr(jstate, (Node *) lfirst(temp));
|
|
|
|
}
|
|
|
|
break;
|
Support GROUPING SETS, CUBE and ROLLUP.
This SQL standard functionality allows to aggregate data by different
GROUP BY clauses at once. Each grouping set returns rows with columns
grouped by in other sets set to NULL.
This could previously be achieved by doing each grouping as a separate
query, conjoined by UNION ALLs. Besides being considerably more concise,
grouping sets will in many cases be faster, requiring only one scan over
the underlying data.
The current implementation of grouping sets only supports using sorting
for input. Individual sets that share a sort order are computed in one
pass. If there are sets that don't share a sort order, additional sort &
aggregation steps are performed. These additional passes are sourced by
the previous sort step; thus avoiding repeated scans of the source data.
The code is structured in a way that adding support for purely using
hash aggregation or a mix of hashing and sorting is possible. Sorting
was chosen to be supported first, as it is the most generic method of
implementation.
Instead of, as in an earlier versions of the patch, representing the
chain of sort and aggregation steps as full blown planner and executor
nodes, all but the first sort are performed inside the aggregation node
itself. This avoids the need to do some unusual gymnastics to handle
having to return aggregated and non-aggregated tuples from underlying
nodes, as well as having to shut down underlying nodes early to limit
memory usage. The optimizer still builds Sort/Agg node to describe each
phase, but they're not part of the plan tree, but instead additional
data for the aggregation node. They're a convenient and preexisting way
to describe aggregation and sorting. The first (and possibly only) sort
step is still performed as a separate execution step. That retains
similarity with existing group by plans, makes rescans fairly simple,
avoids very deep plans (leading to slow explains) and easily allows to
avoid the sorting step if the underlying data is sorted by other means.
A somewhat ugly side of this patch is having to deal with a grammar
ambiguity between the new CUBE keyword and the cube extension/functions
named cube (and rollup). To avoid breaking existing deployments of the
cube extension it has not been renamed, neither has cube been made a
reserved keyword. Instead precedence hacking is used to make GROUP BY
cube(..) refer to the CUBE grouping sets feature, and not the function
cube(). To actually group by a function cube(), unlikely as that might
be, the function name has to be quoted.
Needs a catversion bump because stored rules may change.
Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund
Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas
Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule
Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com
2015-05-16 03:40:59 +02:00
|
|
|
case T_IntList:
|
|
|
|
foreach(temp, (List *) node)
|
|
|
|
{
|
|
|
|
APP_JUMB(lfirst_int(temp));
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_SortGroupClause:
|
|
|
|
{
|
|
|
|
SortGroupClause *sgc = (SortGroupClause *) node;
|
|
|
|
|
|
|
|
APP_JUMB(sgc->tleSortGroupRef);
|
|
|
|
APP_JUMB(sgc->eqop);
|
|
|
|
APP_JUMB(sgc->sortop);
|
|
|
|
APP_JUMB(sgc->nulls_first);
|
|
|
|
}
|
|
|
|
break;
|
Support GROUPING SETS, CUBE and ROLLUP.
This SQL standard functionality allows to aggregate data by different
GROUP BY clauses at once. Each grouping set returns rows with columns
grouped by in other sets set to NULL.
This could previously be achieved by doing each grouping as a separate
query, conjoined by UNION ALLs. Besides being considerably more concise,
grouping sets will in many cases be faster, requiring only one scan over
the underlying data.
The current implementation of grouping sets only supports using sorting
for input. Individual sets that share a sort order are computed in one
pass. If there are sets that don't share a sort order, additional sort &
aggregation steps are performed. These additional passes are sourced by
the previous sort step; thus avoiding repeated scans of the source data.
The code is structured in a way that adding support for purely using
hash aggregation or a mix of hashing and sorting is possible. Sorting
was chosen to be supported first, as it is the most generic method of
implementation.
Instead of, as in an earlier versions of the patch, representing the
chain of sort and aggregation steps as full blown planner and executor
nodes, all but the first sort are performed inside the aggregation node
itself. This avoids the need to do some unusual gymnastics to handle
having to return aggregated and non-aggregated tuples from underlying
nodes, as well as having to shut down underlying nodes early to limit
memory usage. The optimizer still builds Sort/Agg node to describe each
phase, but they're not part of the plan tree, but instead additional
data for the aggregation node. They're a convenient and preexisting way
to describe aggregation and sorting. The first (and possibly only) sort
step is still performed as a separate execution step. That retains
similarity with existing group by plans, makes rescans fairly simple,
avoids very deep plans (leading to slow explains) and easily allows to
avoid the sorting step if the underlying data is sorted by other means.
A somewhat ugly side of this patch is having to deal with a grammar
ambiguity between the new CUBE keyword and the cube extension/functions
named cube (and rollup). To avoid breaking existing deployments of the
cube extension it has not been renamed, neither has cube been made a
reserved keyword. Instead precedence hacking is used to make GROUP BY
cube(..) refer to the CUBE grouping sets feature, and not the function
cube(). To actually group by a function cube(), unlikely as that might
be, the function name has to be quoted.
Needs a catversion bump because stored rules may change.
Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund
Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas
Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule
Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com
2015-05-16 03:40:59 +02:00
|
|
|
case T_GroupingSet:
|
|
|
|
{
|
|
|
|
GroupingSet *gsnode = (GroupingSet *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, (Node *) gsnode->content);
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
case T_WindowClause:
|
|
|
|
{
|
|
|
|
WindowClause *wc = (WindowClause *) node;
|
|
|
|
|
|
|
|
APP_JUMB(wc->winref);
|
|
|
|
APP_JUMB(wc->frameOptions);
|
|
|
|
JumbleExpr(jstate, (Node *) wc->partitionClause);
|
|
|
|
JumbleExpr(jstate, (Node *) wc->orderClause);
|
|
|
|
JumbleExpr(jstate, wc->startOffset);
|
|
|
|
JumbleExpr(jstate, wc->endOffset);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_CommonTableExpr:
|
|
|
|
{
|
|
|
|
CommonTableExpr *cte = (CommonTableExpr *) node;
|
|
|
|
|
2012-03-29 21:32:50 +02:00
|
|
|
/* we store the string name because RTE_CTE RTEs need it */
|
|
|
|
APP_JUMB_STRING(cte->ctename);
|
2017-01-27 01:47:03 +01:00
|
|
|
JumbleQuery(jstate, castNode(Query, cte->ctequery));
|
2012-03-29 03:00:31 +02:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case T_SetOperationStmt:
|
|
|
|
{
|
|
|
|
SetOperationStmt *setop = (SetOperationStmt *) node;
|
|
|
|
|
|
|
|
APP_JUMB(setop->op);
|
|
|
|
APP_JUMB(setop->all);
|
|
|
|
JumbleExpr(jstate, setop->larg);
|
|
|
|
JumbleExpr(jstate, setop->rarg);
|
|
|
|
}
|
|
|
|
break;
|
Support multi-argument UNNEST(), and TABLE() syntax for multiple functions.
This patch adds the ability to write TABLE( function1(), function2(), ...)
as a single FROM-clause entry. The result is the concatenation of the
first row from each function, followed by the second row from each
function, etc; with NULLs inserted if any function produces fewer rows than
others. This is believed to be a much more useful behavior than what
Postgres currently does with multiple SRFs in a SELECT list.
This syntax also provides a reasonable way to combine use of column
definition lists with WITH ORDINALITY: put the column definition list
inside TABLE(), where it's clear that it doesn't control the ordinality
column as well.
Also implement SQL-compliant multiple-argument UNNEST(), by turning
UNNEST(a,b,c) into TABLE(unnest(a), unnest(b), unnest(c)).
The SQL standard specifies TABLE() with only a single function, not
multiple functions, and it seems to require an implicit UNNEST() which is
not what this patch does. There may be something wrong with that reading
of the spec, though, because if it's right then the spec's TABLE() is just
a pointless alternative spelling of UNNEST(). After further review of
that, we might choose to adopt a different syntax for what this patch does,
but in any case this functionality seems clearly worthwhile.
Andrew Gierth, reviewed by Zoltán Böszörményi and Heikki Linnakangas, and
significantly revised by me
2013-11-22 01:37:02 +01:00
|
|
|
case T_RangeTblFunction:
|
|
|
|
{
|
|
|
|
RangeTblFunction *rtfunc = (RangeTblFunction *) node;
|
|
|
|
|
|
|
|
JumbleExpr(jstate, rtfunc->funcexpr);
|
|
|
|
}
|
|
|
|
break;
|
2017-03-08 16:39:37 +01:00
|
|
|
case T_TableFunc:
|
|
|
|
{
|
2017-03-28 02:14:36 +02:00
|
|
|
TableFunc *tablefunc = (TableFunc *) node;
|
2017-03-08 16:39:37 +01:00
|
|
|
|
|
|
|
JumbleExpr(jstate, tablefunc->docexpr);
|
|
|
|
JumbleExpr(jstate, tablefunc->rowexpr);
|
|
|
|
JumbleExpr(jstate, (Node *) tablefunc->colexprs);
|
|
|
|
}
|
|
|
|
break;
|
Redesign tablesample method API, and do extensive code review.
The original implementation of TABLESAMPLE modeled the tablesample method
API on index access methods, which wasn't a good choice because, without
specialized DDL commands, there's no way to build an extension that can
implement a TSM. (Raw inserts into system catalogs are not an acceptable
thing to do, because we can't undo them during DROP EXTENSION, nor will
pg_upgrade behave sanely.) Instead adopt an API more like procedural
language handlers or foreign data wrappers, wherein the only SQL-level
support object needed is a single handler function identified by having
a special return type. This lets us get rid of the supporting catalog
altogether, so that no custom DDL support is needed for the feature.
Adjust the API so that it can support non-constant tablesample arguments
(the original coding assumed we could evaluate the argument expressions at
ExecInitSampleScan time, which is undesirable even if it weren't outright
unsafe), and discourage sampling methods from looking at invisible tuples.
Make sure that the BERNOULLI and SYSTEM methods are genuinely repeatable
within and across queries, as required by the SQL standard, and deal more
honestly with methods that can't support that requirement.
Make a full code-review pass over the tablesample additions, and fix
assorted bugs, omissions, infelicities, and cosmetic issues (such as
failure to put the added code stanzas in a consistent ordering).
Improve EXPLAIN's output of tablesample plans, too.
Back-patch to 9.5 so that we don't have to support the original API
in production.
2015-07-25 20:39:00 +02:00
|
|
|
case T_TableSampleClause:
|
|
|
|
{
|
|
|
|
TableSampleClause *tsc = (TableSampleClause *) node;
|
|
|
|
|
|
|
|
APP_JUMB(tsc->tsmhandler);
|
|
|
|
JumbleExpr(jstate, (Node *) tsc->args);
|
|
|
|
JumbleExpr(jstate, (Node *) tsc->repeatable);
|
|
|
|
}
|
|
|
|
break;
|
2012-03-29 03:00:31 +02:00
|
|
|
default:
|
|
|
|
/* Only a warning, since we can stumble along anyway */
|
|
|
|
elog(WARNING, "unrecognized node type: %d",
|
|
|
|
(int) nodeTag(node));
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Record location of constant within query string of query tree
|
|
|
|
* that is currently being walked.
|
|
|
|
*/
|
|
|
|
static void
|
2012-06-10 21:20:04 +02:00
|
|
|
RecordConstLocation(pgssJumbleState *jstate, int location)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
/* -1 indicates unknown or undefined location */
|
|
|
|
if (location >= 0)
|
|
|
|
{
|
|
|
|
/* enlarge array if needed */
|
|
|
|
if (jstate->clocations_count >= jstate->clocations_buf_size)
|
|
|
|
{
|
|
|
|
jstate->clocations_buf_size *= 2;
|
|
|
|
jstate->clocations = (pgssLocationLen *)
|
|
|
|
repalloc(jstate->clocations,
|
|
|
|
jstate->clocations_buf_size *
|
|
|
|
sizeof(pgssLocationLen));
|
|
|
|
}
|
|
|
|
jstate->clocations[jstate->clocations_count].location = location;
|
|
|
|
/* initialize lengths to -1 to simplify fill_in_constant_lengths */
|
|
|
|
jstate->clocations[jstate->clocations_count].length = -1;
|
|
|
|
jstate->clocations_count++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Generate a normalized version of the query string that will be used to
|
|
|
|
* represent all similar queries.
|
|
|
|
*
|
|
|
|
* Note that the normalized representation may well vary depending on
|
|
|
|
* just which "equivalent" query is used to create the hashtable entry.
|
|
|
|
* We assume this is OK.
|
|
|
|
*
|
2017-01-14 22:17:30 +01:00
|
|
|
* If query_loc > 0, then "query" has been advanced by that much compared to
|
|
|
|
* the original string start, so we need to translate the provided locations
|
|
|
|
* to compensate. (This lets us avoid re-scanning statements before the one
|
|
|
|
* of interest, so it's worth doing.)
|
|
|
|
*
|
2012-03-29 03:00:31 +02:00
|
|
|
* *query_len_p contains the input string length, and is updated with
|
2017-03-28 02:14:36 +02:00
|
|
|
* the result string length on exit. The resulting string might be longer
|
|
|
|
* or shorter depending on what happens with replacement of constants.
|
2012-03-29 03:00:31 +02:00
|
|
|
*
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* Returns a palloc'd string.
|
2012-03-29 03:00:31 +02:00
|
|
|
*/
|
|
|
|
static char *
|
2012-06-10 21:20:04 +02:00
|
|
|
generate_normalized_query(pgssJumbleState *jstate, const char *query,
|
2017-01-14 22:17:30 +01:00
|
|
|
int query_loc, int *query_len_p, int encoding)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
char *norm_query;
|
|
|
|
int query_len = *query_len_p;
|
|
|
|
int i,
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
norm_query_buflen, /* Space allowed for norm_query */
|
2012-03-29 03:00:31 +02:00
|
|
|
len_to_wrt, /* Length (in bytes) to write */
|
|
|
|
quer_loc = 0, /* Source query byte location */
|
|
|
|
n_quer_loc = 0, /* Normalized query byte location */
|
|
|
|
last_off = 0, /* Offset from start for previous tok */
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
last_tok_len = 0; /* Length (in bytes) of that tok */
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Get constants' lengths (core system only gives us locations). Note
|
|
|
|
* this also ensures the items are sorted by location.
|
|
|
|
*/
|
2017-01-14 22:17:30 +01:00
|
|
|
fill_in_constant_lengths(jstate, query, query_loc);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
2017-03-28 02:14:36 +02:00
|
|
|
/*
|
|
|
|
* Allow for $n symbols to be longer than the constants they replace.
|
|
|
|
* Constants must take at least one byte in text form, while a $n symbol
|
|
|
|
* certainly isn't more than 11 bytes, even if n reaches INT_MAX. We
|
|
|
|
* could refine that limit based on the max value of n for the current
|
|
|
|
* query, but it hardly seems worth any extra effort to do so.
|
|
|
|
*/
|
|
|
|
norm_query_buflen = query_len + jstate->clocations_count * 10;
|
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
/* Allocate result buffer */
|
2017-03-28 02:14:36 +02:00
|
|
|
norm_query = palloc(norm_query_buflen + 1);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
for (i = 0; i < jstate->clocations_count; i++)
|
|
|
|
{
|
|
|
|
int off, /* Offset from start for cur tok */
|
|
|
|
tok_len; /* Length (in bytes) of that tok */
|
|
|
|
|
|
|
|
off = jstate->clocations[i].location;
|
2017-01-14 22:17:30 +01:00
|
|
|
/* Adjust recorded location if we're dealing with partial string */
|
|
|
|
off -= query_loc;
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
tok_len = jstate->clocations[i].length;
|
|
|
|
|
|
|
|
if (tok_len < 0)
|
|
|
|
continue; /* ignore any duplicates */
|
|
|
|
|
2014-01-28 18:34:29 +01:00
|
|
|
/* Copy next chunk (what precedes the next constant) */
|
2012-03-29 03:00:31 +02:00
|
|
|
len_to_wrt = off - last_off;
|
|
|
|
len_to_wrt -= last_tok_len;
|
|
|
|
|
|
|
|
Assert(len_to_wrt >= 0);
|
|
|
|
memcpy(norm_query + n_quer_loc, query + quer_loc, len_to_wrt);
|
|
|
|
n_quer_loc += len_to_wrt;
|
|
|
|
|
2017-03-28 02:14:36 +02:00
|
|
|
/* And insert a param symbol in place of the constant token */
|
|
|
|
n_quer_loc += sprintf(norm_query + n_quer_loc, "$%d",
|
|
|
|
i + 1 + jstate->highest_extern_param_id);
|
2012-03-29 03:00:31 +02:00
|
|
|
|
|
|
|
quer_loc = off + tok_len;
|
|
|
|
last_off = off;
|
|
|
|
last_tok_len = tok_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We've copied up until the last ignorable constant. Copy over the
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
* remaining bytes of the original query string.
|
2012-03-29 03:00:31 +02:00
|
|
|
*/
|
|
|
|
len_to_wrt = query_len - quer_loc;
|
|
|
|
|
|
|
|
Assert(len_to_wrt >= 0);
|
|
|
|
memcpy(norm_query + n_quer_loc, query + quer_loc, len_to_wrt);
|
|
|
|
n_quer_loc += len_to_wrt;
|
|
|
|
|
2017-03-28 02:14:36 +02:00
|
|
|
Assert(n_quer_loc <= norm_query_buflen);
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
norm_query[n_quer_loc] = '\0';
|
2012-03-29 03:00:31 +02:00
|
|
|
|
Keep pg_stat_statements' query texts in a file, not in shared memory.
This change allows us to eliminate the previous limit on stored query
length, and it makes the shared-memory hash table very much smaller,
allowing more statements to be tracked. (The default value of
pg_stat_statements.max is therefore increased from 1000 to 5000.)
In typical scenarios, the hash table can be large enough to hold all the
statements commonly issued by an application, so that there is little
"churn" in the set of tracked statements, and thus little need to do I/O
to the file.
To further reduce the need for I/O to the query-texts file, add a way
to retrieve all the columns of the pg_stat_statements view except for
the query text column. This is probably not of much interest for human
use but it could be exploited by programs, which will prefer using the
queryid anyway.
Ordinarily, we'd need to bump the extension version number for the latter
change. But since we already advanced pg_stat_statements' version number
from 1.1 to 1.2 in the 9.4 development cycle, it seems all right to just
redefine what 1.2 means.
Peter Geoghegan, reviewed by Pavel Stehule
2014-01-27 21:37:54 +01:00
|
|
|
*query_len_p = n_quer_loc;
|
2012-03-29 03:00:31 +02:00
|
|
|
return norm_query;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Given a valid SQL string and an array of constant-location records,
|
|
|
|
* fill in the textual lengths of those constants.
|
|
|
|
*
|
|
|
|
* The constants may use any allowed constant syntax, such as float literals,
|
|
|
|
* bit-strings, single-quoted strings and dollar-quoted strings. This is
|
|
|
|
* accomplished by using the public API for the core scanner.
|
|
|
|
*
|
|
|
|
* It is the caller's job to ensure that the string is a valid SQL statement
|
|
|
|
* with constants at the indicated locations. Since in practice the string
|
|
|
|
* has already been parsed, and the locations that the caller provides will
|
|
|
|
* have originated from within the authoritative parser, this should not be
|
|
|
|
* a problem.
|
|
|
|
*
|
|
|
|
* Duplicate constant pointers are possible, and will have their lengths
|
2014-05-06 18:12:18 +02:00
|
|
|
* marked as '-1', so that they are later ignored. (Actually, we assume the
|
2012-03-29 03:00:31 +02:00
|
|
|
* lengths were initialized as -1 to start with, and don't change them here.)
|
|
|
|
*
|
2017-01-14 22:17:30 +01:00
|
|
|
* If query_loc > 0, then "query" has been advanced by that much compared to
|
|
|
|
* the original string start, so we need to translate the provided locations
|
|
|
|
* to compensate. (This lets us avoid re-scanning statements before the one
|
|
|
|
* of interest, so it's worth doing.)
|
|
|
|
*
|
2012-03-29 03:00:31 +02:00
|
|
|
* N.B. There is an assumption that a '-' character at a Const location begins
|
|
|
|
* a negative numeric constant. This precludes there ever being another
|
|
|
|
* reason for a constant to start with a '-'.
|
|
|
|
*/
|
|
|
|
static void
|
2017-01-14 22:17:30 +01:00
|
|
|
fill_in_constant_lengths(pgssJumbleState *jstate, const char *query,
|
|
|
|
int query_loc)
|
2012-03-29 03:00:31 +02:00
|
|
|
{
|
|
|
|
pgssLocationLen *locs;
|
|
|
|
core_yyscan_t yyscanner;
|
|
|
|
core_yy_extra_type yyextra;
|
|
|
|
core_YYSTYPE yylval;
|
|
|
|
YYLTYPE yylloc;
|
|
|
|
int last_loc = -1;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Sort the records by location so that we can process them in order while
|
|
|
|
* scanning the query text.
|
|
|
|
*/
|
|
|
|
if (jstate->clocations_count > 1)
|
|
|
|
qsort(jstate->clocations, jstate->clocations_count,
|
|
|
|
sizeof(pgssLocationLen), comp_location);
|
|
|
|
locs = jstate->clocations;
|
|
|
|
|
|
|
|
/* initialize the flex scanner --- should match raw_parser() */
|
|
|
|
yyscanner = scanner_init(query,
|
|
|
|
&yyextra,
|
|
|
|
ScanKeywords,
|
|
|
|
NumScanKeywords);
|
|
|
|
|
Prevent duplicate escape-string warnings when using pg_stat_statements.
contrib/pg_stat_statements will sometimes run the core lexer a second time
on submitted statements. Formerly, if you had standard_conforming_strings
turned off, this led to sometimes getting two copies of any warnings
enabled by escape_string_warning. While this is probably no longer a big
deal in the field, it's a pain for regression testing.
To fix, change the lexer so it doesn't consult the escape_string_warning
GUC variable directly, but looks at a copy in the core_yy_extra_type state
struct. Then, pg_stat_statements can change that copy to disable warnings
while it's redoing the lexing.
It seemed like a good idea to make this happen for all three of the GUCs
consulted by the lexer, not just escape_string_warning. There's not an
immediate use-case for callers to adjust the other two AFAIK, but making
it possible is easy enough and seems like good future-proofing.
Arguably this is a bug fix, but there doesn't seem to be enough interest to
justify a back-patch. We'd not be able to back-patch exactly as-is anyway,
for fear of breaking ABI compatibility of the struct. (We could perhaps
back-patch the addition of only escape_string_warning by adding it at the
end of the struct, where there's currently alignment padding space.)
2015-01-23 00:10:47 +01:00
|
|
|
/* we don't want to re-emit any escape string warnings */
|
|
|
|
yyextra.escape_string_warning = false;
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
/* Search for each constant, in sequence */
|
|
|
|
for (i = 0; i < jstate->clocations_count; i++)
|
|
|
|
{
|
|
|
|
int loc = locs[i].location;
|
|
|
|
int tok;
|
|
|
|
|
2017-01-14 22:17:30 +01:00
|
|
|
/* Adjust recorded location if we're dealing with partial string */
|
|
|
|
loc -= query_loc;
|
|
|
|
|
2012-03-29 03:00:31 +02:00
|
|
|
Assert(loc >= 0);
|
|
|
|
|
|
|
|
if (loc <= last_loc)
|
|
|
|
continue; /* Duplicate constant, ignore */
|
|
|
|
|
|
|
|
/* Lex tokens until we find the desired constant */
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
tok = core_yylex(&yylval, &yylloc, yyscanner);
|
|
|
|
|
|
|
|
/* We should not hit end-of-string, but if we do, behave sanely */
|
|
|
|
if (tok == 0)
|
|
|
|
break; /* out of inner for-loop */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We should find the token position exactly, but if we somehow
|
|
|
|
* run past it, work with that.
|
|
|
|
*/
|
|
|
|
if (yylloc >= loc)
|
|
|
|
{
|
|
|
|
if (query[loc] == '-')
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* It's a negative value - this is the one and only case
|
|
|
|
* where we replace more than a single token.
|
|
|
|
*
|
|
|
|
* Do not compensate for the core system's special-case
|
|
|
|
* adjustment of location to that of the leading '-'
|
|
|
|
* operator in the event of a negative constant. It is
|
|
|
|
* also useful for our purposes to start from the minus
|
2014-05-06 18:12:18 +02:00
|
|
|
* symbol. In this way, queries like "select * from foo
|
2012-03-29 03:00:31 +02:00
|
|
|
* where bar = 1" and "select * from foo where bar = -2"
|
|
|
|
* will have identical normalized query strings.
|
|
|
|
*/
|
|
|
|
tok = core_yylex(&yylval, &yylloc, yyscanner);
|
|
|
|
if (tok == 0)
|
|
|
|
break; /* out of inner for-loop */
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We now rely on the assumption that flex has placed a zero
|
|
|
|
* byte after the text of the current token in scanbuf.
|
|
|
|
*/
|
|
|
|
locs[i].length = strlen(yyextra.scanbuf + loc);
|
|
|
|
break; /* out of inner for-loop */
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If we hit end-of-string, give up, leaving remaining lengths -1 */
|
|
|
|
if (tok == 0)
|
|
|
|
break;
|
|
|
|
|
|
|
|
last_loc = loc;
|
|
|
|
}
|
|
|
|
|
|
|
|
scanner_finish(yyscanner);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* comp_location: comparator for qsorting pgssLocationLen structs by location
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
comp_location(const void *a, const void *b)
|
|
|
|
{
|
|
|
|
int l = ((const pgssLocationLen *) a)->location;
|
|
|
|
int r = ((const pgssLocationLen *) b)->location;
|
|
|
|
|
|
|
|
if (l < r)
|
|
|
|
return -1;
|
|
|
|
else if (l > r)
|
|
|
|
return +1;
|
|
|
|
else
|
|
|
|
return 0;
|
|
|
|
}
|