From b3abca68106d518ce5d3c0d9a1e0ec02a647ceda Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Thu, 7 Apr 2022 21:35:35 -0700 Subject: [PATCH] pgstat: Update docs to match the shared memory stats reality. This includes removing documentation for stats_temp_directory, adding documentation for stats_fetch_consistency, rephrasing references to the stats collector and documenting that starting a cleanly shut down standby will not remove stats anymore. The latter point might require further wordsmithing, it wasn't easy to adjust some of the existing content. Author: Kyotaro Horiguchi Author: Andres Freund Reviewed-By: Thomas Munro Reviewed-By: Justin Pryzby Reviewed-By: "David G. Johnston" Reviewed-By: Lukas Fittl Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- doc/src/sgml/backup.sgml | 2 - doc/src/sgml/catalogs.sgml | 6 +- doc/src/sgml/config.sgml | 49 ++++++---- doc/src/sgml/func.sgml | 4 +- doc/src/sgml/glossary.sgml | 38 ++++---- doc/src/sgml/high-availability.sgml | 13 +-- doc/src/sgml/maintenance.sgml | 2 +- doc/src/sgml/monitoring.sgml | 142 +++++++++++++++------------- doc/src/sgml/ref/pg_dump.sgml | 2 +- 9 files changed, 139 insertions(+), 119 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index fc2ec68758..6812e9ec66 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1036,8 +1036,6 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ (but not the directories themselves) can be omitted from the backup as they will be initialized on postmaster startup. - If is set and is under the data - directory then the contents of that directory can also be omitted. diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index e8f850a9c0..c1c89a95c6 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9593,9 +9593,9 @@ SCRAM-SHA-256$<iteration count>:&l lists the system views described here. More detailed documentation of each view follows below. - There are some additional views that provide access to the results of - the statistics collector; they are described in . + There are some additional views that provide access to accumulated + statistics; they are described in + . diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c2c7a95a82..6e3e27bed7 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7885,15 +7885,15 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Run-time Statistics - - Query and Index Statistics Collector + + Cumulative Query and Index Statistics - These parameters control server-wide statistics collection features. - When statistics collection is enabled, the data that is produced can be - accessed via the pg_stat and - pg_statio family of system views. - Refer to for more information. + These parameters control the server-wide cumulative statistics system. + When enabled, the data that is collected can be accessed via the + pg_stat and pg_statio + family of system views. Refer to for more + information. @@ -8031,22 +8031,37 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - - stats_temp_directory (string) + + stats_fetch_consistency (enum) - stats_temp_directory configuration parameter + stats_fetch_consistency configuration parameter - Sets the directory to store temporary statistics data in. This can be - a path relative to the data directory or an absolute path. The default - is pg_stat_tmp. Pointing this at a RAM-based - file system will decrease physical I/O requirements and can lead to - improved performance. - This parameter can only be set in the postgresql.conf - file or on the server command line. + Determines the behavior when cumulative statistics are accessed + multiple times within a transaction. When set to + none, each access re-fetches counters from shared + memory. When set to cache, the first access to + statistics for an object caches those statistics until the end of the + transaction unless pg_stat_clear_snapshot() is + called. When set to snapshot, the first statistics + access caches all statistics accessible in the current database, until + the end of the transaction unless + pg_stat_clear_snapshot() is called. The default + is cache. + + + none is most suitable for monitoring systems. If + values are only accessed once, it is the most + efficient. cache ensures repeat accesses yield the + same values, which is important for queries involving + e.g. self-joins. snapshot can be useful when + interactively inspecting statistics, but has higher overhead, + particularly if many database objects exist. + + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 569c78e792..29c4376886 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28019,8 +28019,8 @@ SELECT collation for ('foo' COLLATE "de_DE"); Requests to log the memory contexts of the backend with the specified process ID. This function can send the request to - backends and auxiliary processes except logger and statistics - collector. These memory contexts will be logged at + backends and auxiliary processes except logger. These memory contexts + will be logged at LOG message level. They will appear in the server log based on the log configuration set (See for more information), diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index 1835d0e65a..b187506d5e 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -136,9 +136,9 @@ The auxiliary processes consist of the autovacuum launcher (but not the autovacuum workers), @@ -146,7 +146,6 @@ the checkpointer, the logger, the startup process, - the statistics collector, the WAL archiver, the WAL receiver (but not the WAL senders), @@ -434,6 +433,21 @@ + + Cumulative Statistics System + + + A system which, if enabled, accumulates statistical information + about the instance's + activities. + + + For more information, see + . + + + + Data area @@ -1563,22 +1577,6 @@ - - Stats collector (process) - - - An auxiliary process - which, if enabled, receives statistical information - about the instance's - activities. - - - For more information, see - . - - - - System catalog diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 3247e05666..b0a653373d 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -2227,12 +2227,13 @@ HINT: You can then restart the server after making the necessary configuration - The statistics collector is active during recovery. All scans, reads, blocks, - index usage, etc., will be recorded normally on the standby. Replayed - actions will not duplicate their effects on primary, so replaying an - insert will not increment the Inserts column of pg_stat_user_tables. - The stats file is deleted at the start of recovery, so stats from primary - and standby will differ; this is considered a feature, not a bug. + The cumulative statistics system is active during recovery. All scans, + reads, blocks, index usage, etc., will be recorded normally on the + standby. However, WAL replay will not increment relation and database + specific counters. I.e. replay will not increment pg_stat_all_tables + columns (like n_tup_ins), nor will reads or writes performed by the + startup process be tracked in the pg_statio views, nor will associated + pg_stat_database columns be incremented. diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index 2e09fee5ae..a209a63304 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -839,7 +839,7 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac it may be beneficial to lower the table's as this may allow tuples to be frozen by earlier vacuums. The number of obsolete tuples and - the number of inserted tuples are obtained from the statistics collector; + the number of inserted tuples are obtained from the cumulative statistics system; it is a semi-accurate count updated by each UPDATE, DELETE and INSERT operation. (It is only semi-accurate because some information might be lost under heavy diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 8717df060e..2f44113caa 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -22,7 +22,7 @@ Several tools are available for monitoring database activity and analyzing performance. Most of this chapter is devoted to describing - PostgreSQL's statistics collector, + PostgreSQL's cumulative statistics system, but one should not neglect regular Unix monitoring programs such as ps, top, iostat, and vmstat. Also, once one has identified a @@ -53,7 +53,6 @@ postgres 15554 0.0 0.0 57536 1184 ? Ss 18:02 0:00 postgres: back postgres 15555 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: checkpointer postgres 15556 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: walwriter postgres 15557 0.0 0.0 58504 2244 ? Ss 18:02 0:00 postgres: autovacuum launcher -postgres 15558 0.0 0.0 17512 1068 ? Ss 18:02 0:00 postgres: stats collector postgres 15582 0.0 0.0 58772 3080 ? Ss 18:04 0:00 postgres: joe runbug 127.0.0.1 idle postgres 15606 0.0 0.0 58772 3052 ? Ss 18:07 0:00 postgres: tgl regression [local] SELECT waiting postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl regression [local] idle in transaction @@ -63,11 +62,10 @@ postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl platforms, as do the details of what is shown. This example is from a recent Linux system.) The first process listed here is the primary server process. The command arguments - shown for it are the same ones used when it was launched. The next five + shown for it are the same ones used when it was launched. The next four processes are background worker processes automatically launched by the - primary process. (The stats collector process will not be present - if you have set the system not to start the statistics collector; likewise - the autovacuum launcher process can be disabled.) + primary process. (The autovacuum launcher process will not + be present if you have set the system not to run autovacuum.) Each of the remaining processes is a server process handling one client connection. Each such process sets its command line display in the form @@ -130,20 +128,20 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser - The Statistics Collector + The Cumulative Statistics System statistics - PostgreSQL's statistics collector - is a subsystem that supports collection and reporting of information about - server activity. Presently, the collector can count accesses to tables - and indexes in both disk-block and individual-row terms. It also tracks - the total number of rows in each table, and information about vacuum and - analyze actions for each table. It can also count calls to user-defined - functions and the total time spent in each one. + PostgreSQL's cumulative statistics + system supports collection and reporting of information about + server activity. Presently, accesses to tables and indexes in both + disk-block and individual-row terms are counted. The total number of rows + in each table, and information about vacuum and analyze actions for each + table are also counted. If enabled, calls to user-defined functions and + the total time spent in each one are counted as well. @@ -151,7 +149,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser information about exactly what is going on in the system right now, such as the exact command currently being executed by other server processes, and which other connections exist in the system. This facility is independent - of the collector process. + of the cumulative statistics system. @@ -172,7 +170,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser The parameter controls whether - statistics are collected about table and index accesses. + cumulative statistics are collected about table and index accesses. @@ -201,18 +199,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser - The statistics collector transmits the collected information to other - PostgreSQL processes through temporary files. - These files are stored in the directory named by the - parameter, - pg_stat_tmp by default. - For better performance, stats_temp_directory can be - pointed at a RAM-based file system, decreasing physical I/O requirements. - When the server shuts down cleanly, a permanent copy of the statistics - data is stored in the pg_stat subdirectory, so that - statistics can be retained across server restarts. When recovery is - performed at server start (e.g., after immediate shutdown, server crash, - and point-in-time recovery), all statistics counters are reset. + Cumulative statistics are collected in shared memory. Every + PostgreSQL process collects statistics locally + then updates the shared data at appropriate intervals. When a server, + including a physical replica, shuts down cleanly, a permanent copy of the + statistics data is stored in the pg_stat subdirectory, + so that statistics can be retained across server restarts. In contrast, + when starting from an unclean shutdown (e.g., after an immediate shutdown, + a server crash, starting from a base backup, and point-in-time recovery), + all statistics counters are reset. @@ -225,48 +220,58 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser linkend="monitoring-stats-dynamic-views-table"/>, are available to show the current state of the system. There are also several other views, listed in , available to show the results - of statistics collection. Alternatively, one can - build custom views using the underlying statistics functions, as discussed - in . + linkend="monitoring-stats-views-table"/>, available to show the accumulated + statistics. Alternatively, one can + build custom views using the underlying cumulative statistics functions, as + discussed in . - When using the statistics to monitor collected data, it is important - to realize that the information does not update instantaneously. - Each individual server process transmits new statistical counts to - the collector just before going idle; so a query or transaction still in - progress does not affect the displayed totals. Also, the collector itself - emits a new report at most once per PGSTAT_STAT_INTERVAL - milliseconds (500 ms unless altered while building the server). So the - displayed information lags behind actual activity. However, current-query - information collected by track_activities is - always up-to-date. + When using the cumulative statistics views and functions to monitor + collected data, it is important to realize that the information does not + update instantaneously. Each individual server process flushes out + accumulated statistics to shared memory just before going idle, but not + more frequently than once per PGSTAT_MIN_INTERVAL + milliseconds (1 second unless altered while building the server); so a + query or transaction still in progress does not affect the displayed totals + and the displayed information lags behind actual activity. However, + current-query information collected by track_activities + is always up-to-date. Another important point is that when a server process is asked to display - any of these statistics, it first fetches the most recent report emitted by - the collector process and then continues to use this snapshot for all - statistical views and functions until the end of its current transaction. - So the statistics will show static information as long as you continue the - current transaction. Similarly, information about the current queries of - all sessions is collected when any such information is first requested - within a transaction, and the same information will be displayed throughout - the transaction. - This is a feature, not a bug, because it allows you to perform several - queries on the statistics and correlate the results without worrying that - the numbers are changing underneath you. But if you want to see new - results with each query, be sure to do the queries outside any transaction - block. Alternatively, you can invoke - pg_stat_clear_snapshot(), which will discard the - current transaction's statistics snapshot (if any). The next use of - statistical information will cause a new snapshot to be fetched. + any of the accumulated statistics, accessed values are cached until the end + of its current transaction in the default configuration. So the statistics + will show static information as long as you continue the current + transaction. Similarly, information about the current queries of all + sessions is collected when any such information is first requested within a + transaction, and the same information will be displayed throughout the + transaction. This is a feature, not a bug, because it allows you to perform + several queries on the statistics and correlate the results without + worrying that the numbers are changing underneath you. + + When analyzing statistics interactively, or with expensive queries, the + time delta between accesses to individual statistics can lead to + significant skew in the cached statistics. To minimize skew, + stats_fetch_consistency can be set to + snapshot, at the price of increased memory usage for + caching not-needed statistics data. Conversely, if it's known that + statistics are only accessed once, caching accessed statistics is + unnecessary and can be avoided by setting + stats_fetch_consistency to none. + + You can invoke pg_stat_clear_snapshot() to discard the + current transaction's statistics snapshot or cached values (if any). The + next use of statistical information will (when in snapshot mode) cause a + new snapshot to be built or (when in cache mode) accessed statistics to be + cached. - A transaction can also see its own statistics (as yet untransmitted to the - collector) in the views pg_stat_xact_all_tables, + A transaction can also see its own statistics (not yet flushed out to the + shared memory statistics) in the views + pg_stat_xact_all_tables, pg_stat_xact_sys_tables, pg_stat_xact_user_tables, and pg_stat_xact_user_functions. These numbers do not act as @@ -663,7 +668,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser kernel's I/O cache, and might therefore still be fetched without requiring a physical read. Users interested in obtaining more detailed information on PostgreSQL I/O behavior are - advised to use the PostgreSQL statistics collector + advised to use the PostgreSQL statistics views in combination with operating system utilities that allow insight into the kernel's handling of I/O. @@ -5171,8 +5176,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i - Additional functions related to statistics collection are listed in . + Additional functions related to the cumulative statistics system are listed + in . @@ -5228,7 +5233,10 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i Returns the timestamp of the current statistics snapshot, or NULL if - no statistics snapshot has been taken. + no statistics snapshot has been taken. A snapshot is taken the first + time cumulative statistics are accessed in a transaction if + stats_fetch_consistency is set to + snapshot @@ -5241,7 +5249,7 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i void - Discards the current statistics snapshot. + Discards the current statistics snapshot or cached information. @@ -6405,8 +6413,8 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, VACUUM is performing final cleanup. During this phase, VACUUM will vacuum the free space map, update statistics - in pg_class, and report statistics to the statistics - collector. When this phase is completed, VACUUM will end. + in pg_class, and report statistics to the cumulative + statistics system. When this phase is completed, VACUUM will end. diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index 723b2a1a66..c946755737 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -1329,7 +1329,7 @@ PostgreSQL documentation The database activity of pg_dump is - normally collected by the statistics collector. If this is + normally collected by the cumulative statistics system. If this is undesirable, you can set parameter track_counts to false via PGOPTIONS or the ALTER USER command.