From a70bce43fbceab6bf0b49498a064c4e87cdbf72f Mon Sep 17 00:00:00 2001 From: Robert Haas Date: Mon, 16 Oct 2023 12:57:39 -0400 Subject: [PATCH] Update the documentation on recovering from (M)XID exhaustion. The old documentation encourages entering single-user mode for no reason, which is a bad plan in most cases. Instead, discourage users from doing that, and explain the limited cases in which it may be desirable. The old documentation claims that running VACUUM as anyone but the superuser can't possibly work, which is not really true, because it might be that some other user has enough permissions to VACUUM all the tables that matter. Weaken the language just a bit. The old documentation claims that you can't run any commands when near XID exhaustion, which is false because you can still run commands that don't require an XID, like a SELECT without a locking clause. The old documentation doesn't clearly explain that it's a good idea to get rid of prepared transactons, long-running transactions, and replication slots that are preventing (M)XID horizon advancement. Spell out the steps to do that. Also, discourage the use of VACUUM FULL and VACUUM FREEZE in this type of scenario. Back-patch to v14. Much of this is good advice on all supported versions, but before 60f1f09ff44308667ef6c72fbafd68235e55ae27 the chances of VACUUM failing in multi-user mode were much higher. Alexander Alekseev, John Naylor, Robert Haas, reviewed at various times by Peter Geoghegan, Hannu Krosing, and Andres Freund. Discussion: http://postgr.es/m/CA+TgmoYtsUDrzaHcmjFhLzTk1VEv29mO_u-MT+XWHrBJ_4nD8A@mail.gmail.com --- doc/src/sgml/maintenance.sgml | 112 +++++++++++++++++++++++++++++----- 1 file changed, 97 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index 3018d8f64c..66f2c6a02e 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -660,29 +660,79 @@ HINT: To avoid a database shutdown, execute a database-wide VACUUM in that data (A manual VACUUM should fix the problem, as suggested by the - hint; but note that the VACUUM must be performed by a - superuser, else it will fail to process system catalogs and thus not - be able to advance the database's datfrozenxid.) - If these warnings are - ignored, the system will shut down and refuse to start any new - transactions once there are fewer than three million transactions left - until wraparound: + hint; but note that the VACUUM should be performed by a + superuser, else it will fail to process system catalogs, which prevent it from + being able to advance the database's datfrozenxid.) + If these warnings are ignored, the system will refuse to assign new XIDs once + there are fewer than three million transactions left until wraparound: ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb" HINT: Stop the postmaster and vacuum that database in single-user mode. - The three-million-transaction safety margin exists to let the - administrator recover without data loss, by manually executing the - required VACUUM commands. However, since the system will not - execute commands once it has gone into the safety shutdown mode, - the only way to do this is to stop the server and start the server in single-user - mode to execute VACUUM. The shutdown mode is not enforced - in single-user mode. See the reference - page for details about using single-user mode. + In this condition any transactions already in progress can continue, + but only read-only transactions can be started. Operations that + modify database records or truncate relations will fail. + The VACUUM command can still be run normally. + Contrary to what the hint states, it is not necessary or desirable to stop the + postmaster or enter single user-mode in order to restore normal operation. + Instead, follow these steps: + + + + Resolve old prepared transactions. You can find these by checking + pg_prepared_xacts for rows where + age(transactionid) is large. Such transactions should be + committed or rolled back. + + + End long-running open transactions. You can find these by checking + pg_stat_activity for rows where + age(backend_xid) or age(backend_xmin) is + large. Such transactions should be committed or rolled back, or the session + can be terminated using pg_terminate_backend. + + + Drop any old replication slots. Use + pg_stat_replication to + find slots where age(xmin) or age(catalog_xmin) + is large. In many cases, such slots were created for replication to servers that no + longer exist, or that have been down for a long time. If you drop a slot for a server + that still exists and might still try to connect to that slot, that replica may + need to be rebuilt. + + + Execute VACUUM in the target database. A database-wide + VACUUM is simplest; to reduce the time required, it as also possible + to issue manual VACUUM commands on the tables where + relminxid is oldest. Do not use VACUUM FULL + in this scenario, because it requires an XID and will therefore fail, except in super-user + mode, where it will instead consume an XID and thus increase the risk of transaction ID + wraparound. Do not use VACUUM FREEZE either, because it will do + more than the minimum amount of work required to restore normal operation. + + + Once normal operation is restored, ensure that autovacuum is properly configured + in the target database in order to avoid future problems. + + + + + In earlier versions, it was sometimes necessary to stop the postmaster and + VACUUM the database in a single-user mode. In typical scenarios, this + is no longer necessary, and should be avoided whenever possible, since it involves taking + the system down. It is also riskier, since it disables transaction ID wraparound safeguards + that are designed to prevent data loss. The only reason to use single-user mode in this + scenario is if you wish to TRUNCATE or DROP unneeded + tables to avoid needing to VACUUM them. The three-million-transaction + safety margin exists to let the administrator do this. See the + reference page for details about using single-user mode. + + + Multixacts and Wraparound @@ -747,6 +797,38 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. have the oldest multixact-age. Both of these kinds of aggressive scans will occur even if autovacuum is nominally disabled. + + + Similar to the XID case, if autovacuum fails to clear old MXIDs from a table, the + system will begin to emit warning messages when the database's oldest MXIDs reach forty + million transactions from the wraparound point. And, just as an the XID case, if these + warnings are ignored, the system will refuse to generate new MXIDs once there are fewer + than three million left until wraparound. + + + + Normal operation when MXIDs are exhausted can be restored in much the same way as + when XIDs are exhausted. Follow the same steps in the previous section, but with the + following differences: + + + + Running transactions and prepared transactions can be ignored if there + is no chance that they might appear in a multixact. + + + MXID information is not directly visible in system views such as + pg_stat_activity; however, looking for old XIDs is still a good + way of determining which transactions are causing MXID wraparound problems. + + + XID exhaustion will block all write transactions, but MXID exhaustion will + only block a subset of write transactions, specifically those that involve + row locks that require an MXID. + + + +