rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/common/controldata_utils.c

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

284 lines
7.1 KiB
C
Raw Permalink Normal View History

Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
/*-------------------------------------------------------------------------
*
* controldata_utils.c
* Common code for control data file output.
*
*
Update copyright for 2024 Reported-by: Michael Paquier Discussion: https://postgr.es/m/ZZKTDPxBBMt3C0J9@paquier.xyz Backpatch-through: 12
2024-01-04 02:49:05 +01:00
* Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
* Portions Copyright (c) 1994, Regents of the University of California
*
*
* IDENTIFICATION
* src/common/controldata_utils.c
*
*-------------------------------------------------------------------------
*/
#ifndef FRONTEND
#include "postgres.h"
#else
#include "postgres_fe.h"
#endif
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
Centralize timestamp computation of control file on updates This commit moves the timestamp computation of the control file within the routine of src/common/ in charge of updating the backend's control file, which is shared by multiple frontend tools (pg_rewind, pg_checksums and pg_resetwal) and the backend itself. This change has as direct effect to update the control file's timestamp when writing the control file in pg_rewind and pg_checksums, something that is helpful to keep track of control file updates for those operations, something also tracked by the backend at startup within its logs. This part is arguably a bug, as ControlFileData->time should be updated each time a new version of the control file is written, but this is a behavior change so no backpatch is done. Author: Amul Sul Reviewed-by: Nathan Bossart, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/CAAJ_b97nd_ghRpyFV9Djf9RLXkoTbOUqnocq11WGq9TisX09Fw@mail.gmail.com
2021-11-29 05:36:13 +01:00
#include <time.h>
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#include "access/xlog_internal.h"
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
#include "catalog/pg_control.h"
#include "common/controldata_utils.h"
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#include "common/file_perm.h"
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
#ifdef FRONTEND
Move logging.h and logging.c from src/fe_utils/ to src/common/. The original placement of this module in src/fe_utils/ is ill-considered, because several src/common/ modules have dependencies on it, meaning that libpgcommon and libpgfeutils now have mutual dependencies. That makes it pointless to have distinct libraries at all. The intended design is that libpgcommon is lower-level than libpgfeutils, so only dependencies from the latter to the former are acceptable. We already have the precedent that fe_memutils and a couple of other modules in src/common/ are frontend-only, so it's not stretching anything out of whack to treat logging.c as a frontend-only module in src/common/. To the extent that such modules help provide a common frontend/backend environment for the rest of common/ to use, it's a reasonable design. (logging.c does not yet provide an ereport() emulation, but one can dream.) Hence, move these files over, and revert basically all of the build-system changes made by commit cc8d41511. There are no places that need to grow new dependencies on libpgcommon, further reinforcing the idea that this is the right solution. Discussion: https://postgr.es/m/a912ffff-f6e4-778a-c86a-cf5c47a12933@2ndquadrant.com
2019-05-14 20:19:49 +02:00
#include "common/logging.h"
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
#include "port/pg_crc32c.h"
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
#ifndef FRONTEND
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
#include "pgstat.h"
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
#include "storage/fd.h"
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
/*
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
* get_controlfile()
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
*
Fix CRC check handling in get_controlfile The previous patch broke this by returning NULL for a failed CRC check, which pg_controldata would then try to read. Fix by returning the result of the CRC check in a separate argument. Michael Paquier and myself
2016-09-28 18:00:00 +02:00
* Get controlfile values. The result is returned as a palloc'd copy of the
* control file data.
pg_ctl: Detect current standby state from pg_control pg_ctl used to determine whether a server was in standby mode by looking for a recovery.conf file. With this change, it instead looks into pg_control, which is potentially more accurate. There are also occasional discussions about removing recovery.conf, so this removes one dependency. Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2016-07-26 17:23:43 +02:00
*
Fix CRC check handling in get_controlfile The previous patch broke this by returning NULL for a failed CRC check, which pg_controldata would then try to read. Fix by returning the result of the CRC check in a separate argument. Michael Paquier and myself
2016-09-28 18:00:00 +02:00
* crc_ok_p can be used by the caller to see whether the CRC of the control
* file data is correct.
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
*/
ControlFileData *
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
get_controlfile(const char *DataDir, bool *crc_ok_p)
Expose new function get_controlfile_by_exact_path(). This works just like get_controlfile(), but expects the path to the control file rather than the path to the data directory that contains the control file. This makes more sense in cases where the caller has already constructed the path to the control file itself. Amul Sul and Robert Haas, reviewed by Michael Paquier
2024-03-13 17:06:44 +01:00
{
char ControlFilePath[MAXPGPATH];
snprintf(ControlFilePath, MAXPGPATH, "%s/global/pg_control", DataDir);
return get_controlfile_by_exact_path(ControlFilePath, crc_ok_p);
}
/*
* get_controlfile_by_exact_path()
*
* As above, but the caller specifies the path to the control file itself,
* rather than the path to the data directory.
*/
ControlFileData *
get_controlfile_by_exact_path(const char *ControlFilePath, bool *crc_ok_p)
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
{
ControlFileData *ControlFile;
int fd;
pg_crc32c crc;
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
int r;
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by retrying up to 10 times if the checksum doesn't match, until we get two reads in a row with the same bad checksum. This is not guaranteed to reach the right conclusion, but it seems very likely to. Thanks to Tom Lane for this suggestion. Various ideas for interlocking or atomicity were considered too complicated, unportable or expensive given the lack of field reports, but remain open for future reconsideration. Back-patch as far as 12. It doesn't seem like a good idea to put a heuristic change for a very rare problem into the final release of 11. Reviewed-by: Anton A. Melnikov <aamelnikov@inbox.ru> Reviewed-by: David Steele <david@pgmasters.net> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20221123014224.xisi44byq3cf5psi%40awork3.anarazel.de
2023-10-16 06:10:13 +02:00
#ifdef FRONTEND
pg_crc32c last_crc;
int retries = 0;
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Remove AssertArg and AssertState These don't offer anything over plain Assert, and their usage had already been declared obsolescent. Author: Nathan Bossart <nathandbossart@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://www.postgresql.org/message-id/20221009210148.GA900071@nathanxps13
2022-10-28 09:19:06 +02:00
Assert(crc_ok_p);
Fix CRC check handling in get_controlfile The previous patch broke this by returning NULL for a failed CRC check, which pg_controldata would then try to read. Fix by returning the result of the CRC check in a separate argument. Michael Paquier and myself
2016-09-28 18:00:00 +02:00
Assorted examples of expanded type-safer palloc/pg_malloc API This adds some uses of the new palloc/pg_malloc variants here and there as a demonstration and test. This is kept separate from the actual API patch, since the latter might be backpatched at some point. Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://www.postgresql.org/message-id/flat/bb755632-2a43-d523-36f8-a1e7a389a907@enterprisedb.com
2022-09-12 08:31:56 +02:00
ControlFile = palloc_object(ControlFileData);
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by retrying up to 10 times if the checksum doesn't match, until we get two reads in a row with the same bad checksum. This is not guaranteed to reach the right conclusion, but it seems very likely to. Thanks to Tom Lane for this suggestion. Various ideas for interlocking or atomicity were considered too complicated, unportable or expensive given the lack of field reports, but remain open for future reconsideration. Back-patch as far as 12. It doesn't seem like a good idea to put a heuristic change for a very rare problem into the final release of 11. Reviewed-by: Anton A. Melnikov <aamelnikov@inbox.ru> Reviewed-by: David Steele <david@pgmasters.net> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20221123014224.xisi44byq3cf5psi%40awork3.anarazel.de
2023-10-16 06:10:13 +02:00
#ifdef FRONTEND
INIT_CRC32C(last_crc);
retry:
#endif
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
#ifndef FRONTEND
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
if ((fd = OpenTransientFile(ControlFilePath, O_RDONLY | PG_BINARY)) == -1)
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not open file \"%s\" for reading: %m",
ControlFilePath)));
#else
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
if ((fd = open(ControlFilePath, O_RDONLY | PG_BINARY, 0)) == -1)
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not open file \"%s\" for reading: %m",
ControlFilePath);
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
r = read(fd, ControlFile, sizeof(ControlFileData));
if (r != sizeof(ControlFileData))
{
if (r < 0)
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
#ifndef FRONTEND
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read file \"%s\": %m", ControlFilePath)));
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
#else
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not read file \"%s\": %m", ControlFilePath);
Make get_controlfile() error logging consistent with src/common As originally committed, get_controlfile() used a non-standard approach to error logging. Make it consistent with the majority of error logging done in src/common. Applies to master only.
2016-03-08 00:14:20 +01:00
#endif
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
else
#ifndef FRONTEND
ereport(ERROR,
Add proper errcodes to new error messages for read() failures Those would use the default ERRCODE_INTERNAL_ERROR, but for foreseeable failures an errcode ought to be set, ERRCODE_DATA_CORRUPTED making the most sense here. While on the way, fix one errcode_for_file_access missing in origin.c since the code has been created, and remove one assignment of errno to 0 before calling read(), as this was around to fit with what was present before 811b6e36 where errno would not be set when not enough bytes are read. I have noticed the first one, and Tom has pinged me about the second one. Author: Michael Paquier Reported-by: Tom Lane Discussion: https://postgr.es/m/27265.1531925836@sss.pgh.pa.us
2018-07-23 02:37:36 +02:00
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg("could not read file \"%s\": read %d of %zu",
Rework error messages around file handling Some error messages related to file handling are using the code path context to define their state. For example, 2PC-related errors are referring to "two-phase status files", or "relation mapping file" is used for catalog-to-filenode mapping, however those prove to be difficult to translate, and are not more helpful than just referring to the path of the file being worked on. So simplify all those error messages by just referring to files with their path used. In some cases, like the manipulation of WAL segments, the context is actually helpful so those are kept. Calls to the system function read() have also been rather inconsistent with their error handling sometimes not reporting the number of bytes read, and some other code paths trying to use an errno which has not been set. The in-core functions are using a more consistent pattern with this patch, which checks for both errno if set or if an inconsistent read is happening. So as to care about pluralization when reading an unexpected number of byte(s), "could not read: read %d of %zu" is used as error message, with %d field being the output result of read() and %zu the expected size. This simplifies the work of translators with less variations of the same message. Author: Michael Paquier Reviewed-by: Álvaro Herrera Discussion: https://postgr.es/m/20180520000522.GB1603@paquier.xyz
2018-07-18 01:01:23 +02:00
ControlFilePath, r, sizeof(ControlFileData))));
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
#else
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not read file \"%s\": read %d of %zu",
ControlFilePath, r, sizeof(ControlFileData));
Fix error message on short read of pg_control Instead of saying "error: success", indicate that we got a working read but it was too short.
2018-05-18 17:52:18 +02:00
#endif
}
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
#ifndef FRONTEND
Use consistent style for checking return from system calls Use if (something() != 0) error ... instead of just if (something) error ... The latter is not incorrect, but it's a bit confusing and not the common style. Discussion: https://www.postgresql.org/message-id/flat/5de61b6b-8be9-7771-0048-860328efe027%402ndquadrant.com
2019-07-06 23:18:46 +02:00
if (CloseTransientFile(fd) != 0)
Tighten use of OpenTransientFile and CloseTransientFile This fixes two sets of issues related to the use of transient files in the backend: 1) OpenTransientFile() has been used in some code paths with read-write flags while read-only is sufficient, so switch those calls to be read-only where necessary. These have been reported by Joe Conway. 2) When opening transient files, it is up to the caller to close the file descriptors opened. In error code paths, CloseTransientFile() gets called to clean up things before issuing an error. However in normal exit paths, a lot of callers of CloseTransientFile() never actually reported errors, which could leave a file descriptor open without knowing about it. This is an issue I complained about a couple of times, but never had the courage to write and submit a patch, so here we go. Note that one frontend code path is impacted by this commit so as an error is issued when fetching control file data, making backend and frontend to be treated consistently. Reported-by: Joe Conway, Michael Paquier Author: Michael Paquier Reviewed-by: Álvaro Herrera, Georgios Kokolatos, Joe Conway Discussion: https://postgr.es/m/20190301023338.GD1348@paquier.xyz Discussion: https://postgr.es/m/c49b69ec-e2f7-ff33-4f17-0eaa4f2cef27@joeconway.com
2019-03-09 00:50:55 +01:00
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not close file \"%s\": %m",
ControlFilePath)));
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
#else
Use consistent style for checking return from system calls Use if (something() != 0) error ... instead of just if (something) error ... The latter is not incorrect, but it's a bit confusing and not the common style. Discussion: https://www.postgresql.org/message-id/flat/5de61b6b-8be9-7771-0048-860328efe027%402ndquadrant.com
2019-07-06 23:18:46 +02:00
if (close(fd) != 0)
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not close file \"%s\": %m", ControlFilePath);
Make get_controlfile not leak file descriptors When backend functions were added to expose controldata via SQL, reading of pg_control was consolidated under src/common so that both frontend and backend could share the same code. That move from frontend-only to shared frontend-backend failed to recognize the risk (and coding standards violation) of using a bare open(). In particular, it risked leaking file descriptors if transient errors occurred while reading the file. Fix that by using OpenTransientFile() instead in the backend case, which is purpose-built for this type of usage. Since there have been no complaints from the field, and an intermittent failure low risk, no backpatch. Hard failure would of course be bad, but in that case these functions are probably the least of your worries. Author: Joe Conway Reviewed-By: Michael Paquier Reported by: Michael Paquier Discussion: https://postgr.es/m/20190227074728.GA15710@paquier.xyz
2019-02-28 21:57:40 +01:00
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
/* Check the CRC. */
INIT_CRC32C(crc);
COMP_CRC32C(crc,
(char *) ControlFile,
offsetof(ControlFileData, crc));
FIN_CRC32C(crc);
Fix CRC check handling in get_controlfile The previous patch broke this by returning NULL for a failed CRC check, which pg_controldata would then try to read. Fix by returning the result of the CRC check in a separate argument. Michael Paquier and myself
2016-09-28 18:00:00 +02:00
*crc_ok_p = EQ_CRC32C(crc, ControlFile->crc);
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by retrying up to 10 times if the checksum doesn't match, until we get two reads in a row with the same bad checksum. This is not guaranteed to reach the right conclusion, but it seems very likely to. Thanks to Tom Lane for this suggestion. Various ideas for interlocking or atomicity were considered too complicated, unportable or expensive given the lack of field reports, but remain open for future reconsideration. Back-patch as far as 12. It doesn't seem like a good idea to put a heuristic change for a very rare problem into the final release of 11. Reviewed-by: Anton A. Melnikov <aamelnikov@inbox.ru> Reviewed-by: David Steele <david@pgmasters.net> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20221123014224.xisi44byq3cf5psi%40awork3.anarazel.de
2023-10-16 06:10:13 +02:00
#ifdef FRONTEND
/*
* If the server was writing at the same time, it is possible that we read
* partially updated contents on some systems. If the CRC doesn't match,
* retry a limited number of times until we compute the same bad CRC twice
* in a row with a short sleep in between. Then the failure is unlikely
* to be due to a concurrent write.
*/
if (!*crc_ok_p &&
(retries == 0 || !EQ_CRC32C(crc, last_crc)) &&
retries < 10)
{
retries++;
last_crc = crc;
pg_usleep(10000);
goto retry;
}
#endif
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
/* Make sure the control file is valid byte order. */
if (ControlFile->pg_control_version % 65536 == 0 &&
ControlFile->pg_control_version / 65536 != 0)
#ifndef FRONTEND
elog(ERROR, _("byte ordering mismatch"));
#else
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
pg_log_warning("possible byte ordering mismatch\n"
"The byte ordering used to store the pg_control file might not match the one\n"
"used by this program. In that case the results below would be incorrect, and\n"
"the PostgreSQL installation would be incompatible with this data directory.");
Expose control file data via SQL accessible functions. Add four new SQL accessible functions: pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() which expose a subset of the control file data. Along the way move the code to read and validate the control file to src/common, where it can be shared by the new backend functions and the original pg_controldata frontend program. Patch by me, significant input, testing, and review by Michael Paquier.
2016-03-05 20:10:19 +01:00
#endif
return ControlFile;
}
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
/*
* update_controlfile()
*
* Update controlfile values with the contents given by caller. The
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
* contents to write are included in "ControlFile". "do_sync" can be
* optionally used to flush the updated control file. Note that it is up
* to the caller to properly lock ControlFileLock when calling this
* routine in the backend.
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
*/
void
Unified logging system for command-line programs This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
update_controlfile(const char *DataDir,
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
ControlFileData *ControlFile, bool do_sync)
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
{
int fd;
char buffer[PG_CONTROL_FILE_SIZE];
char ControlFilePath[MAXPGPATH];
Centralize timestamp computation of control file on updates This commit moves the timestamp computation of the control file within the routine of src/common/ in charge of updating the backend's control file, which is shared by multiple frontend tools (pg_rewind, pg_checksums and pg_resetwal) and the backend itself. This change has as direct effect to update the control file's timestamp when writing the control file in pg_rewind and pg_checksums, something that is helpful to keep track of control file updates for those operations, something also tracked by the backend at startup within its logs. This part is arguably a bug, as ControlFileData->time should be updated each time a new version of the control file is written, but this is a behavior change so no backpatch is done. Author: Amul Sul Reviewed-by: Nathan Bossart, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/CAAJ_b97nd_ghRpyFV9Djf9RLXkoTbOUqnocq11WGq9TisX09Fw@mail.gmail.com
2021-11-29 05:36:13 +01:00
/* Update timestamp */
ControlFile->time = (pg_time_t) time(NULL);
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
/* Recalculate CRC of control file */
INIT_CRC32C(ControlFile->crc);
COMP_CRC32C(ControlFile->crc,
(char *) ControlFile,
offsetof(ControlFileData, crc));
FIN_CRC32C(ControlFile->crc);
/*
* Write out PG_CONTROL_FILE_SIZE bytes into pg_control by zero-padding
* the excess over sizeof(ControlFileData), to avoid premature EOF related
* errors when reading it.
*/
memset(buffer, 0, PG_CONTROL_FILE_SIZE);
memcpy(buffer, ControlFile, sizeof(ControlFileData));
snprintf(ControlFilePath, sizeof(ControlFilePath), "%s/%s", DataDir, XLOG_CONTROL_FILE);
#ifndef FRONTEND
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
/*
* All errors issue a PANIC, so no need to use OpenTransientFile() and to
* worry about file descriptor leaks.
*/
if ((fd = BasicOpenFile(ControlFilePath, O_RDWR | PG_BINARY)) < 0)
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
ereport(PANIC,
(errcode_for_file_access(),
errmsg("could not open file \"%s\": %m",
ControlFilePath)));
#else
if ((fd = open(ControlFilePath, O_WRONLY | PG_BINARY,
pg_file_create_mode)) == -1)
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not open file \"%s\": %m", ControlFilePath);
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#endif
errno = 0;
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
#ifndef FRONTEND
pgstat_report_wait_start(WAIT_EVENT_CONTROL_FILE_WRITE_UPDATE);
#endif
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
if (write(fd, buffer, PG_CONTROL_FILE_SIZE) != PG_CONTROL_FILE_SIZE)
{
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
errno = ENOSPC;
#ifndef FRONTEND
ereport(PANIC,
(errcode_for_file_access(),
errmsg("could not write file \"%s\": %m",
ControlFilePath)));
#else
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not write file \"%s\": %m", ControlFilePath);
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#endif
}
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
#ifndef FRONTEND
pgstat_report_wait_end();
#endif
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
if (do_sync)
{
#ifndef FRONTEND
pgstat_report_wait_start(WAIT_EVENT_CONTROL_FILE_SYNC_UPDATE);
if (pg_fsync(fd) != 0)
ereport(PANIC,
(errcode_for_file_access(),
errmsg("could not fsync file \"%s\": %m",
ControlFilePath)));
pgstat_report_wait_end();
#else
if (fsync(fd) != 0)
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not fsync file \"%s\": %m", ControlFilePath);
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
#endif
}
Use consistent style for checking return from system calls Use if (something() != 0) error ... instead of just if (something) error ... The latter is not incorrect, but it's a bit confusing and not the common style. Discussion: https://www.postgresql.org/message-id/flat/5de61b6b-8be9-7771-0048-860328efe027%402ndquadrant.com
2019-07-06 23:18:46 +02:00
if (close(fd) != 0)
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
{
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#ifndef FRONTEND
ereport(PANIC,
(errcode_for_file_access(),
errmsg("could not close file \"%s\": %m",
ControlFilePath)));
#else
Improve frontend error logging style. Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-04-08 20:55:14 +02:00
pg_fatal("could not close file \"%s\": %m", ControlFilePath);
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
#endif
Refactor more code logic to update the control file ce6afc6 has begun the refactoring work by plugging pg_rewind into a central routine to update the control file, and left around two extra copies, with one in xlog.c for the backend and one in pg_resetwal.c. By adding an extra option to the central routine in controldata_utils.c to control if a flush of the control file needs to be done, it is proving to be straight-forward to make xlog.c and pg_resetwal.c use the central code path at the condition of moving the wait event tracking there. Hence, this allows to have only one central code path to update the control file, shaving the code from the duplicates. This refactoring actually fixes a problem in pg_resetwal. Previously, the control file was first removed before being recreated. So if a crash happened between the moment the file was removed and the moment the file was created, then it would have been possible to not have a control file anymore in the database folder. Author: Fabien Coelho Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/alpine.DEB.2.21.1903170935210.2506@lancre
2019-03-18 04:59:35 +01:00
}
Add routine able to update the control file to src/common/ This adds a new routine to src/common/ which is compatible with both the frontend and backend code, able to update the control file's contents. This is now getting used only by pg_rewind, but some upcoming patches which add more control on checksums for offline instances will make use of it. This could also get used more by the backend as xlog.c has its own flavor of the same logic with some wait events and an additional flush phase before closing the opened file descriptor, but this is let as separate work. Author: Michael Banck, Michael Paquier Reviewed-by: Fabien Coelho, Sergei Kornilov Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-12 02:03:33 +01:00
}