1996-08-28 03:59:28 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
|
*
|
1999-02-14 00:22:53 +01:00
|
|
|
* fd.h
|
1997-09-07 07:04:48 +02:00
|
|
|
* Virtual file descriptor definitions.
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
|
|
|
|
*
|
2017-01-03 19:48:53 +01:00
|
|
|
* Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
|
2000-01-26 06:58:53 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/include/storage/fd.h
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
|
*/
|
1999-10-13 17:02:32 +02:00
|
|
|
|
1996-08-28 03:59:28 +02:00
|
|
|
/*
|
|
|
|
|
* calls:
|
1997-09-07 07:04:48 +02:00
|
|
|
*
|
2004-05-31 05:48:10 +02:00
|
|
|
* File {Close, Read, Write, Seek, Tell, Sync}
|
Add OpenTransientFile, with automatic cleanup at end-of-xact.
Files opened with BasicOpenFile or PathNameOpenFile are not automatically
cleaned up on error. That puts unnecessary burden on callers that only want
to keep the file open for a short time. There is AllocateFile, but that
returns a buffered FILE * stream, which in many cases is not the nicest API
to work with. So add function called OpenTransientFile, which returns a
unbuffered fd that's cleaned up like the FILE* returned by AllocateFile().
This plugs a few rare fd leaks in error cases:
1. copy_file() - fixed by by using OpenTransientFile instead of BasicOpenFile
2. XLogFileInit() - fixed by adding close() calls to the error cases. Can't
use OpenTransientFile here because the fd is supposed to persist over
transaction boundaries.
3. lo_import/lo_export - fixed by using OpenTransientFile instead of
PathNameOpenFile.
In addition to plugging those leaks, this replaces many BasicOpenFile() calls
with OpenTransientFile() that were not leaking, because the code meticulously
closed the file on error. That wasn't strictly necessary, but IMHO it's good
for robustness.
The same leaks exist in older versions, but given the rarity of the issues,
I'm not backpatching this. Not yet, anyway - it might be good to backpatch
later, after this mechanism has had some more testing in master branch.
2012-11-27 09:25:50 +01:00
|
|
|
* {Path Name Open, Allocate, Free} File
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
|
|
|
|
* These are NOT JUST RENAMINGS OF THE UNIX ROUTINES.
|
1999-05-09 02:52:08 +02:00
|
|
|
* Use them for all file activity...
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
1997-09-07 07:04:48 +02:00
|
|
|
* File fd;
|
Add OpenTransientFile, with automatic cleanup at end-of-xact.
Files opened with BasicOpenFile or PathNameOpenFile are not automatically
cleaned up on error. That puts unnecessary burden on callers that only want
to keep the file open for a short time. There is AllocateFile, but that
returns a buffered FILE * stream, which in many cases is not the nicest API
to work with. So add function called OpenTransientFile, which returns a
unbuffered fd that's cleaned up like the FILE* returned by AllocateFile().
This plugs a few rare fd leaks in error cases:
1. copy_file() - fixed by by using OpenTransientFile instead of BasicOpenFile
2. XLogFileInit() - fixed by adding close() calls to the error cases. Can't
use OpenTransientFile here because the fd is supposed to persist over
transaction boundaries.
3. lo_import/lo_export - fixed by using OpenTransientFile instead of
PathNameOpenFile.
In addition to plugging those leaks, this replaces many BasicOpenFile() calls
with OpenTransientFile() that were not leaking, because the code meticulously
closed the file on error. That wasn't strictly necessary, but IMHO it's good
for robustness.
The same leaks exist in older versions, but given the rarity of the issues,
I'm not backpatching this. Not yet, anyway - it might be good to backpatch
later, after this mechanism has had some more testing in master branch.
2012-11-27 09:25:50 +01:00
|
|
|
* fd = PathNameOpenFile("foo", O_RDONLY, 0600);
|
1996-08-28 03:59:28 +02:00
|
|
|
*
|
1997-09-07 07:04:48 +02:00
|
|
|
* AllocateFile();
|
|
|
|
|
* FreeFile();
|
1999-05-09 02:52:08 +02:00
|
|
|
*
|
|
|
|
|
* Use AllocateFile, not fopen, if you need a stdio file (FILE*); then
|
|
|
|
|
* use FreeFile, not fclose, to close it. AVOID using stdio for files
|
|
|
|
|
* that you intend to hold open for any length of time, since there is
|
|
|
|
|
* no way for them to share kernel file descriptors with other files.
|
2004-02-24 00:03:10 +01:00
|
|
|
*
|
|
|
|
|
* Likewise, use AllocateDir/FreeDir, not opendir/closedir, to allocate
|
Add OpenTransientFile, with automatic cleanup at end-of-xact.
Files opened with BasicOpenFile or PathNameOpenFile are not automatically
cleaned up on error. That puts unnecessary burden on callers that only want
to keep the file open for a short time. There is AllocateFile, but that
returns a buffered FILE * stream, which in many cases is not the nicest API
to work with. So add function called OpenTransientFile, which returns a
unbuffered fd that's cleaned up like the FILE* returned by AllocateFile().
This plugs a few rare fd leaks in error cases:
1. copy_file() - fixed by by using OpenTransientFile instead of BasicOpenFile
2. XLogFileInit() - fixed by adding close() calls to the error cases. Can't
use OpenTransientFile here because the fd is supposed to persist over
transaction boundaries.
3. lo_import/lo_export - fixed by using OpenTransientFile instead of
PathNameOpenFile.
In addition to plugging those leaks, this replaces many BasicOpenFile() calls
with OpenTransientFile() that were not leaking, because the code meticulously
closed the file on error. That wasn't strictly necessary, but IMHO it's good
for robustness.
The same leaks exist in older versions, but given the rarity of the issues,
I'm not backpatching this. Not yet, anyway - it might be good to backpatch
later, after this mechanism has had some more testing in master branch.
2012-11-27 09:25:50 +01:00
|
|
|
* open directories (DIR*), and OpenTransientFile/CloseTransient File for an
|
|
|
|
|
* unbuffered file descriptor.
|
1996-08-28 03:59:28 +02:00
|
|
|
*/
|
1997-09-07 07:04:48 +02:00
|
|
|
#ifndef FD_H
|
1996-08-28 03:59:28 +02:00
|
|
|
#define FD_H
|
|
|
|
|
|
2004-02-24 00:03:10 +01:00
|
|
|
#include <dirent.h>
|
|
|
|
|
|
|
|
|
|
|
1996-08-28 03:59:28 +02:00
|
|
|
/*
|
|
|
|
|
* FileSeek uses the standard UNIX lseek(2) flags.
|
|
|
|
|
*/
|
|
|
|
|
|
1997-09-08 04:41:22 +02:00
|
|
|
typedef char *FileName;
|
1996-08-28 03:59:28 +02:00
|
|
|
|
1997-09-08 04:41:22 +02:00
|
|
|
typedef int File;
|
1996-08-28 03:59:28 +02:00
|
|
|
|
2001-09-30 20:57:45 +02:00
|
|
|
|
|
|
|
|
/* GUC parameter */
|
2001-10-25 07:50:21 +02:00
|
|
|
extern int max_files_per_process;
|
2001-09-30 20:57:45 +02:00
|
|
|
|
2012-03-29 07:19:11 +02:00
|
|
|
/*
|
|
|
|
|
* This is private to fd.c, but exported for save/restore_backend_variables()
|
|
|
|
|
*/
|
|
|
|
|
extern int max_safe_fds;
|
|
|
|
|
|
2001-09-30 20:57:45 +02:00
|
|
|
|
1996-08-28 03:59:28 +02:00
|
|
|
/*
|
|
|
|
|
* prototypes for functions in fd.c
|
|
|
|
|
*/
|
1999-05-09 02:52:08 +02:00
|
|
|
|
|
|
|
|
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
|
1997-09-08 04:41:22 +02:00
|
|
|
extern File PathNameOpenFile(FileName fileName, int fileFlags, int fileMode);
|
2007-06-07 21:19:57 +02:00
|
|
|
extern File OpenTemporaryFile(bool interXact);
|
1997-09-08 04:41:22 +02:00
|
|
|
extern void FileClose(File file);
|
Create and use wait events for read, write, and fsync operations.
Previous commits, notably 53be0b1add7064ca5db3cd884302dfc3268d884e and
6f3bd98ebfc008cbd676da777bb0b2376c4c4bfa, made it possible to see from
pg_stat_activity when a backend was stuck waiting for another backend,
but it's also fairly common for a backend to be stuck waiting for an
I/O. Add wait events for those operations, too.
Rushabh Lathia, with further hacking by me. Reviewed and tested by
Michael Paquier, Amit Kapila, Rajkumar Raghuwanshi, and Rahila Syed.
Discussion: http://postgr.es/m/CAGPqQf0LsYHXREPAZqYGVkDqHSyjf=KsD=k0GTVPAuzyThh-VQ@mail.gmail.com
2017-03-18 12:43:01 +01:00
|
|
|
extern int FilePrefetch(File file, off_t offset, int amount, uint32 wait_event_info);
|
|
|
|
|
extern int FileRead(File file, char *buffer, int amount, uint32 wait_event_info);
|
|
|
|
|
extern int FileWrite(File file, char *buffer, int amount, uint32 wait_event_info);
|
|
|
|
|
extern int FileSync(File file, uint32 wait_event_info);
|
2008-03-10 21:06:27 +01:00
|
|
|
extern off_t FileSeek(File file, off_t offset, int whence);
|
Create and use wait events for read, write, and fsync operations.
Previous commits, notably 53be0b1add7064ca5db3cd884302dfc3268d884e and
6f3bd98ebfc008cbd676da777bb0b2376c4c4bfa, made it possible to see from
pg_stat_activity when a backend was stuck waiting for another backend,
but it's also fairly common for a backend to be stuck waiting for an
I/O. Add wait events for those operations, too.
Rushabh Lathia, with further hacking by me. Reviewed and tested by
Michael Paquier, Amit Kapila, Rajkumar Raghuwanshi, and Rahila Syed.
Discussion: http://postgr.es/m/CAGPqQf0LsYHXREPAZqYGVkDqHSyjf=KsD=k0GTVPAuzyThh-VQ@mail.gmail.com
2017-03-18 12:43:01 +01:00
|
|
|
extern int FileTruncate(File file, off_t offset, uint32 wait_event_info);
|
|
|
|
|
extern void FileWriteback(File file, off_t offset, off_t nbytes, uint32 wait_event_info);
|
2009-08-05 20:01:54 +02:00
|
|
|
extern char *FilePathName(File file);
|
2016-03-08 16:09:50 +01:00
|
|
|
extern int FileGetRawDesc(File file);
|
2016-06-10 00:02:36 +02:00
|
|
|
extern int FileGetRawFlags(File file);
|
2016-03-08 16:09:50 +01:00
|
|
|
extern int FileGetRawMode(File file);
|
1999-05-09 02:52:08 +02:00
|
|
|
|
|
|
|
|
/* Operations that allow use of regular stdio --- USE WITH CAUTION */
|
2006-03-04 22:32:47 +01:00
|
|
|
extern FILE *AllocateFile(const char *name, const char *mode);
|
2004-02-24 00:03:10 +01:00
|
|
|
extern int FreeFile(FILE *file);
|
|
|
|
|
|
Add support for piping COPY to/from an external program.
This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding
psql \copy syntax. Like with reading/writing files, the backend version is
superuser-only, and in the psql version, the program is run in the client.
In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you
the stdin/stdout is quoted, it's now interpreted as a filename. For example,
"\copy foo from 'stdin'" now reads from a file called 'stdin', not from
standard input. Before this, there was no way to specify a filename called
stdin, stdout, pstdin or pstdout.
This creates a new function in pgport, wait_result_to_str(), which can
be used to convert the exit status of a process, as returned by wait(3),
to a human-readable string.
Etsuro Fujita, reviewed by Amit Kapila.
2013-02-27 17:17:21 +01:00
|
|
|
/* Operations that allow use of pipe streams (popen/pclose) */
|
|
|
|
|
extern FILE *OpenPipeStream(const char *command, const char *mode);
|
|
|
|
|
extern int ClosePipeStream(FILE *file);
|
|
|
|
|
|
2004-02-24 00:03:10 +01:00
|
|
|
/* Operations to allow use of the <dirent.h> library routines */
|
|
|
|
|
extern DIR *AllocateDir(const char *dirname);
|
2005-06-19 23:34:03 +02:00
|
|
|
extern struct dirent *ReadDir(DIR *dir, const char *dirname);
|
2004-02-24 00:03:10 +01:00
|
|
|
extern int FreeDir(DIR *dir);
|
1999-05-09 02:52:08 +02:00
|
|
|
|
Add OpenTransientFile, with automatic cleanup at end-of-xact.
Files opened with BasicOpenFile or PathNameOpenFile are not automatically
cleaned up on error. That puts unnecessary burden on callers that only want
to keep the file open for a short time. There is AllocateFile, but that
returns a buffered FILE * stream, which in many cases is not the nicest API
to work with. So add function called OpenTransientFile, which returns a
unbuffered fd that's cleaned up like the FILE* returned by AllocateFile().
This plugs a few rare fd leaks in error cases:
1. copy_file() - fixed by by using OpenTransientFile instead of BasicOpenFile
2. XLogFileInit() - fixed by adding close() calls to the error cases. Can't
use OpenTransientFile here because the fd is supposed to persist over
transaction boundaries.
3. lo_import/lo_export - fixed by using OpenTransientFile instead of
PathNameOpenFile.
In addition to plugging those leaks, this replaces many BasicOpenFile() calls
with OpenTransientFile() that were not leaking, because the code meticulously
closed the file on error. That wasn't strictly necessary, but IMHO it's good
for robustness.
The same leaks exist in older versions, but given the rarity of the issues,
I'm not backpatching this. Not yet, anyway - it might be good to backpatch
later, after this mechanism has had some more testing in master branch.
2012-11-27 09:25:50 +01:00
|
|
|
/* Operations to allow use of a plain kernel FD, with automatic cleanup */
|
|
|
|
|
extern int OpenTransientFile(FileName fileName, int fileFlags, int fileMode);
|
|
|
|
|
extern int CloseTransientFile(int fd);
|
|
|
|
|
|
2000-06-02 05:58:34 +02:00
|
|
|
/* If you've really really gotta have a plain kernel FD, use this */
|
|
|
|
|
extern int BasicOpenFile(FileName fileName, int fileFlags, int fileMode);
|
|
|
|
|
|
1999-05-09 02:52:08 +02:00
|
|
|
/* Miscellaneous support routines */
|
2005-08-08 05:12:16 +02:00
|
|
|
extern void InitFileAccess(void);
|
2004-02-23 21:45:59 +01:00
|
|
|
extern void set_max_safe_fds(void);
|
1997-09-08 04:41:22 +02:00
|
|
|
extern void closeAllVfds(void);
|
2007-06-07 21:19:57 +02:00
|
|
|
extern void SetTempTablespaces(Oid *tableSpaces, int numSpaces);
|
|
|
|
|
extern bool TempTablespacesAreSet(void);
|
|
|
|
|
extern Oid GetNextTempTableSpace(void);
|
1999-05-09 02:52:08 +02:00
|
|
|
extern void AtEOXact_Files(void);
|
2004-09-16 18:58:44 +02:00
|
|
|
extern void AtEOSubXact_Files(bool isCommit, SubTransactionId mySubid,
|
2005-10-15 04:49:52 +02:00
|
|
|
SubTransactionId parentSubid);
|
2001-06-11 06:12:29 +02:00
|
|
|
extern void RemovePgTempFiles(void);
|
2007-06-07 21:19:57 +02:00
|
|
|
|
2000-12-08 23:21:33 +01:00
|
|
|
extern int pg_fsync(int fd);
|
2005-05-20 16:53:26 +02:00
|
|
|
extern int pg_fsync_no_writethrough(int fd);
|
|
|
|
|
extern int pg_fsync_writethrough(int fd);
|
2001-02-18 05:39:42 +01:00
|
|
|
extern int pg_fdatasync(int fd);
|
Allow to trigger kernel writeback after a configurable number of writes.
Currently writes to the main data files of postgres all go through the
OS page cache. This means that some operating systems can end up
collecting a large number of dirty buffers in their respective page
caches. When these dirty buffers are flushed to storage rapidly, be it
because of fsync(), timeouts, or dirty ratios, latency for other reads
and writes can increase massively. This is the primary reason for
regular massive stalls observed in real world scenarios and artificial
benchmarks; on rotating disks stalls on the order of hundreds of seconds
have been observed.
On linux it is possible to control this by reducing the global dirty
limits significantly, reducing the above problem. But global
configuration is rather problematic because it'll affect other
applications; also PostgreSQL itself doesn't always generally want this
behavior, e.g. for temporary files it's undesirable.
Several operating systems allow some control over the kernel page
cache. Linux has sync_file_range(2), several posix systems have msync(2)
and posix_fadvise(2). sync_file_range(2) is preferable because it
requires no special setup, whereas msync() requires the to-be-flushed
range to be mmap'ed. For the purpose of flushing dirty data
posix_fadvise(2) is the worst alternative, as flushing dirty data is
just a side-effect of POSIX_FADV_DONTNEED, which also removes the pages
from the page cache. Thus the feature is enabled by default only on
linux, but can be enabled on all systems that have any of the above
APIs.
While desirable and likely possible this patch does not contain an
implementation for windows.
With the infrastructure added, writes made via checkpointer, bgwriter
and normal user backends can be flushed after a configurable number of
writes. Each of these sources of writes controlled by a separate GUC,
checkpointer_flush_after, bgwriter_flush_after and backend_flush_after
respectively; they're separate because the number of flushes that are
good are separate, and because the performance considerations of
controlled flushing for each of these are different.
A later patch will add checkpoint sorting - after that flushes from the
ckeckpoint will almost always be desirable. Bgwriter flushes are most of
the time going to be random, which are slow on lots of storage hardware.
Flushing in backends works well if the storage and bgwriter can keep up,
but if not it can have negative consequences. This patch is likely to
have negative performance consequences without checkpoint sorting, but
unfortunately so has sorting without flush control.
Discussion: alpine.DEB.2.10.1506011320000.28433@sto
Author: Fabien Coelho and Andres Freund
2016-02-19 21:13:05 +01:00
|
|
|
extern void pg_flush_data(int fd, off_t offset, off_t amount);
|
2016-03-10 03:53:53 +01:00
|
|
|
extern void fsync_fname(const char *fname, bool isdir);
|
|
|
|
|
extern int durable_rename(const char *oldfile, const char *newfile, int loglevel);
|
2017-03-27 18:33:01 +02:00
|
|
|
extern int durable_unlink(const char *fname, int loglevel);
|
2016-03-10 03:53:53 +01:00
|
|
|
extern int durable_link_or_rename(const char *oldfile, const char *newfile, int loglevel);
|
Fix fsync-at-startup code to not treat errors as fatal.
Commit 2ce439f3379aed857517c8ce207485655000fc8e introduced a rather serious
regression, namely that if its scan of the data directory came across any
un-fsync-able files, it would fail and thereby prevent database startup.
Worse yet, symlinks to such files also caused the problem, which meant that
crash restart was guaranteed to fail on certain common installations such
as older Debian.
After discussion, we agreed that (1) failure to start is worse than any
consequence of not fsync'ing is likely to be, therefore treat all errors
in this code as nonfatal; (2) we should not chase symlinks other than
those that are expected to exist, namely pg_xlog/ and tablespace links
under pg_tblspc/. The latter restriction avoids possibly fsync'ing a
much larger part of the filesystem than intended, if the user has left
random symlinks hanging about in the data directory.
This commit takes care of that and also does some code beautification,
mainly moving the relevant code into fd.c, which seems a much better place
for it than xlog.c, and making sure that the conditional compilation for
the pre_sync_fname pass has something to do with whether pg_flush_data
works.
I also relocated the call site in xlog.c down a few lines; it seems a
bit silly to be doing this before ValidateXLOGDirectoryStructure().
The similar logic in initdb.c ought to be made to match this, but that
change is noncritical and will be dealt with separately.
Back-patch to all active branches, like the prior commit.
Abhijit Menon-Sen and Tom Lane
2015-05-28 23:33:03 +02:00
|
|
|
extern void SyncDataDirectory(void);
|
2001-10-28 07:26:15 +01:00
|
|
|
|
2003-12-20 18:31:21 +01:00
|
|
|
/* Filename components for OpenTemporaryFile */
|
|
|
|
|
#define PG_TEMP_FILES_DIR "pgsql_tmp"
|
|
|
|
|
#define PG_TEMP_FILE_PREFIX "pgsql_tmp"
|
|
|
|
|
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
#endif /* FD_H */
|
