rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/include/utils/memutils.h

175 lines
6.2 KiB
C
Raw Normal View History

More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
/*-------------------------------------------------------------------------
*
Change my-function-name-- to my_function_name, and optimizer renames.
1999-02-14 00:22:53 +01:00
* memutils.h
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* This file contains declarations for memory allocation utility
* functions. These are functions that are not quite widely used
* enough to justify going in utils/palloc.h, but are still part
* of the API of the memory management subsystem.
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*
*
Update copyright via script for 2017
2017-01-03 19:48:53 +01:00
* Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
Add: * Portions Copyright (c) 1996-2000, PostgreSQL, Inc to all files copyright Regents of Berkeley. Man, that's a lot of files.
2000-01-26 06:58:53 +01:00
* Portions Copyright (c) 1994, Regents of the University of California
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*
Remove cvs keywords from all files.
2010-09-20 22:08:53 +02:00
* src/include/utils/memutils.h
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*
*-------------------------------------------------------------------------
*/
#ifndef MEMUTILS_H
#define MEMUTILS_H
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
#include "nodes/memnodes.h"
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
/*
Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). The MaxAllocSize guard is convenient for most callers, because it reduces the need for careful attention to overflow, data type selection, and the SET_VARSIZE() limit. A handful of callers are happy to navigate those hazards in exchange for the ability to allocate a larger chunk. Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX tuples, a factor-of-48 increase. In particular, B-tree index builds can now benefit from much-larger maintenance_work_mem settings. Reviewed by Stephen Frost, Simon Riggs and Jeff Janes.
2013-06-27 20:53:57 +02:00
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*
* Note:
Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). The MaxAllocSize guard is convenient for most callers, because it reduces the need for careful attention to overflow, data type selection, and the SET_VARSIZE() limit. A handful of callers are happy to navigate those hazards in exchange for the ability to allocate a larger chunk. Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX tuples, a factor-of-48 increase. In particular, B-tree index builds can now benefit from much-larger maintenance_work_mem settings. Reviewed by Stephen Frost, Simon Riggs and Jeff Janes.
2013-06-27 20:53:57 +02:00
* There is no guarantee that smaller allocations will succeed, but
* larger requests will be summarily denied.
*
* palloc() enforces MaxAllocSize, chosen to correspond to the limiting size
* of varlena objects under TOAST. See VARSIZE_4B() and related macros in
* postgres.h. Many datatypes assume that any allocatable size can be
* represented in a varlena header. This limit also permits a caller to use
* an "int" variable for an index into or length of an allocation. Callers
* careful to avoid these hazards can access the higher limit with
* MemoryContextAllocHuge(). Both limits permit code to assume that it may
* compute twice an allocation's size without overflow.
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
#define MaxAllocSize ((Size) 0x3fffffff) /* 1 gigabyte - 1 */
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
Tweak palloc/repalloc to allow zero bytes to be requested, as per recent proposal. Eliminate several dozen now-unnecessary hacks to avoid palloc(0). (It's likely there are more that I didn't find.)
2004-06-05 21:48:09 +02:00
#define AllocSizeIsValid(size) ((Size) (size) <= MaxAllocSize)
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). The MaxAllocSize guard is convenient for most callers, because it reduces the need for careful attention to overflow, data type selection, and the SET_VARSIZE() limit. A handful of callers are happy to navigate those hazards in exchange for the ability to allocate a larger chunk. Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX tuples, a factor-of-48 increase. In particular, B-tree index builds can now benefit from much-larger maintenance_work_mem settings. Reviewed by Stephen Frost, Simon Riggs and Jeff Janes.
2013-06-27 20:53:57 +02:00
#define MaxAllocHugeSize ((Size) -1 >> 1) /* SIZE_MAX / 2 */
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
/*
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* All chunks allocated by any memory context manager are required to be
* preceded by a StandardChunkHeader at a spacing of STANDARDCHUNKHEADERSIZE.
* A currently-allocated chunk must contain a backpointer to its owning
pgindent run for 9.4 This includes removing tabs after periods in C comments, which was applied to back branches, so this change should not effect backpatching.
2014-05-06 18:12:18 +02:00
* context as well as the allocated size of the chunk. The backpointer is
* used by pfree() and repalloc() to find the context to call. The allocated
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* size is not absolutely essential, but it's expected to be needed by any
* reasonable implementation.
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*/
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
typedef struct StandardChunkHeader
Massive commit to run PGINDENT on all *.c and *.h files.
1997-09-07 07:04:48 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
MemoryContext context; /* owning context */
Clean up MEMORY_CONTEXT_CHECKING code, and apply it more thoroughly. Also, apply Karel Zak's patch to recycle residual space in an exhausted allocation block. (Bet you thought I'd forgot about that, Karel?)
2000-12-01 06:16:45 +01:00
Size size; /* size of data space allocated in chunk */
Here is the patch with memory leak checker. This checker allow detect in-chunk leaks, overwrite-next-chunk leaks and overwrite block-freeptr leaks. A in-chunk leak --- if something overwrite space after wanted (via palloc() size, but it is still inside chunk. For example x = palloc(12); /* create 16b chunk */ memset(x, '#', 13); this leak is in the current source total invisible, because chunk is 16b and leak is in the "align space". For this feature I add data_size to StandardChunk, and all memory which go from AllocSetAlloc() is marked as 0x7F. The MemoryContextCheck() is compiled '#ifdef USE_ASSERT_CHECKING'. I add this checking to 'tcop/postgres.c' and is active after each backend query, but it is probably not sufficient, because some MemoryContext exist only during memory processing --- will good if someone who known where it is needful (Tom:-) add it for others contexts; A problem in the current source is that we have still some malloc() allocation that is not needful and this allocation is total invisible for all context routines. For example Dllist in backend (pretty dirty it is in catcache where values in Dllist are palloc-ed, but list is malloc-ed). --- and BTW. this Dllist design stand in the way for query cache :-) Tom, if you agree I start replace some mallocs. BTW. --- Tom, have you idea for across transaction presistent allocation for SQL functions? (like regex - now it is via malloc) I almost forget. I add one if() to AllocSetAlloc(), for 'size' that are greater than ALLOC_BIGCHUNK_LIMIT is not needful check AllocSetFreeIndex(), because 'fidx' is always 'ALLOCSET_NUM_FREELISTS - 1'. It a little brisk up allocation for very large chunks. Right? Karel
2000-07-11 16:30:37 +02:00
#ifdef MEMORY_CONTEXT_CHECKING
Clean up MEMORY_CONTEXT_CHECKING code, and apply it more thoroughly. Also, apply Karel Zak's patch to recycle residual space in an exhausted allocation block. (Bet you thought I'd forgot about that, Karel?)
2000-12-01 06:16:45 +01:00
/* when debugging memory usage, also store actual requested size */
Size requested_size;
Here is the patch with memory leak checker. This checker allow detect in-chunk leaks, overwrite-next-chunk leaks and overwrite block-freeptr leaks. A in-chunk leak --- if something overwrite space after wanted (via palloc() size, but it is still inside chunk. For example x = palloc(12); /* create 16b chunk */ memset(x, '#', 13); this leak is in the current source total invisible, because chunk is 16b and leak is in the "align space". For this feature I add data_size to StandardChunk, and all memory which go from AllocSetAlloc() is marked as 0x7F. The MemoryContextCheck() is compiled '#ifdef USE_ASSERT_CHECKING'. I add this checking to 'tcop/postgres.c' and is active after each backend query, but it is probably not sufficient, because some MemoryContext exist only during memory processing --- will good if someone who known where it is needful (Tom:-) add it for others contexts; A problem in the current source is that we have still some malloc() allocation that is not needful and this allocation is total invisible for all context routines. For example Dllist in backend (pretty dirty it is in catcache where values in Dllist are palloc-ed, but list is malloc-ed). --- and BTW. this Dllist design stand in the way for query cache :-) Tom, if you agree I start replace some mallocs. BTW. --- Tom, have you idea for across transaction presistent allocation for SQL functions? (like regex - now it is via malloc) I almost forget. I add one if() to AllocSetAlloc(), for 'size' that are greater than ALLOC_BIGCHUNK_LIMIT is not needful check AllocSetFreeIndex(), because 'fidx' is always 'ALLOCSET_NUM_FREELISTS - 1'. It a little brisk up allocation for very large chunks. Right? Karel
2000-07-11 16:30:37 +02:00
#endif
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
} StandardChunkHeader;
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
#define STANDARDCHUNKHEADERSIZE MAXALIGN(sizeof(StandardChunkHeader))
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
Alter AllocSet routines so that requests larger than ALLOC_BIGCHUNK_LIMIT are always allocated as separate malloc() blocks, and are free()d immediately upon pfree(). Also, if such a chunk is enlarged with repalloc(), translate the operation into a realloc() so as to minimize memory usage. Of course, these large chunks still get freed automatically if the alloc set is reset. I have set ALLOC_BIGCHUNK_LIMIT at 64K for now, but perhaps another size would be better?
1999-08-24 22:11:19 +02:00
New alloc set code using a memory block pool for small allocations. Jan
1999-02-06 17:50:34 +01:00
/*
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* Standard top-level memory contexts.
Alter AllocSet routines so that requests larger than ALLOC_BIGCHUNK_LIMIT are always allocated as separate malloc() blocks, and are free()d immediately upon pfree(). Also, if such a chunk is enlarged with repalloc(), translate the operation into a realloc() so as to minimize memory usage. Of course, these large chunks still get freed automatically if the alloc set is reset. I have set ALLOC_BIGCHUNK_LIMIT at 64K for now, but perhaps another size would be better?
1999-08-24 22:11:19 +02:00
*
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* Only TopMemoryContext and ErrorContext are initialized by
* MemoryContextInit() itself.
New alloc set code using a memory block pool for small allocations. Jan
1999-02-06 17:50:34 +01:00
*/
Rename DLLIMPORT macro to PGDLLIMPORT to avoid conflict with third party includes (like tcl) that define DLLIMPORT.
2007-07-25 14:22:54 +02:00
extern PGDLLIMPORT MemoryContext TopMemoryContext;
extern PGDLLIMPORT MemoryContext ErrorContext;
extern PGDLLIMPORT MemoryContext PostmasterContext;
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
extern PGDLLIMPORT MemoryContext MessageContext;
extern PGDLLIMPORT MemoryContext TopTransactionContext;
extern PGDLLIMPORT MemoryContext CurTransactionContext;
pgindent run.
2003-08-04 02:43:34 +02:00
First phase of plan-invalidation project: create a plan cache management module and teach PREPARE and protocol-level prepared statements to use it. In service of this, rearrange utility-statement processing so that parse analysis does not assume table schemas can't change before execution for utility statements (necessary because we don't attempt to re-acquire locks for utility statements when reusing a stored plan). This requires some refactoring of the ProcessUtility API, but it ends up cleaner anyway, for instance we can get rid of the QueryContext global. Still to do: fix up SPI and related code to use the plan cache; I'm tempted to try to make SQL functions use it too. Also, there are at least some aspects of system state that we want to ensure remain the same during a replan as in the original processing; search_path certainly ought to behave that way for instance, and perhaps there are others.
2007-03-13 01:33:44 +01:00
/* This is a transient link to the active portal's memory context: */
Rename DLLIMPORT macro to PGDLLIMPORT to avoid conflict with third party includes (like tcl) that define DLLIMPORT.
2007-07-25 14:22:54 +02:00
extern PGDLLIMPORT MemoryContext PortalContext;
New alloc set code using a memory block pool for small allocations. Jan
1999-02-06 17:50:34 +01:00
Redefine MemoryContextReset() as deleting, not resetting, child contexts. That is, MemoryContextReset() now means what was formerly meant by MemoryContextResetAndDeleteChildren(), and the latter is now just a macro alias for the former. If you really want the functionality that was formerly provided by MemoryContextReset(), what you have to do is MemoryContextResetChildren() plus MemoryContextResetOnly() (which is a new API to reset *only* the named context and not touch its children). The reason for this change is that near fifteen years of experience has proven that there is noplace where old-style MemoryContextReset() is actually what you want. Making that the default behavior has led to lots of context-leakage bugs, while we've not found anyplace where it's actually necessary to keep the child contexts; at least the standard regression tests do not reveal anyplace where this change breaks anything. And there are upcoming patches that will introduce additional reasons why child contexts need to be removed. We could change existing calls of MemoryContextResetAndDeleteChildren to be just MemoryContextReset, but for the moment I'll leave them alone; they're not costing anything.
2015-02-28 00:09:42 +01:00
/* Backwards compatibility macro */
#define MemoryContextResetAndDeleteChildren(ctx) MemoryContextReset(ctx)
New alloc set code using a memory block pool for small allocations. Jan
1999-02-06 17:50:34 +01:00
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
/*
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* Memory-context-type-independent functions in mcxt.c
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*/
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
extern void MemoryContextInit(void);
extern void MemoryContextReset(MemoryContext context);
extern void MemoryContextDelete(MemoryContext context);
Redefine MemoryContextReset() as deleting, not resetting, child contexts. That is, MemoryContextReset() now means what was formerly meant by MemoryContextResetAndDeleteChildren(), and the latter is now just a macro alias for the former. If you really want the functionality that was formerly provided by MemoryContextReset(), what you have to do is MemoryContextResetChildren() plus MemoryContextResetOnly() (which is a new API to reset *only* the named context and not touch its children). The reason for this change is that near fifteen years of experience has proven that there is noplace where old-style MemoryContextReset() is actually what you want. Making that the default behavior has led to lots of context-leakage bugs, while we've not found anyplace where it's actually necessary to keep the child contexts; at least the standard regression tests do not reveal anyplace where this change breaks anything. And there are upcoming patches that will introduce additional reasons why child contexts need to be removed. We could change existing calls of MemoryContextResetAndDeleteChildren to be just MemoryContextReset, but for the moment I'll leave them alone; they're not costing anything.
2015-02-28 00:09:42 +01:00
extern void MemoryContextResetOnly(MemoryContext context);
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
extern void MemoryContextResetChildren(MemoryContext context);
extern void MemoryContextDeleteChildren(MemoryContext context);
Invent a new memory context primitive, MemoryContextSetParent. This function will be useful for altering the lifespan of a context after creation (for example, by creating it under a transient context and later reparenting it to belong to a long-lived context). It costs almost no new code, since we can refactor what was there. Per my proposal of yesterday.
2011-09-11 22:29:42 +02:00
extern void MemoryContextSetParent(MemoryContext context,
Run pgindent on 9.2 source tree in preparation for first 9.3 commit-fest.
2012-06-10 21:20:04 +02:00
MemoryContext new_parent);
Change memory-space accounting mechanism in tuplesort.c and tuplestore.c to make a reasonable attempt at accounting for palloc overhead, not just the requested size of each memory chunk. Since in many scenarios this will make for a significant reduction in the amount of space acquired, partially compensate by doubling the default value of SORT_MEM to 1Mb. Per discussion in pgsql-general around 9-Jun-2002..
2002-08-12 02:36:12 +02:00
extern Size GetMemoryChunkSpace(void *pointer);
Code review for palloc0 patch --- avoid dangerous and unnecessary practice of evaluating MemSet's arguments multiple times, except for the special case of newNode(), where we can assume the argument is a constant sizeof() operator. Also, add GetMemoryChunkContext() to mcxt.c's API, in preparation for fixing recent GEQO breakage.
2002-12-16 17:22:46 +01:00
extern MemoryContext GetMemoryChunkContext(void *pointer);
Redesign the plancache mechanism for more flexibility and efficiency. Rewrite plancache.c so that a "cached plan" (which is rather a misnomer at this point) can support generation of custom, parameter-value-dependent plans, and can make an intelligent choice between using custom plans and the traditional generic-plan approach. The specific choice algorithm implemented here can probably be improved in future, but this commit is all about getting the mechanism in place, not the policy. In addition, restructure the API to greatly reduce the amount of extraneous data copying needed. The main compromise needed to make that possible was to split the initial creation of a CachedPlanSource into two steps. It's worth noting in particular that SPI_saveplan is now deprecated in favor of SPI_keepplan, which accomplishes the same end result with zero data copying, and no need to then spend even more cycles throwing away the original SPIPlan. The risk of long-term memory leaks while manipulating SPIPlans has also been greatly reduced. Most of this improvement is based on use of the recently-added MemoryContextSetParent primitive.
2011-09-16 06:42:53 +02:00
extern MemoryContext MemoryContextGetParent(MemoryContext context);
Add some marginal tweaks to eliminate memory leakages associated with subtransactions. Trivial subxacts (such as a plpgsql exception block containing no database access) now demonstrably leak zero bytes.
2004-09-16 22:17:49 +02:00
extern bool MemoryContextIsEmpty(MemoryContext context);
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
extern void MemoryContextStats(MemoryContext context);
Limit the verbosity of memory context statistics dumps. We had a report from Stefan Kaltenbrunner of a case in which postmaster log files overran available disk space because multiple backends spewed enormous context stats dumps upon hitting an out-of-memory condition. Given the lack of similar reports, this isn't a common problem, but it still seems worth doing something about. However, we don't want to just blindly truncate the output, because that might prevent diagnosis of OOM problems. What seems like a workable compromise is to limit the dump to 100 child contexts per parent, and summarize the space used within any additional child contexts. That should help because practical cases where the dump gets long will typically be huge numbers of siblings under the same parent context; while the additional debugging value from seeing details about individual siblings beyond 100 will not be large, we hope. Anyway it doesn't take much code or memory space to do this, so let's try it like this and see how things go. Since the summarization mechanism requires passing totals back up anyway, I took the opportunity to add a "grand total" line to the end of the printout.
2015-08-25 19:09:48 +02:00
extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
Fix and enhance the assertion of no palloc's in a critical section. The assertion failed if WAL_DEBUG or LWLOCK_STATS was enabled; fix that by using separate memory contexts for the allocations made within those code blocks. This patch introduces a mechanism for marking any memory context as allowed in a critical section. Previously ErrorContext was exempt as a special case. Instead of a blanket exception of the checkpointer process, only exempt the memory context used for the pending ops hash table.
2014-06-30 09:13:48 +02:00
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
pgindent run.
2002-09-04 22:31:48 +02:00
Change memory-space accounting mechanism in tuplesort.c and tuplestore.c to make a reasonable attempt at accounting for palloc overhead, not just the requested size of each memory chunk. Since in many scenarios this will make for a significant reduction in the amount of space acquired, partially compensate by doubling the default value of SORT_MEM to 1Mb. Per discussion in pgsql-general around 9-Jun-2002..
2002-08-12 02:36:12 +02:00
#ifdef MEMORY_CONTEXT_CHECKING
Here is the patch with memory leak checker. This checker allow detect in-chunk leaks, overwrite-next-chunk leaks and overwrite block-freeptr leaks. A in-chunk leak --- if something overwrite space after wanted (via palloc() size, but it is still inside chunk. For example x = palloc(12); /* create 16b chunk */ memset(x, '#', 13); this leak is in the current source total invisible, because chunk is 16b and leak is in the "align space". For this feature I add data_size to StandardChunk, and all memory which go from AllocSetAlloc() is marked as 0x7F. The MemoryContextCheck() is compiled '#ifdef USE_ASSERT_CHECKING'. I add this checking to 'tcop/postgres.c' and is active after each backend query, but it is probably not sufficient, because some MemoryContext exist only during memory processing --- will good if someone who known where it is needful (Tom:-) add it for others contexts; A problem in the current source is that we have still some malloc() allocation that is not needful and this allocation is total invisible for all context routines. For example Dllist in backend (pretty dirty it is in catcache where values in Dllist are palloc-ed, but list is malloc-ed). --- and BTW. this Dllist design stand in the way for query cache :-) Tom, if you agree I start replace some mallocs. BTW. --- Tom, have you idea for across transaction presistent allocation for SQL functions? (like regex - now it is via malloc) I almost forget. I add one if() to AllocSetAlloc(), for 'size' that are greater than ALLOC_BIGCHUNK_LIMIT is not needful check AllocSetFreeIndex(), because 'fidx' is always 'ALLOCSET_NUM_FREELISTS - 1'. It a little brisk up allocation for very large chunks. Right? Karel
2000-07-11 16:30:37 +02:00
extern void MemoryContextCheck(MemoryContext context);
Change memory-space accounting mechanism in tuplesort.c and tuplestore.c to make a reasonable attempt at accounting for palloc overhead, not just the requested size of each memory chunk. Since in many scenarios this will make for a significant reduction in the amount of space acquired, partially compensate by doubling the default value of SORT_MEM to 1Mb. Per discussion in pgsql-general around 9-Jun-2002..
2002-08-12 02:36:12 +02:00
#endif
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
extern bool MemoryContextContains(MemoryContext context, void *pointer);
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
/*
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
* This routine handles the context-type-independent part of memory
* context creation. It's intended to be called from context-type-
* specific creation routines, and noplace else.
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
*/
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
extern MemoryContext MemoryContextCreate(NodeTag tag, Size size,
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
MemoryContextMethods *methods,
MemoryContext parent,
const char *name);
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
/*
* Memory-context-type-specific functions
*/
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
/* aset.c */
extern MemoryContext AllocSetContextCreate(MemoryContext parent,
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
const char *name,
Size minContextSize,
Size initBlockSize,
Size maxBlockSize);
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
/*
* Recommended default alloc parameters, suitable for "ordinary" contexts
* that might hold quite a lot of data.
*/
Tweak default memory context allocation policy so that a context is not given any malloc block until something is first allocated in it; but thereafter, MemoryContextReset won't release that first malloc block. This preserves the quick-reset property of the original policy, without forcing 8K to be allocated to every context whether any of it is ever used or not. Also, remove some more no-longer-needed explicit freeing during ExecEndPlan.
2002-12-15 22:01:34 +01:00
#define ALLOCSET_DEFAULT_MINSIZE 0
First phase of memory management rewrite (see backend/utils/mmgr/README for details). It doesn't really do that much yet, since there are no short-term memory contexts in the executor, but the infrastructure is in place and long-term contexts are handled reasonably. A few long- standing bugs have been fixed, such as 'VACUUM; anything' in a single query string crashing. Also, out-of-memory is now considered a recoverable ERROR, not FATAL. Eliminate a large amount of crufty, now-dead code in and around memory management. Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and backend startup.
2000-06-28 05:33:33 +02:00
#define ALLOCSET_DEFAULT_INITSIZE (8 * 1024)
#define ALLOCSET_DEFAULT_MAXSIZE (8 * 1024 * 1024)
Add macros to make AllocSetContextCreate() calls simpler and safer. I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls had typos in the context-sizing parameters. While none of these led to especially significant problems, they did create minor inefficiencies, and it's now clear that expecting people to copy-and-paste those calls accurately is not a great idea. Let's reduce the risk of future errors by introducing single macros that encapsulate the common use-cases. Three such macros are enough to cover all but two special-purpose contexts; those two calls can be left as-is, I think. While this patch doesn't in itself improve matters for third-party extensions, it doesn't break anything for them either, and they can gradually adopt the simplified notation over time. In passing, change TopMemoryContext to use the default allocation parameters. Formerly it could only be extended 8K at a time. That was probably reasonable when this code was written; but nowadays we create many more contexts than we did then, so that it's not unusual to have a couple hundred K in TopMemoryContext, even without considering various dubious code that sticks other things there. There seems no good reason not to let it use growing blocks like most other contexts. Back-patch to 9.6, mostly because that's still close enough to HEAD that it's easy to do so, and keeping the branches in sync can be expected to avoid some future back-patching pain. The bugs fixed by these changes don't seem to be significant enough to justify fixing them further back. Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
#define ALLOCSET_DEFAULT_SIZES \
ALLOCSET_DEFAULT_MINSIZE, ALLOCSET_DEFAULT_INITSIZE, ALLOCSET_DEFAULT_MAXSIZE
More cleanups of the include files - centralizing to simplify the -I's required to compile
1996-08-28 03:59:28 +02:00
Tweak default memory context allocation policy so that a context is not given any malloc block until something is first allocated in it; but thereafter, MemoryContextReset won't release that first malloc block. This preserves the quick-reset property of the original policy, without forcing 8K to be allocated to every context whether any of it is ever used or not. Also, remove some more no-longer-needed explicit freeing during ExecEndPlan.
2002-12-15 22:01:34 +01:00
/*
Add macros to make AllocSetContextCreate() calls simpler and safer. I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls had typos in the context-sizing parameters. While none of these led to especially significant problems, they did create minor inefficiencies, and it's now clear that expecting people to copy-and-paste those calls accurately is not a great idea. Let's reduce the risk of future errors by introducing single macros that encapsulate the common use-cases. Three such macros are enough to cover all but two special-purpose contexts; those two calls can be left as-is, I think. While this patch doesn't in itself improve matters for third-party extensions, it doesn't break anything for them either, and they can gradually adopt the simplified notation over time. In passing, change TopMemoryContext to use the default allocation parameters. Formerly it could only be extended 8K at a time. That was probably reasonable when this code was written; but nowadays we create many more contexts than we did then, so that it's not unusual to have a couple hundred K in TopMemoryContext, even without considering various dubious code that sticks other things there. There seems no good reason not to let it use growing blocks like most other contexts. Back-patch to 9.6, mostly because that's still close enough to HEAD that it's easy to do so, and keeping the branches in sync can be expected to avoid some future back-patching pain. The bugs fixed by these changes don't seem to be significant enough to justify fixing them further back. Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
* Recommended alloc parameters for "small" contexts that are never expected
Tweak default memory context allocation policy so that a context is not given any malloc block until something is first allocated in it; but thereafter, MemoryContextReset won't release that first malloc block. This preserves the quick-reset property of the original policy, without forcing 8K to be allocated to every context whether any of it is ever used or not. Also, remove some more no-longer-needed explicit freeing during ExecEndPlan.
2002-12-15 22:01:34 +01:00
* to contain much data (for example, a context to contain a query plan).
*/
pgindent run.
2003-08-04 02:43:34 +02:00
#define ALLOCSET_SMALL_MINSIZE 0
Tweak default memory context allocation policy so that a context is not given any malloc block until something is first allocated in it; but thereafter, MemoryContextReset won't release that first malloc block. This preserves the quick-reset property of the original policy, without forcing 8K to be allocated to every context whether any of it is ever used or not. Also, remove some more no-longer-needed explicit freeing during ExecEndPlan.
2002-12-15 22:01:34 +01:00
#define ALLOCSET_SMALL_INITSIZE (1 * 1024)
pgindent run.
2003-08-04 02:43:34 +02:00
#define ALLOCSET_SMALL_MAXSIZE (8 * 1024)
Add macros to make AllocSetContextCreate() calls simpler and safer. I found that half a dozen (nearly 5%) of our AllocSetContextCreate calls had typos in the context-sizing parameters. While none of these led to especially significant problems, they did create minor inefficiencies, and it's now clear that expecting people to copy-and-paste those calls accurately is not a great idea. Let's reduce the risk of future errors by introducing single macros that encapsulate the common use-cases. Three such macros are enough to cover all but two special-purpose contexts; those two calls can be left as-is, I think. While this patch doesn't in itself improve matters for third-party extensions, it doesn't break anything for them either, and they can gradually adopt the simplified notation over time. In passing, change TopMemoryContext to use the default allocation parameters. Formerly it could only be extended 8K at a time. That was probably reasonable when this code was written; but nowadays we create many more contexts than we did then, so that it's not unusual to have a couple hundred K in TopMemoryContext, even without considering various dubious code that sticks other things there. There seems no good reason not to let it use growing blocks like most other contexts. Back-patch to 9.6, mostly because that's still close enough to HEAD that it's easy to do so, and keeping the branches in sync can be expected to avoid some future back-patching pain. The bugs fixed by these changes don't seem to be significant enough to justify fixing them further back. Discussion: <21072.1472321324@sss.pgh.pa.us>
2016-08-27 23:50:38 +02:00
#define ALLOCSET_SMALL_SIZES \
ALLOCSET_SMALL_MINSIZE, ALLOCSET_SMALL_INITSIZE, ALLOCSET_SMALL_MAXSIZE
/*
* Recommended alloc parameters for contexts that should start out small,
* but might sometimes grow big.
*/
#define ALLOCSET_START_SMALL_SIZES \
ALLOCSET_SMALL_MINSIZE, ALLOCSET_SMALL_INITSIZE, ALLOCSET_DEFAULT_MAXSIZE
Tweak default memory context allocation policy so that a context is not given any malloc block until something is first allocated in it; but thereafter, MemoryContextReset won't release that first malloc block. This preserves the quick-reset property of the original policy, without forcing 8K to be allocated to every context whether any of it is ever used or not. Also, remove some more no-longer-needed explicit freeing during ExecEndPlan.
2002-12-15 22:01:34 +01:00
Fix bogus "out of memory" reports in tuplestore.c. The tuplesort/tuplestore memory management logic assumed that the chunk allocation overhead for its memtuples array could not increase when increasing the array size. This is and always was true for tuplesort, but we (I, I think) blindly copied that logic into tuplestore.c without noticing that the assumption failed to hold for the much smaller array elements used by tuplestore. Given rather small work_mem, this could result in an improper complaint about "unexpected out-of-memory situation", as reported by Brent DeSpain in bug #13530. The easiest way to fix this is just to increase tuplestore's initial array size so that the assumption holds. Rather than relying on magic constants, though, let's export a #define from aset.c that represents the safe allocation threshold, and make tuplestore's calculation depend on that. Do the same in tuplesort.c to keep the logic looking parallel, even though tuplesort.c isn't actually at risk at present. This will keep us from breaking it if we ever muck with the allocation parameters in aset.c. Back-patch to all supported versions. The error message doesn't occur pre-9.3, not so much because the problem can't happen as because the pre-9.3 tuplestore code neglected to check for it. (The chance of trouble is a great deal larger as of 9.3, though, due to changes in the array-size-increasing strategy.) However, allowing LACKMEM() to become true unexpectedly could still result in less-than-desirable behavior, so let's patch it all the way back.
2015-08-05 00:18:46 +02:00
/*
* Threshold above which a request in an AllocSet context is certain to be
* allocated separately (and thereby have constant allocation overhead).
* Few callers should be interested in this, but tuplesort/tuplestore need
* to know it.
*/
#define ALLOCSET_SEPARATE_THRESHOLD 8192
New pgindent run with fixes suggested by Tom. Patch manually reviewed, initdb/regression tests pass.
2001-11-05 18:46:40 +01:00
#endif /* MEMUTILS_H */