Add backend support for injection points

Injection points are a new facility that makes possible for developers
to run custom code in pre-defined code paths.  Its goal is to provide
ways to design and run advanced tests, for cases like:
- Race conditions, where processes need to do actions in a controlled
ordered manner.
- Forcing a state, like an ERROR, FATAL or even PANIC for OOM, to force
recovery, etc.
- Arbitrary sleeps.

This implements some basics, and there are plans to extend it more in
the future depending on what's required.  Hence, this commit adds a set
of routines in the backend that allows developers to attach, detach and
run injection points:
- A code path calling an injection point can be declared with the macro
INJECTION_POINT(name).
- InjectionPointAttach() and InjectionPointDetach() to respectively
attach and detach a callback to/from an injection point.  An injection
point name is registered in a shmem hash table with a library name and a
function name, which will be used to load the callback attached to an
injection point when its code path is run.

Injection point names are just strings, so as an injection point can be
declared and run by out-of-core extensions and modules, with callbacks
defined in external libraries.

This facility is hidden behind a dedicated switch for ./configure and
meson, disabled by default.

Note that backends use a local cache to store callbacks already loaded,
cleaning up their cache if a callback has found to be removed on a
best-effort basis.  This could be refined further but any tests but what
we have here was fine with the tests I've written while implementing
these backend APIs.

Author: Michael Paquier, with doc suggestions from Ashutosh Bapat.
Reviewed-by: Ashutosh Bapat, Nathan Bossart, Álvaro Herrera, Dilip
Kumar, Amul Sul, Nazir Bilal Yavuz
Discussion: https://postgr.es/m/ZTiV8tn_MIb_H2rE@paquier.xyz
This commit is contained in:
Michael Paquier 2024-01-22 10:15:50 +09:00
parent c03d91d9be
commit d86d20f0ba
17 changed files with 512 additions and 0 deletions

34
configure vendored
View File

@ -759,6 +759,7 @@ CPPFLAGS
LDFLAGS
CFLAGS
CC
enable_injection_points
enable_tap_tests
enable_dtrace
DTRACEFLAGS
@ -839,6 +840,7 @@ enable_profiling
enable_coverage
enable_dtrace
enable_tap_tests
enable_injection_points
with_blocksize
with_segsize
with_segsize_blocks
@ -1532,6 +1534,8 @@ Optional Features:
--enable-coverage build with coverage testing instrumentation
--enable-dtrace build with DTrace support
--enable-tap-tests enable TAP tests (requires Perl and IPC::Run)
--enable-injection-points
enable injection points (for testing)
--enable-depend turn on automatic dependency tracking
--enable-cassert enable assertion checks (for debugging)
--disable-largefile omit support for large files
@ -3682,6 +3686,36 @@ fi
#
# Injection points
#
# Check whether --enable-injection-points was given.
if test "${enable_injection_points+set}" = set; then :
enableval=$enable_injection_points;
case $enableval in
yes)
$as_echo "#define USE_INJECTION_POINTS 1" >>confdefs.h
;;
no)
:
;;
*)
as_fn_error $? "no argument expected for --enable-injection-points option" "$LINENO" 5
;;
esac
else
enable_injection_points=no
fi
#
# Block size
#

View File

@ -250,6 +250,13 @@ PGAC_ARG_BOOL(enable, tap-tests, no,
[enable TAP tests (requires Perl and IPC::Run)])
AC_SUBST(enable_tap_tests)
#
# Injection points
#
PGAC_ARG_BOOL(enable, injection-points, no, [enable injection points (for testing)],
[AC_DEFINE([USE_INJECTION_POINTS], 1, [Define to 1 to build with injection points. (--enable-injection-points)])])
AC_SUBST(enable_injection_points)
#
# Block size
#

View File

@ -1656,6 +1656,21 @@ build-postgresql:
</listitem>
</varlistentry>
<varlistentry id="configure-option-enable-injection-points">
<term><option>--enable-injection-points</option></term>
<listitem>
<para>
Compiles <productname>PostgreSQL</productname> with support for
injection points in the server. Injection points allow to run
user-defined code from within the server in pre-defined code paths.
This helps in testing and in the investigation of concurrency scenarios
in a controlled fashion. This option is disabled by default. See
<xref linkend="xfunc-addin-injection-points"/> for more details. This
option is intended to be used only by developers for testing.
</para>
</listitem>
</varlistentry>
<varlistentry id="configure-option-with-segsize-blocks">
<term><option>--with-segsize-blocks=SEGSIZE_BLOCKS</option></term>
<listitem>
@ -3160,6 +3175,21 @@ ninja install
</listitem>
</varlistentry>
<varlistentry id="configure-injection-points-meson">
<term><option>-Dinjection_points={ true | false }</option></term>
<listitem>
<para>
Compiles <productname>PostgreSQL</productname> with support for
injection points in the server. Injection points allow to run
user-defined code from within the server in pre-defined code paths.
This helps in testing and in the investigation of concurrency scenarios
in a controlled fashion. This option is disabled by default. See
<xref linkend="xfunc-addin-injection-points"/> for more details. This
option is intended to be used only by developers for testing.
</para>
</listitem>
</varlistentry>
<varlistentry id="configure-segsize-blocks-meson">
<term><option>-Dsegsize_blocks=SEGSIZE_BLOCKS</option></term>
<listitem>

View File

@ -3599,6 +3599,75 @@ uint32 WaitEventExtensionNew(const char *wait_event_name)
</para>
</sect2>
<sect2 id="xfunc-addin-injection-points">
<title>Injection Points</title>
<para>
An injection point with a given <literal>name</literal> is declared using
macro:
<programlisting>
INJECTION_POINT(name);
</programlisting>
There are a few injection points already declared at strategic points
within the server code. After adding a new injection point the code needs
to be compiled in order for that injection point to be available in the
binary. Add-ins written in C-language can declare injection points in
their own code using the same macro.
</para>
<para>
Add-ins can attach callbacks to an already-declared injection point by
calling:
<programlisting>
extern void InjectionPointAttach(const char *name,
const char *library,
const char *function);
</programlisting>
<literal>name</literal> is the name of the injection point, which when
reached during execution will execute the <literal>function</literal>
loaded from <literal>library</literal>.
</para>
<para>
Here is an example of callback for
<literal>InjectionPointCallback</literal>:
<programlisting>
static void
custom_injection_callback(const char *name)
{
elog(NOTICE, "%s: executed custom callback", name);
}
</programlisting>
This callback prints a message to server error log with severity
<literal>NOTICE</literal>, but callbacks may implement more complex
logic.
</para>
<para>
Optionally, it is possible to detach an injection point by calling:
<programlisting>
extern void InjectionPointDetach(const char *name);
</programlisting>
</para>
<para>
A callback attached to an injection point is available across all the
backends including the backends started after
<literal>InjectionPointAttach</literal> is called. It remains attached
while the server is running or until the injection point is detached
using <literal>InjectionPointDetach</literal>.
</para>
<para>
Enabling injections points requires
<option>--enable-injection-points</option> with
<command>configure</command> or <option>-Dinjection_points=true</option>
with <application>Meson</application>.
</para>
</sect2>
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>

View File

@ -431,6 +431,7 @@ meson_bin = find_program(meson_binpath, native: true)
###############################################################
cdata.set('USE_ASSERT_CHECKING', get_option('cassert') ? 1 : false)
cdata.set('USE_INJECTION_POINTS', get_option('injection_points') ? 1 : false)
blocksize = get_option('blocksize').to_int() * 1024

View File

@ -43,6 +43,9 @@ option('cassert', type: 'boolean', value: false,
option('tap_tests', type: 'feature', value: 'auto',
description: 'Enable TAP tests')
option('injection_points', type: 'boolean', value: false,
description: 'Enable injection points')
option('PG_TEST_EXTRA', type: 'string', value: '',
description: 'Enable selected extra tests')

View File

@ -203,6 +203,7 @@ enable_nls = @enable_nls@
enable_debug = @enable_debug@
enable_dtrace = @enable_dtrace@
enable_coverage = @enable_coverage@
enable_injection_points = @enable_injection_points@
enable_tap_tests = @enable_tap_tests@
python_includespec = @python_includespec@

View File

@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
#include "utils/snapmgr.h"
#include "utils/wait_event.h"
@ -151,6 +152,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, AsyncShmemSize());
size = add_size(size, StatsShmemSize());
size = add_size(size, WaitEventExtensionShmemSize());
size = add_size(size, InjectionPointShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@ -354,6 +356,7 @@ CreateOrAttachShmemStructs(void)
AsyncShmemInit();
StatsShmemInit();
WaitEventExtensionShmemInit();
InjectionPointShmemInit();
}
/*

View File

@ -56,3 +56,4 @@ NotifyQueueTailLock 47
WaitEventExtensionLock 48
WALSummarizerLock 49
DSMRegistryLock 50
InjectionPointLock 51

View File

@ -330,6 +330,7 @@ NotifyQueueTail "Waiting to update limit on <command>NOTIFY</command> message st
WaitEventExtension "Waiting to read or update custom wait events information for extensions."
WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)

View File

@ -21,6 +21,7 @@ OBJS = \
guc_funcs.o \
guc_tables.o \
help_config.o \
injection_point.o \
pg_config.o \
pg_controldata.o \
pg_rusage.o \

View File

@ -0,0 +1,317 @@
/*-------------------------------------------------------------------------
*
* injection_point.c
* Routines to control and run injection points in the code.
*
* Injection points can be used to run arbitrary code by attaching callbacks
* that would be executed in place of the named injection point.
*
* Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
*
* IDENTIFICATION
* src/backend/utils/misc/injection_point.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include <sys/stat.h>
#include "fmgr.h"
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/fd.h"
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "utils/hsearch.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#ifdef USE_INJECTION_POINTS
/*
* Hash table for storing injection points.
*
* InjectionPointHash is used to find an injection point by name.
*/
static HTAB *InjectionPointHash; /* find points from names */
/* Field sizes */
#define INJ_NAME_MAXLEN 64
#define INJ_LIB_MAXLEN 128
#define INJ_FUNC_MAXLEN 128
/* Single injection point stored in InjectionPointHash */
typedef struct InjectionPointEntry
{
char name[INJ_NAME_MAXLEN]; /* hash key */
char library[INJ_LIB_MAXLEN]; /* library */
char function[INJ_FUNC_MAXLEN]; /* function */
} InjectionPointEntry;
#define INJECTION_POINT_HASH_INIT_SIZE 16
#define INJECTION_POINT_HASH_MAX_SIZE 128
/*
* Backend local cache of injection callbacks already loaded, stored in
* TopMemoryContext.
*/
typedef struct InjectionPointCacheEntry
{
char name[INJ_NAME_MAXLEN];
InjectionPointCallback callback;
} InjectionPointCacheEntry;
static HTAB *InjectionPointCache = NULL;
/*
* injection_point_cache_add
*
* Add an injection point to the local cache.
*/
static void
injection_point_cache_add(const char *name,
InjectionPointCallback callback)
{
InjectionPointCacheEntry *entry;
bool found;
/* If first time, initialize */
if (InjectionPointCache == NULL)
{
HASHCTL hash_ctl;
hash_ctl.keysize = sizeof(char[INJ_NAME_MAXLEN]);
hash_ctl.entrysize = sizeof(InjectionPointCacheEntry);
hash_ctl.hcxt = TopMemoryContext;
InjectionPointCache = hash_create("InjectionPoint cache hash",
INJECTION_POINT_HASH_MAX_SIZE,
&hash_ctl,
HASH_ELEM | HASH_STRINGS | HASH_CONTEXT);
}
entry = (InjectionPointCacheEntry *)
hash_search(InjectionPointCache, name, HASH_ENTER, &found);
Assert(!found);
memcpy(entry->name, name, strlen(name));
entry->callback = callback;
}
/*
* injection_point_cache_remove
*
* Remove entry from the local cache. Note that this leaks a callback
* loaded but removed later on, which should have no consequence from
* a testing perspective.
*/
static void
injection_point_cache_remove(const char *name)
{
/* leave if no cache */
if (InjectionPointCache == NULL)
return;
(void) hash_search(InjectionPointCache, name, HASH_REMOVE, NULL);
}
/*
* injection_point_cache_get
*
* Retrieve an injection point from the local cache, if any.
*/
static InjectionPointCallback
injection_point_cache_get(const char *name)
{
bool found;
InjectionPointCacheEntry *entry;
/* no callback if no cache yet */
if (InjectionPointCache == NULL)
return NULL;
entry = (InjectionPointCacheEntry *)
hash_search(InjectionPointCache, name, HASH_FIND, &found);
if (found)
return entry->callback;
return NULL;
}
#endif /* USE_INJECTION_POINTS */
/*
* Return the space for dynamic shared hash table.
*/
Size
InjectionPointShmemSize(void)
{
#ifdef USE_INJECTION_POINTS
Size sz = 0;
sz = add_size(sz, hash_estimate_size(INJECTION_POINT_HASH_MAX_SIZE,
sizeof(InjectionPointEntry)));
return sz;
#else
return 0;
#endif
}
/*
* Allocate shmem space for dynamic shared hash.
*/
void
InjectionPointShmemInit(void)
{
#ifdef USE_INJECTION_POINTS
HASHCTL info;
/* key is a NULL-terminated string */
info.keysize = sizeof(char[INJ_NAME_MAXLEN]);
info.entrysize = sizeof(InjectionPointEntry);
InjectionPointHash = ShmemInitHash("InjectionPoint hash",
INJECTION_POINT_HASH_INIT_SIZE,
INJECTION_POINT_HASH_MAX_SIZE,
&info,
HASH_ELEM | HASH_FIXED_SIZE | HASH_STRINGS);
#endif
}
/*
* Attach a new injection point.
*/
void
InjectionPointAttach(const char *name,
const char *library,
const char *function)
{
#ifdef USE_INJECTION_POINTS
InjectionPointEntry *entry_by_name;
bool found;
if (strlen(name) >= INJ_NAME_MAXLEN)
elog(ERROR, "injection point name %s too long (maximum of %u)",
name, INJ_NAME_MAXLEN);
if (strlen(library) >= INJ_LIB_MAXLEN)
elog(ERROR, "injection point library %s too long (maximum of %u)",
library, INJ_LIB_MAXLEN);
if (strlen(function) >= INJ_FUNC_MAXLEN)
elog(ERROR, "injection point function %s too long (maximum of %u)",
function, INJ_FUNC_MAXLEN);
/*
* Allocate and register a new injection point. A new point should not
* exist. For testing purposes this should be fine.
*/
LWLockAcquire(InjectionPointLock, LW_EXCLUSIVE);
entry_by_name = (InjectionPointEntry *)
hash_search(InjectionPointHash, name,
HASH_ENTER, &found);
if (found)
{
LWLockRelease(InjectionPointLock);
elog(ERROR, "injection point \"%s\" already defined", name);
}
/* Save the entry */
memcpy(entry_by_name->name, name, sizeof(entry_by_name->name));
entry_by_name->name[INJ_NAME_MAXLEN - 1] = '\0';
memcpy(entry_by_name->library, library, sizeof(entry_by_name->library));
entry_by_name->library[INJ_LIB_MAXLEN - 1] = '\0';
memcpy(entry_by_name->function, function, sizeof(entry_by_name->function));
entry_by_name->function[INJ_FUNC_MAXLEN - 1] = '\0';
LWLockRelease(InjectionPointLock);
#else
elog(ERROR, "injection points are not supported by this build");
#endif
}
/*
* Detach an existing injection point.
*/
void
InjectionPointDetach(const char *name)
{
#ifdef USE_INJECTION_POINTS
bool found;
LWLockAcquire(InjectionPointLock, LW_EXCLUSIVE);
hash_search(InjectionPointHash, name, HASH_REMOVE, &found);
LWLockRelease(InjectionPointLock);
if (!found)
elog(ERROR, "injection point \"%s\" not found", name);
#else
elog(ERROR, "Injection points are not supported by this build");
#endif
}
/*
* Execute an injection point, if defined.
*
* Check first the shared hash table, and adapt the local cache depending
* on that as it could be possible that an entry to run has been removed.
*/
void
InjectionPointRun(const char *name)
{
#ifdef USE_INJECTION_POINTS
InjectionPointEntry *entry_by_name;
bool found;
InjectionPointCallback injection_callback;
LWLockAcquire(InjectionPointLock, LW_SHARED);
entry_by_name = (InjectionPointEntry *)
hash_search(InjectionPointHash, name,
HASH_FIND, &found);
LWLockRelease(InjectionPointLock);
/*
* If not found, do nothing and remove it from the local cache if it
* existed there.
*/
if (!found)
{
injection_point_cache_remove(name);
return;
}
/*
* Check if the callback exists in the local cache, to avoid unnecessary
* external loads.
*/
injection_callback = injection_point_cache_get(name);
if (injection_callback == NULL)
{
char path[MAXPGPATH];
/* not found in local cache, so load and register */
snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path,
entry_by_name->library, DLSUFFIX);
if (!pg_file_exists(path))
elog(ERROR, "could not find library \"%s\" for injection point \"%s\"",
path, name);
injection_callback = (InjectionPointCallback)
load_external_function(path, entry_by_name->function, true, NULL);
if (injection_callback == NULL)
elog(ERROR, "could not find function \"%s\" in library \"%s\" for injection point \"%s\"",
name, entry_by_name->function, path);
/* add it to the local cache when found */
injection_point_cache_add(name, injection_callback);
}
injection_callback(name);
#else
elog(ERROR, "Injection points are not supported by this build");
#endif
}

View File

@ -6,6 +6,7 @@ backend_sources += files(
'guc_funcs.c',
'guc_tables.c',
'help_config.c',
'injection_point.c',
'pg_config.c',
'pg_controldata.c',
'pg_rusage.c',

View File

@ -698,6 +698,9 @@
/* Define to build with ICU support. (--with-icu) */
#undef USE_ICU
/* Define to 1 to build with injection points. (--enable-injection-points) */
#undef USE_INJECTION_POINTS
/* Define to 1 to build with LDAP support. (--with-ldap) */
#undef USE_LDAP

View File

@ -0,0 +1,37 @@
/*-------------------------------------------------------------------------
* injection_point.h
* Definitions related to injection points.
*
* Copyright (c) 2001-2024, PostgreSQL Global Development Group
*
* src/include/utils/injection_point.h
*-------------------------------------------------------------------------
*/
#ifndef INJECTION_POINT_H
#define INJECTION_POINT_H
/*
* Injections points require --enable-injection-points.
*/
#ifdef USE_INJECTION_POINTS
#define INJECTION_POINT(name) InjectionPointRun(name)
#else
#define INJECTION_POINT(name) ((void) name)
#endif
/*
* Typedef for callback function launched by an injection point.
*/
typedef void (*InjectionPointCallback) (const char *name);
extern Size InjectionPointShmemSize(void);
extern void InjectionPointShmemInit(void);
extern void InjectionPointAttach(const char *name,
const char *library,
const char *function);
extern void InjectionPointRun(const char *name);
extern void InjectionPointDetach(const char *name);
#endif /* INJECTION_POINT_H */

View File

@ -54,6 +54,7 @@ pgxs_kv = {
'enable_rpath': get_option('rpath') ? 'yes' : 'no',
'enable_nls': libintl.found() ? 'yes' : 'no',
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
'enable_tap_tests': tap_tests_enabled ? 'yes' : 'no',
'enable_debug': get_option('debug') ? 'yes' : 'no',
'enable_coverage': 'no',

View File

@ -1205,6 +1205,8 @@ InheritableSocket
InitSampleScan_function
InitializeDSMForeignScan_function
InitializeWorkerForeignScan_function
InjectionPointCacheEntry
InjectionPointEntry
InlineCodeBlock
InsertStmt
Instrumentation