2011-09-12 20:24:29 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
|
*
|
|
|
|
|
* walsender_private.h
|
|
|
|
|
* Private definitions from replication/walsender.c.
|
|
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Portions Copyright (c) 2010-2023, PostgreSQL Global Development Group
|
2011-09-12 20:24:29 +02:00
|
|
|
*
|
|
|
|
|
* src/include/replication/walsender_private.h
|
|
|
|
|
*
|
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
|
*/
|
|
|
|
|
#ifndef _WALSENDER_PRIVATE_H
|
|
|
|
|
#define _WALSENDER_PRIVATE_H
|
|
|
|
|
|
|
|
|
|
#include "access/xlog.h"
|
2023-01-18 21:15:05 +01:00
|
|
|
#include "lib/ilist.h"
|
2011-09-12 20:24:29 +02:00
|
|
|
#include "nodes/nodes.h"
|
2023-04-08 09:24:24 +02:00
|
|
|
#include "nodes/replnodes.h"
|
2012-01-24 21:22:37 +01:00
|
|
|
#include "replication/syncrep.h"
|
Optimize walsender wake up logic using condition variables
WalSndWakeup() currently loops through all the walsenders slots, with a
spinlock acquisition and release for every iteration, to wake up waiting
walsenders.
This commonly was not a problem before e101dfac3a53c. But, to allow logical
decoding on standbys, we need to wake up logical walsenders after every WAL
record is applied on the standby, rather just when flushing WAL or switching
timelines. This causes a performance regression for workloads replaying a lot
of WAL records.
To solve this, we use condition variable (CV) to efficiently wake up
walsenders in WalSndWakeup().
Every walsender prepares to sleep on a shared memory CV. Note that it just
prepares to sleep on the CV (i.e., adds itself to the CV's waitlist), but does
not actually wait on the CV (IOW, it never calls ConditionVariableSleep()). It
still uses WaitEventSetWait() for waiting, because CV infrastructure doesn't
handle FeBe socket events currently. The processes (startup process,
walreceiver etc.) wanting to wake up walsenders use
ConditionVariableBroadcast(), which in turn calls SetLatch(), helping
walsenders come out of WaitEventSetWait().
We use separate shared memory CVs for physical and logical walsenders for
selective wake ups, see WalSndWakeup() for more details.
This approach is simple and reasonably efficient. But not very elegant. But
for 16 it seems to be a better path than a larger redesign of the CV
mechanism. A desirable future improvement would be to add support for CVs
into WaitEventSetWait().
This still leaves us with a small regression in very extreme workloads (due to
the spinlock acquisition in ConditionVariableBroadcast() when there are no
waiters) - but that seems acceptable.
Reported-by: Andres Freund <andres@anarazel.de>
Suggested-by: Andres Freund <andres@anarazel.de>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Discussion: https://www.postgresql.org/message-id/20230509190247.3rrplhdgem6su6cg%40awork3.anarazel.de
2023-05-21 18:44:55 +02:00
|
|
|
#include "storage/condition_variable.h"
|
2011-09-12 20:24:29 +02:00
|
|
|
#include "storage/latch.h"
|
|
|
|
|
#include "storage/shmem.h"
|
|
|
|
|
#include "storage/spin.h"
|
|
|
|
|
|
|
|
|
|
typedef enum WalSndState
|
|
|
|
|
{
|
|
|
|
|
WALSNDSTATE_STARTUP = 0,
|
|
|
|
|
WALSNDSTATE_BACKUP,
|
|
|
|
|
WALSNDSTATE_CATCHUP,
|
Prevent possibility of panics during shutdown checkpoint.
When the checkpointer writes the shutdown checkpoint, it checks
afterwards whether any WAL has been written since it started and
throws a PANIC if so. At that point, only walsenders are still
active, so one might think this could not happen, but walsenders can
also generate WAL, for instance in BASE_BACKUP and logical decoding
related commands (e.g. via hint bits). So they can trigger this panic
if such a command is run while the shutdown checkpoint is being
written.
To fix this, divide the walsender shutdown into two phases. First,
checkpointer, itself triggered by postmaster, sends a
PROCSIG_WALSND_INIT_STOPPING signal to all walsenders. If the backend
is idle or runs an SQL query this causes the backend to shutdown, if
logical replication is in progress all existing WAL records are
processed followed by a shutdown. Otherwise this causes the walsender
to switch to the "stopping" state. In this state, the walsender will
reject any further replication commands. The checkpointer begins the
shutdown checkpoint once all walsenders are confirmed as
stopping. When the shutdown checkpoint finishes, the postmaster sends
us SIGUSR2. This instructs walsender to send any outstanding WAL,
including the shutdown checkpoint record, wait for it to be replicated
to the standby, and then exit.
Author: Andres Freund, based on an earlier patch by Michael Paquier
Reported-By: Fujii Masao, Andres Freund
Reviewed-By: Michael Paquier
Discussion: https://postgr.es/m/20170602002912.tqlwn4gymzlxpvs2@alap3.anarazel.de
Backpatch: 9.4, where logical decoding was introduced
2017-06-06 03:53:41 +02:00
|
|
|
WALSNDSTATE_STREAMING,
|
|
|
|
|
WALSNDSTATE_STOPPING
|
2011-09-12 20:24:29 +02:00
|
|
|
} WalSndState;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Each walsender has a WalSnd struct in shared memory.
|
Fix locking in WAL receiver/sender shmem state structs
In WAL receiver and WAL server, some accesses to their corresponding
shared memory control structs were done without holding any kind of
lock, which could lead to inconsistent and possibly insecure results.
In walsender, fix by clarifying the locking rules and following them
correctly, as documented in the new comment in walsender_private.h;
namely that some members can be read in walsender itself without a lock,
because the only writes occur in the same process. The rest of the
struct requires spinlock for accesses, as usual.
In walreceiver, fix by always holding spinlock while accessing the
struct.
While there is potentially a problem in all branches, it is minor in
stable ones. This only became a real problem in pg10 because of quorum
commit in synchronous replication (commit 3901fd70cc7c), and a potential
security problem in walreceiver because a superuser() check was removed
by default monitoring roles (commit 25fff40798fc). Thus, no backpatch.
In passing, clean up some leftover braces which were used to create
unconditional blocks. Once upon a time these were used for
volatile-izing accesses to those shmem structs, which is no longer
required. Many other occurrences of this pattern remain.
Author: Michaël Paquier
Reported-by: Michaël Paquier
Reviewed-by: Masahiko Sawada, Kyotaro Horiguchi, Thomas Munro,
Robert Haas
Discussion: https://postgr.es/m/CAB7nPqTWYqtzD=LN_oDaf9r-hAjUEPAy0B9yRkhcsLdRN8fzrw@mail.gmail.com
2017-07-01 00:06:33 +02:00
|
|
|
*
|
Fix race conditions in synchronous standby management.
We have repeatedly seen the buildfarm reach the Assert(false) in
SyncRepGetSyncStandbysPriority. This apparently is due to failing to
consider the possibility that the sync_standby_priority values in
shared memory might be inconsistent; but they will be whenever only
some of the walsenders have updated their values after a change in
the synchronous_standby_names setting. That function is vastly too
complex for what it does, anyway, so rewriting it seems better than
trying to apply a band-aid fix.
Furthermore, the API of SyncRepGetSyncStandbys is broken by design:
it returns a list of WalSnd array indexes, but there is nothing
guaranteeing that the contents of the WalSnd array remain stable.
Thus, if some walsender exits and then a new walsender process
takes over that WalSnd array slot, a caller might make use of
WAL position data that it should not, potentially leading to
incorrect decisions about whether to release transactions that
are waiting for synchronous commit.
To fix, replace SyncRepGetSyncStandbys with a new function
SyncRepGetCandidateStandbys that copies all the required data
from shared memory while holding the relevant mutexes. If the
associated walsender process then exits, this data is still safe to
make release decisions with, since we know that that much WAL *was*
sent to a valid standby server. This incidentally means that we no
longer need to treat sync_standby_priority as protected by the
SyncRepLock rather than the per-walsender mutex.
SyncRepGetSyncStandbys is no longer used by the core code, so remove
it entirely in HEAD. However, it seems possible that external code is
relying on that function, so do not remove it from the back branches.
Instead, just remove the known-incorrect Assert. When the bug occurs,
the function will return a too-short list, which callers should treat
as meaning there are not enough sync standbys, which seems like a
reasonably safe fallback until the inconsistent state is resolved.
Moreover it's bug-compatible with what has been happening in non-assert
builds. We cannot do anything about the walsender-replacement race
condition without an API/ABI break.
The bogus assertion exists back to 9.6, but 9.6 is sufficiently
different from the later branches that the patch doesn't apply at all.
I chose to just remove the bogus assertion in 9.6, feeling that the
probability of a bad outcome from the walsender-replacement race
condition is too low to justify rewriting the whole patch for 9.6.
Discussion: https://postgr.es/m/21519.1585272409@sss.pgh.pa.us
2020-04-18 20:02:44 +02:00
|
|
|
* This struct is protected by its 'mutex' spinlock field, except that some
|
Fix locking in WAL receiver/sender shmem state structs
In WAL receiver and WAL server, some accesses to their corresponding
shared memory control structs were done without holding any kind of
lock, which could lead to inconsistent and possibly insecure results.
In walsender, fix by clarifying the locking rules and following them
correctly, as documented in the new comment in walsender_private.h;
namely that some members can be read in walsender itself without a lock,
because the only writes occur in the same process. The rest of the
struct requires spinlock for accesses, as usual.
In walreceiver, fix by always holding spinlock while accessing the
struct.
While there is potentially a problem in all branches, it is minor in
stable ones. This only became a real problem in pg10 because of quorum
commit in synchronous replication (commit 3901fd70cc7c), and a potential
security problem in walreceiver because a superuser() check was removed
by default monitoring roles (commit 25fff40798fc). Thus, no backpatch.
In passing, clean up some leftover braces which were used to create
unconditional blocks. Once upon a time these were used for
volatile-izing accesses to those shmem structs, which is no longer
required. Many other occurrences of this pattern remain.
Author: Michaël Paquier
Reported-by: Michaël Paquier
Reviewed-by: Masahiko Sawada, Kyotaro Horiguchi, Thomas Munro,
Robert Haas
Discussion: https://postgr.es/m/CAB7nPqTWYqtzD=LN_oDaf9r-hAjUEPAy0B9yRkhcsLdRN8fzrw@mail.gmail.com
2017-07-01 00:06:33 +02:00
|
|
|
* members are only written by the walsender process itself, and thus that
|
|
|
|
|
* process is free to read those members without holding spinlock. pid and
|
|
|
|
|
* needreload always require the spinlock to be held for all accesses.
|
2011-09-12 20:24:29 +02:00
|
|
|
*/
|
|
|
|
|
typedef struct WalSnd
|
|
|
|
|
{
|
Fix locking in WAL receiver/sender shmem state structs
In WAL receiver and WAL server, some accesses to their corresponding
shared memory control structs were done without holding any kind of
lock, which could lead to inconsistent and possibly insecure results.
In walsender, fix by clarifying the locking rules and following them
correctly, as documented in the new comment in walsender_private.h;
namely that some members can be read in walsender itself without a lock,
because the only writes occur in the same process. The rest of the
struct requires spinlock for accesses, as usual.
In walreceiver, fix by always holding spinlock while accessing the
struct.
While there is potentially a problem in all branches, it is minor in
stable ones. This only became a real problem in pg10 because of quorum
commit in synchronous replication (commit 3901fd70cc7c), and a potential
security problem in walreceiver because a superuser() check was removed
by default monitoring roles (commit 25fff40798fc). Thus, no backpatch.
In passing, clean up some leftover braces which were used to create
unconditional blocks. Once upon a time these were used for
volatile-izing accesses to those shmem structs, which is no longer
required. Many other occurrences of this pattern remain.
Author: Michaël Paquier
Reported-by: Michaël Paquier
Reviewed-by: Masahiko Sawada, Kyotaro Horiguchi, Thomas Munro,
Robert Haas
Discussion: https://postgr.es/m/CAB7nPqTWYqtzD=LN_oDaf9r-hAjUEPAy0B9yRkhcsLdRN8fzrw@mail.gmail.com
2017-07-01 00:06:33 +02:00
|
|
|
pid_t pid; /* this walsender's PID, or 0 if not active */
|
|
|
|
|
|
2011-09-12 20:24:29 +02:00
|
|
|
WalSndState state; /* this walsender's state */
|
|
|
|
|
XLogRecPtr sentPtr; /* WAL has been sent up to this point */
|
|
|
|
|
bool needreload; /* does currently-open file need to be
|
|
|
|
|
* reloaded? */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The xlog locations that have been written, flushed, and applied by
|
|
|
|
|
* standby-side. These may be invalid if the standby-side has not offered
|
|
|
|
|
* values yet.
|
|
|
|
|
*/
|
|
|
|
|
XLogRecPtr write;
|
|
|
|
|
XLogRecPtr flush;
|
|
|
|
|
XLogRecPtr apply;
|
|
|
|
|
|
2017-03-23 15:05:28 +01:00
|
|
|
/* Measured lag times, or -1 for unknown/none. */
|
|
|
|
|
TimeOffset writeLag;
|
|
|
|
|
TimeOffset flushLag;
|
|
|
|
|
TimeOffset applyLag;
|
|
|
|
|
|
Fix race conditions in synchronous standby management.
We have repeatedly seen the buildfarm reach the Assert(false) in
SyncRepGetSyncStandbysPriority. This apparently is due to failing to
consider the possibility that the sync_standby_priority values in
shared memory might be inconsistent; but they will be whenever only
some of the walsenders have updated their values after a change in
the synchronous_standby_names setting. That function is vastly too
complex for what it does, anyway, so rewriting it seems better than
trying to apply a band-aid fix.
Furthermore, the API of SyncRepGetSyncStandbys is broken by design:
it returns a list of WalSnd array indexes, but there is nothing
guaranteeing that the contents of the WalSnd array remain stable.
Thus, if some walsender exits and then a new walsender process
takes over that WalSnd array slot, a caller might make use of
WAL position data that it should not, potentially leading to
incorrect decisions about whether to release transactions that
are waiting for synchronous commit.
To fix, replace SyncRepGetSyncStandbys with a new function
SyncRepGetCandidateStandbys that copies all the required data
from shared memory while holding the relevant mutexes. If the
associated walsender process then exits, this data is still safe to
make release decisions with, since we know that that much WAL *was*
sent to a valid standby server. This incidentally means that we no
longer need to treat sync_standby_priority as protected by the
SyncRepLock rather than the per-walsender mutex.
SyncRepGetSyncStandbys is no longer used by the core code, so remove
it entirely in HEAD. However, it seems possible that external code is
relying on that function, so do not remove it from the back branches.
Instead, just remove the known-incorrect Assert. When the bug occurs,
the function will return a too-short list, which callers should treat
as meaning there are not enough sync standbys, which seems like a
reasonably safe fallback until the inconsistent state is resolved.
Moreover it's bug-compatible with what has been happening in non-assert
builds. We cannot do anything about the walsender-replacement race
condition without an API/ABI break.
The bogus assertion exists back to 9.6, but 9.6 is sufficiently
different from the later branches that the patch doesn't apply at all.
I chose to just remove the bogus assertion in 9.6, feeling that the
probability of a bad outcome from the walsender-replacement race
condition is too low to justify rewriting the whole patch for 9.6.
Discussion: https://postgr.es/m/21519.1585272409@sss.pgh.pa.us
2020-04-18 20:02:44 +02:00
|
|
|
/*
|
|
|
|
|
* The priority order of the standby managed by this WALSender, as listed
|
|
|
|
|
* in synchronous_standby_names, or 0 if not-listed.
|
|
|
|
|
*/
|
|
|
|
|
int sync_standby_priority;
|
|
|
|
|
|
2022-08-22 03:02:53 +02:00
|
|
|
/* Protects shared variables in this structure. */
|
2011-09-12 20:24:29 +02:00
|
|
|
slock_t mutex;
|
|
|
|
|
|
|
|
|
|
/*
|
2015-01-17 13:00:42 +01:00
|
|
|
* Pointer to the walsender's latch. Used by backends to wake up this
|
|
|
|
|
* walsender when it has work to do. NULL if the walsender isn't active.
|
2011-09-12 20:24:29 +02:00
|
|
|
*/
|
2015-02-20 06:11:42 +01:00
|
|
|
Latch *latch;
|
2011-09-12 20:24:29 +02:00
|
|
|
|
2018-12-09 08:35:06 +01:00
|
|
|
/*
|
|
|
|
|
* Timestamp of the last message received from standby.
|
|
|
|
|
*/
|
|
|
|
|
TimestampTz replyTime;
|
2023-04-08 09:24:24 +02:00
|
|
|
|
|
|
|
|
ReplicationKind kind;
|
2011-09-12 20:24:29 +02:00
|
|
|
} WalSnd;
|
|
|
|
|
|
2022-04-08 14:16:38 +02:00
|
|
|
extern PGDLLIMPORT WalSnd *MyWalSnd;
|
2011-09-12 20:24:29 +02:00
|
|
|
|
|
|
|
|
/* There is one WalSndCtl struct for the whole database cluster */
|
|
|
|
|
typedef struct
|
|
|
|
|
{
|
|
|
|
|
/*
|
2012-01-24 21:22:37 +01:00
|
|
|
* Synchronous replication queue with one queue per request type.
|
|
|
|
|
* Protected by SyncRepLock.
|
2011-09-12 20:24:29 +02:00
|
|
|
*/
|
2023-01-18 21:15:05 +01:00
|
|
|
dlist_head SyncRepQueue[NUM_SYNC_REP_WAIT_MODE];
|
2011-09-12 20:24:29 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Current location of the head of the queue. All waiters should have a
|
|
|
|
|
* waitLSN that follows this value. Protected by SyncRepLock.
|
|
|
|
|
*/
|
2012-01-24 21:22:37 +01:00
|
|
|
XLogRecPtr lsn[NUM_SYNC_REP_WAIT_MODE];
|
2011-09-12 20:24:29 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Are any sync standbys defined? Waiting backends can't reload the
|
2012-01-30 15:34:25 +01:00
|
|
|
* config file safely, so checkpointer updates this value as needed.
|
2011-09-12 20:24:29 +02:00
|
|
|
* Protected by SyncRepLock.
|
|
|
|
|
*/
|
|
|
|
|
bool sync_standbys_defined;
|
|
|
|
|
|
Optimize walsender wake up logic using condition variables
WalSndWakeup() currently loops through all the walsenders slots, with a
spinlock acquisition and release for every iteration, to wake up waiting
walsenders.
This commonly was not a problem before e101dfac3a53c. But, to allow logical
decoding on standbys, we need to wake up logical walsenders after every WAL
record is applied on the standby, rather just when flushing WAL or switching
timelines. This causes a performance regression for workloads replaying a lot
of WAL records.
To solve this, we use condition variable (CV) to efficiently wake up
walsenders in WalSndWakeup().
Every walsender prepares to sleep on a shared memory CV. Note that it just
prepares to sleep on the CV (i.e., adds itself to the CV's waitlist), but does
not actually wait on the CV (IOW, it never calls ConditionVariableSleep()). It
still uses WaitEventSetWait() for waiting, because CV infrastructure doesn't
handle FeBe socket events currently. The processes (startup process,
walreceiver etc.) wanting to wake up walsenders use
ConditionVariableBroadcast(), which in turn calls SetLatch(), helping
walsenders come out of WaitEventSetWait().
We use separate shared memory CVs for physical and logical walsenders for
selective wake ups, see WalSndWakeup() for more details.
This approach is simple and reasonably efficient. But not very elegant. But
for 16 it seems to be a better path than a larger redesign of the CV
mechanism. A desirable future improvement would be to add support for CVs
into WaitEventSetWait().
This still leaves us with a small regression in very extreme workloads (due to
the spinlock acquisition in ConditionVariableBroadcast() when there are no
waiters) - but that seems acceptable.
Reported-by: Andres Freund <andres@anarazel.de>
Suggested-by: Andres Freund <andres@anarazel.de>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Discussion: https://www.postgresql.org/message-id/20230509190247.3rrplhdgem6su6cg%40awork3.anarazel.de
2023-05-21 18:44:55 +02:00
|
|
|
/* used as a registry of physical / logical walsenders to wake */
|
|
|
|
|
ConditionVariable wal_flush_cv;
|
|
|
|
|
ConditionVariable wal_replay_cv;
|
|
|
|
|
|
2015-02-20 06:11:42 +01:00
|
|
|
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
|
2011-09-12 20:24:29 +02:00
|
|
|
} WalSndCtlData;
|
|
|
|
|
|
2022-04-08 14:16:38 +02:00
|
|
|
extern PGDLLIMPORT WalSndCtlData *WalSndCtl;
|
2011-09-12 20:24:29 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
extern void WalSndSetState(WalSndState state);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Internal functions for parsing the replication grammar, in repl_gram.y and
|
|
|
|
|
* repl_scanner.l
|
|
|
|
|
*/
|
|
|
|
|
extern int replication_yyparse(void);
|
|
|
|
|
extern int replication_yylex(void);
|
2022-09-22 22:27:16 +02:00
|
|
|
extern void replication_yyerror(const char *message) pg_attribute_noreturn();
|
2022-09-21 22:21:36 +02:00
|
|
|
extern void replication_scanner_init(const char *str);
|
2011-09-12 20:24:29 +02:00
|
|
|
extern void replication_scanner_finish(void);
|
Fix limitations on what SQL commands can be issued to a walsender.
In logical replication mode, a WalSender is supposed to be able
to execute any regular SQL command, as well as the special
replication commands. Poor design of the replication-command
parser caused it to fail in various cases, notably:
* semicolons embedded in a command, or multiple SQL commands
sent in a single message;
* dollar-quoted literals containing odd numbers of single
or double quote marks;
* commands starting with a comment.
The basic problem here is that we're trying to run repl_scanner.l
across the entire input string even when it's not a replication
command. Since repl_scanner.l does not understand all of the
token types known to the core lexer, this is doomed to have
failure modes.
We certainly don't want to make repl_scanner.l as big as scan.l,
so instead rejigger stuff so that we only lex the first token of
a non-replication command. That will usually look like an IDENT
to repl_scanner.l, though a comment would end up getting reported
as a '-' or '/' single-character token. If the token is a replication
command keyword, we push it back and proceed normally with repl_gram.y
parsing. Otherwise, we can drop out of exec_replication_command()
without examining the rest of the string.
(It's still theoretically possible for repl_scanner.l to fail on
the first token; but that could only happen if it's an unterminated
single- or double-quoted string, in which case you'd have gotten
largely the same error from the core lexer too.)
In this way, repl_gram.y isn't involved at all in handling general
SQL commands, so we can get rid of the SQLCmd node type. (In
the back branches, we can't remove it because renumbering enum
NodeTag would be an ABI break; so just leave it sit there unused.)
I failed to resist the temptation to clean up some other sloppy
coding in repl_scanner.l while at it. The only externally-visible
behavior change from that is it now accepts \r and \f as whitespace,
same as the core lexer.
Per bug #17379 from Greg Rychlewski. Back-patch to all supported
branches.
Discussion: https://postgr.es/m/17379-6a5c6cfb3f1f5e77@postgresql.org
2022-01-24 21:33:34 +01:00
|
|
|
extern bool replication_scanner_is_replication_command(void);
|
2011-09-12 20:24:29 +02:00
|
|
|
|
2022-04-08 14:16:38 +02:00
|
|
|
extern PGDLLIMPORT Node *replication_parse_result;
|
2011-09-12 20:24:29 +02:00
|
|
|
|
|
|
|
|
#endif /* _WALSENDER_PRIVATE_H */
|
