2013-03-24 16:27:20 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* parallel.c
|
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Parallel support for pg_dump and pg_restore
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
2016-01-02 19:33:40 +01:00
|
|
|
* Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
|
2013-03-24 16:27:20 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
|
|
|
* src/bin/pg_dump/parallel.c
|
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Parallel operation works like this:
|
|
|
|
*
|
|
|
|
* The original, master process calls ParallelBackupStart(), which forks off
|
|
|
|
* the desired number of worker processes, which each enter WaitForCommands().
|
|
|
|
*
|
|
|
|
* The master process dispatches an individual work item to one of the worker
|
|
|
|
* processes in DispatchJobForTocEntry(). That calls
|
|
|
|
* AH->MasterStartParallelItemPtr, a routine of the output format. This
|
|
|
|
* function's arguments are the parents archive handle AH (containing the full
|
|
|
|
* catalog information), the TocEntry that the worker should work on and a
|
|
|
|
* T_Action value indicating whether this is a backup or a restore task. The
|
|
|
|
* function simply converts the TocEntry assignment into a command string that
|
|
|
|
* is then sent over to the worker process. In the simplest case that would be
|
|
|
|
* something like "DUMP 1234", with 1234 being the TocEntry id.
|
|
|
|
*
|
|
|
|
* The worker process receives and decodes the command and passes it to the
|
|
|
|
* routine pointed to by AH->WorkerJobDumpPtr or AH->WorkerJobRestorePtr,
|
|
|
|
* which are routines of the current archive format. That routine performs
|
|
|
|
* the required action (dump or restore) and returns a malloc'd status string.
|
|
|
|
* The status string is passed back to the master where it is interpreted by
|
|
|
|
* AH->MasterEndParallelItemPtr, another format-specific routine. That
|
|
|
|
* function can update state or catalog information on the master's side,
|
|
|
|
* depending on the reply from the worker process. In the end it returns a
|
|
|
|
* status code, which is 0 for successful execution.
|
|
|
|
*
|
|
|
|
* Remember that we have forked off the workers only after we have read in
|
|
|
|
* the catalog. That's why our worker processes can also access the catalog
|
|
|
|
* information. (In the Windows case, the workers are threads in the same
|
|
|
|
* process. To avoid problems, they work with cloned copies of the Archive
|
|
|
|
* data structure; see init_spawned_worker_win32().)
|
|
|
|
*
|
|
|
|
* In the master process, the workerStatus field for each worker has one of
|
|
|
|
* the following values:
|
|
|
|
* WRKR_IDLE: it's waiting for a command
|
|
|
|
* WRKR_WORKING: it's been sent a command
|
|
|
|
* WRKR_FINISHED: it's returned a result
|
|
|
|
* WRKR_TERMINATED: process ended
|
|
|
|
* The FINISHED state indicates that the worker is idle, but we've not yet
|
|
|
|
* dealt with the status code it returned from the prior command.
|
|
|
|
* ReapWorkerStatus() extracts the unhandled command status value and sets
|
|
|
|
* the workerStatus back to WRKR_IDLE.
|
|
|
|
*/
|
|
|
|
|
2013-03-27 17:10:40 +01:00
|
|
|
#include "postgres_fe.h"
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
#include "parallel.h"
|
2014-10-14 20:00:55 +02:00
|
|
|
#include "pg_backup_utils.h"
|
2016-03-24 20:55:44 +01:00
|
|
|
#include "fe_utils/string_utils.h"
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
#ifndef WIN32
|
|
|
|
#include <sys/types.h>
|
|
|
|
#include <sys/wait.h>
|
|
|
|
#include "signal.h"
|
|
|
|
#include <unistd.h>
|
|
|
|
#include <fcntl.h>
|
|
|
|
#endif
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Mnemonic macros for indexing the fd array returned by pipe(2) */
|
2013-03-24 16:27:20 +01:00
|
|
|
#define PIPE_READ 0
|
|
|
|
#define PIPE_WRITE 1
|
|
|
|
|
|
|
|
#ifdef WIN32
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Structure to hold info passed by _beginthreadex() to the function it calls
|
|
|
|
* via its single allowed argument.
|
|
|
|
*/
|
|
|
|
typedef struct
|
|
|
|
{
|
|
|
|
ArchiveHandle *AH;
|
|
|
|
int pipeRead;
|
|
|
|
int pipeWrite;
|
|
|
|
} WorkerInfo;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Windows implementation of pipe access */
|
|
|
|
static int pgpipe(int handles[2]);
|
|
|
|
static int piperead(int s, char *buf, int len);
|
2013-03-24 16:27:20 +01:00
|
|
|
#define pipewrite(a,b,c) send(a,b,c,0)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
#else /* !WIN32 */
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Variables for handling signals. aborting is only ever used in the master,
|
|
|
|
* the workers just need wantAbort.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static bool aborting = false;
|
|
|
|
static volatile sig_atomic_t wantAbort = 0;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Non-Windows implementation of pipe access */
|
2013-03-24 16:27:20 +01:00
|
|
|
#define pgpipe(a) pipe(a)
|
|
|
|
#define piperead(a,b,c) read(a,b,c)
|
|
|
|
#define pipewrite(a,b,c) write(a,b,c)
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* State info for archive_close_connection() shutdown callback.
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
typedef struct ShutdownInformation
|
|
|
|
{
|
|
|
|
ParallelState *pstate;
|
|
|
|
Archive *AHX;
|
|
|
|
} ShutdownInformation;
|
|
|
|
|
|
|
|
static ShutdownInformation shutdown_info;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#ifdef WIN32
|
|
|
|
/* file-scope variables */
|
|
|
|
static unsigned int tMasterThreadId = 0;
|
|
|
|
static HANDLE termEvent = INVALID_HANDLE_VALUE;
|
|
|
|
static DWORD tls_index;
|
|
|
|
|
|
|
|
/* globally visible variables (needed by exit_nicely) */
|
|
|
|
bool parallel_init_done = false;
|
|
|
|
DWORD mainThreadId;
|
|
|
|
#endif /* WIN32 */
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
static const char *modulename = gettext_noop("parallel archiver");
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Local function prototypes */
|
2013-03-24 16:27:20 +01:00
|
|
|
static ParallelSlot *GetMyPSlot(ParallelState *pstate);
|
|
|
|
static void archive_close_connection(int code, void *arg);
|
|
|
|
static void ShutdownWorkersHard(ParallelState *pstate);
|
|
|
|
static void WaitForTerminatingWorkers(ParallelState *pstate);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
static void RunWorker(ArchiveHandle *AH, int pipefd[2]);
|
2013-03-24 16:27:20 +01:00
|
|
|
static bool HasEveryWorkerTerminated(ParallelState *pstate);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
static void lockTableForWorker(ArchiveHandle *AH, TocEntry *te);
|
2016-01-13 23:48:33 +01:00
|
|
|
static void WaitForCommands(ArchiveHandle *AH, int pipefd[2]);
|
2013-03-24 16:27:20 +01:00
|
|
|
static char *getMessageFromMaster(int pipefd[2]);
|
|
|
|
static void sendMessageToMaster(int pipefd[2], const char *str);
|
|
|
|
static int select_loop(int maxFd, fd_set *workerset);
|
|
|
|
static char *getMessageFromWorker(ParallelState *pstate,
|
|
|
|
bool do_wait, int *worker);
|
|
|
|
static void sendMessageToWorker(ParallelState *pstate,
|
|
|
|
int worker, const char *str);
|
|
|
|
static char *readMessageFromPipe(int fd);
|
|
|
|
|
|
|
|
#define messageStartsWith(msg, prefix) \
|
|
|
|
(strncmp(msg, prefix, strlen(prefix)) == 0)
|
|
|
|
#define messageEquals(msg, pattern) \
|
|
|
|
(strcmp(msg, pattern) == 0)
|
|
|
|
|
2013-03-27 17:10:40 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Shutdown callback to clean up socket access
|
|
|
|
*/
|
2013-03-27 17:10:40 +01:00
|
|
|
#ifdef WIN32
|
|
|
|
static void
|
|
|
|
shutdown_parallel_dump_utils(int code, void *unused)
|
|
|
|
{
|
|
|
|
/* Call the cleanup function only from the main thread */
|
|
|
|
if (mainThreadId == GetCurrentThreadId())
|
|
|
|
WSACleanup();
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Initialize parallel dump support --- should be called early in process
|
|
|
|
* startup. (Currently, this is called whether or not we intend parallel
|
|
|
|
* activity.)
|
|
|
|
*/
|
2013-03-27 17:10:40 +01:00
|
|
|
void
|
|
|
|
init_parallel_dump_utils(void)
|
|
|
|
{
|
|
|
|
#ifdef WIN32
|
|
|
|
if (!parallel_init_done)
|
|
|
|
{
|
|
|
|
WSADATA wsaData;
|
|
|
|
int err;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Prepare for threaded operation */
|
2013-03-27 17:10:40 +01:00
|
|
|
tls_index = TlsAlloc();
|
|
|
|
mainThreadId = GetCurrentThreadId();
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* Initialize socket access */
|
2013-03-27 17:10:40 +01:00
|
|
|
err = WSAStartup(MAKEWORD(2, 2), &wsaData);
|
|
|
|
if (err != 0)
|
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
fprintf(stderr, _("%s: WSAStartup failed: %d\n"), progname, err);
|
2013-03-27 17:10:40 +01:00
|
|
|
exit_nicely(1);
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* ... and arrange to shut it down at exit */
|
2013-03-27 17:10:40 +01:00
|
|
|
on_exit_nicely(shutdown_parallel_dump_utils, NULL);
|
|
|
|
parallel_init_done = true;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Find the ParallelSlot for the current worker process or thread.
|
|
|
|
*
|
|
|
|
* Returns NULL if no matching slot is found (this implies we're the master).
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
static ParallelSlot *
|
|
|
|
GetMyPSlot(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
#ifdef WIN32
|
|
|
|
if (pstate->parallelSlot[i].threadId == GetCurrentThreadId())
|
|
|
|
#else
|
|
|
|
if (pstate->parallelSlot[i].pid == getpid())
|
|
|
|
#endif
|
|
|
|
return &(pstate->parallelSlot[i]);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2013-03-27 17:10:40 +01:00
|
|
|
/*
|
|
|
|
* A thread-local version of getLocalPQExpBuffer().
|
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Non-reentrant but reduces memory leakage: we'll consume one buffer per
|
|
|
|
* thread, which is much better than one per fmtId/fmtQualifiedId call.
|
2013-03-27 17:10:40 +01:00
|
|
|
*/
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#ifdef WIN32
|
2013-03-27 17:10:40 +01:00
|
|
|
static PQExpBuffer
|
|
|
|
getThreadLocalPQExpBuffer(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* The Tls code goes awry if we use a static var, so we provide for both
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* static and auto, and omit any use of the static var when using Tls. We
|
|
|
|
* rely on TlsGetValue() to return 0 if the value is not yet set.
|
2013-03-27 17:10:40 +01:00
|
|
|
*/
|
|
|
|
static PQExpBuffer s_id_return = NULL;
|
|
|
|
PQExpBuffer id_return;
|
|
|
|
|
|
|
|
if (parallel_init_done)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
id_return = (PQExpBuffer) TlsGetValue(tls_index);
|
2013-03-27 17:10:40 +01:00
|
|
|
else
|
|
|
|
id_return = s_id_return;
|
|
|
|
|
|
|
|
if (id_return) /* first time through? */
|
|
|
|
{
|
|
|
|
/* same buffer, just wipe contents */
|
|
|
|
resetPQExpBuffer(id_return);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* new buffer */
|
|
|
|
id_return = createPQExpBuffer();
|
|
|
|
if (parallel_init_done)
|
|
|
|
TlsSetValue(tls_index, id_return);
|
|
|
|
else
|
|
|
|
s_id_return = id_return;
|
|
|
|
}
|
|
|
|
|
|
|
|
return id_return;
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|
2013-03-27 17:10:40 +01:00
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* pg_dump and pg_restore call this to register the cleanup handler
|
|
|
|
* as soon as they've created the ArchiveHandle.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
on_exit_close_archive(Archive *AHX)
|
|
|
|
{
|
|
|
|
shutdown_info.AHX = AHX;
|
|
|
|
on_exit_nicely(archive_close_connection, &shutdown_info);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
* on_exit_nicely handler for shutting down database connections and
|
|
|
|
* worker processes cleanly.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
archive_close_connection(int code, void *arg)
|
|
|
|
{
|
|
|
|
ShutdownInformation *si = (ShutdownInformation *) arg;
|
|
|
|
|
|
|
|
if (si->pstate)
|
|
|
|
{
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
/* In parallel mode, must figure out who we are */
|
2013-03-24 16:27:20 +01:00
|
|
|
ParallelSlot *slot = GetMyPSlot(si->pstate);
|
|
|
|
|
|
|
|
if (!slot)
|
|
|
|
{
|
|
|
|
/*
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
* We're the master. Close our own database connection, if any,
|
|
|
|
* and then forcibly shut down workers.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
if (si->AHX)
|
|
|
|
DisconnectDatabase(si->AHX);
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
#ifndef WIN32
|
|
|
|
|
|
|
|
/*
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
* Setting aborting to true shuts off error/warning messages that
|
|
|
|
* are no longer useful once we start killing workers.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
aborting = true;
|
|
|
|
#endif
|
|
|
|
ShutdownWorkersHard(si->pstate);
|
|
|
|
}
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We're a worker. Shut down our own DB connection if any. On
|
|
|
|
* Windows, we also have to close our communication sockets, to
|
|
|
|
* emulate what will happen on Unix when the worker process exits.
|
|
|
|
* (Without this, if this is a premature exit, the master would
|
|
|
|
* fail to detect it because there would be no EOF condition on
|
|
|
|
* the other end of the pipe.)
|
|
|
|
*/
|
|
|
|
if (slot->args->AH)
|
|
|
|
DisconnectDatabase(&(slot->args->AH->public));
|
|
|
|
|
|
|
|
#ifdef WIN32
|
|
|
|
closesocket(slot->pipeRevRead);
|
|
|
|
closesocket(slot->pipeRevWrite);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Non-parallel operation: just kill the master DB connection */
|
|
|
|
if (si->AHX)
|
|
|
|
DisconnectDatabase(si->AHX);
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Check to see if we've been told to abort, and exit the process/thread if
|
|
|
|
* so. We don't print any error message; that would just clutter the screen.
|
|
|
|
*
|
2013-03-24 16:27:20 +01:00
|
|
|
* If we have one worker that terminates for some reason, we'd like the other
|
|
|
|
* threads to terminate as well (and not finish with their 70 GB table dump
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* first...). In Unix, the master sends SIGTERM and the worker's signal
|
|
|
|
* handler sets wantAbort to 1. In Windows we set a termEvent and this serves
|
|
|
|
* as the signal for worker threads to exit. Note that while we check this
|
|
|
|
* fairly frequently during data transfers, an idle worker doesn't come here
|
|
|
|
* at all, so additional measures are needed to force shutdown.
|
|
|
|
*
|
|
|
|
* XXX in parallel restore, slow server-side operations like CREATE INDEX
|
|
|
|
* are not interrupted by anything we do here. This needs more work.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
checkAborting(ArchiveHandle *AH)
|
|
|
|
{
|
|
|
|
#ifdef WIN32
|
|
|
|
if (WaitForSingleObject(termEvent, 0) == WAIT_OBJECT_0)
|
|
|
|
#else
|
|
|
|
if (wantAbort)
|
|
|
|
#endif
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
exit_nicely(1);
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Forcibly shut down any remaining workers, waiting for them to finish.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
ShutdownWorkersHard(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
2016-05-26 16:50:30 +02:00
|
|
|
* Close our write end of the sockets so that any workers waiting for
|
|
|
|
* commands know they can exit.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
closesocket(pstate->parallelSlot[i].pipeWrite);
|
|
|
|
|
2016-05-26 16:50:30 +02:00
|
|
|
#ifndef WIN32
|
|
|
|
/* On non-Windows, send SIGTERM to abort commands-in-progress. */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
kill(pstate->parallelSlot[i].pid, SIGTERM);
|
|
|
|
#else
|
2016-05-26 16:50:30 +02:00
|
|
|
/* Non-idle workers monitor this event via checkAborting(). */
|
2013-03-24 16:27:20 +01:00
|
|
|
SetEvent(termEvent);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
WaitForTerminatingWorkers(pstate);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
* Wait for all workers to terminate.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
WaitForTerminatingWorkers(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
while (!HasEveryWorkerTerminated(pstate))
|
|
|
|
{
|
|
|
|
ParallelSlot *slot = NULL;
|
|
|
|
int j;
|
|
|
|
|
|
|
|
#ifndef WIN32
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
/* On non-Windows, use wait() to wait for next worker to end */
|
2013-03-24 16:27:20 +01:00
|
|
|
int status;
|
|
|
|
pid_t pid = wait(&status);
|
|
|
|
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
/* Find dead worker's slot, and clear the PID field */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (j = 0; j < pstate->numWorkers; j++)
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
{
|
|
|
|
slot = &(pstate->parallelSlot[j]);
|
|
|
|
if (slot->pid == pid)
|
|
|
|
{
|
|
|
|
slot->pid = 0;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#else /* WIN32 */
|
|
|
|
/* On Windows, we must use WaitForMultipleObjects() */
|
|
|
|
HANDLE *lpHandles = pg_malloc(sizeof(HANDLE) * pstate->numWorkers);
|
2013-03-24 16:27:20 +01:00
|
|
|
int nrun = 0;
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
DWORD ret;
|
|
|
|
uintptr_t hThread;
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
for (j = 0; j < pstate->numWorkers; j++)
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pstate->parallelSlot[j].workerStatus != WRKR_TERMINATED)
|
|
|
|
{
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
lpHandles[nrun] = (HANDLE) pstate->parallelSlot[j].hThread;
|
2013-03-24 16:27:20 +01:00
|
|
|
nrun++;
|
|
|
|
}
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
}
|
|
|
|
ret = WaitForMultipleObjects(nrun, lpHandles, false, INFINITE);
|
2013-03-24 16:27:20 +01:00
|
|
|
Assert(ret != WAIT_FAILED);
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
hThread = (uintptr_t) lpHandles[ret - WAIT_OBJECT_0];
|
|
|
|
free(lpHandles);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
/* Find dead worker's slot, and clear the hThread field */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (j = 0; j < pstate->numWorkers; j++)
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
{
|
|
|
|
slot = &(pstate->parallelSlot[j]);
|
|
|
|
if (slot->hThread == hThread)
|
|
|
|
{
|
|
|
|
/* For cleanliness, close handles for dead threads */
|
|
|
|
CloseHandle((HANDLE) slot->hThread);
|
|
|
|
slot->hThread = (uintptr_t) INVALID_HANDLE_VALUE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif /* WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Clean up thread management in parallel pg_dump for Windows.
Since we start the worker threads with _beginthreadex(), we should use
_endthreadex() to terminate them. We got this right in the normal-exit
code path, but not so much during an error exit from a worker.
In addition, be sure to apply CloseHandle to the thread handle after
each thread exits.
It's not clear that these oversights cause any user-visible problems,
since the pg_dump run is about to terminate anyway. Still, it's clearly
better to follow Microsoft's API specifications than ignore them.
Also a few cosmetic cleanups in WaitForTerminatingWorkers(), including
being a bit less random about where to cast between uintptr_t and HANDLE,
and being sure to clear the worker identity field for each dead worker
(not that false matches should be possible later, but let's be careful).
Original observation and patch by Armin Schöffmann, cosmetic improvements
by Michael Paquier and me. (Armin's patch also included closing sockets
in ShutdownWorkersHard(), but that's been dealt with already in commit
df8d2d8c4.) Back-patch to 9.3 where parallel pg_dump was introduced.
Discussion: <zarafa.570306bd.3418.074bf1420d8f2ba2@root.aegaeon.de>
2016-05-27 18:02:09 +02:00
|
|
|
/* On all platforms, update workerStatus as well */
|
|
|
|
Assert(j < pstate->numWorkers);
|
2013-03-24 16:27:20 +01:00
|
|
|
slot->workerStatus = WRKR_TERMINATED;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Signal handler (Unix only)
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
#ifndef WIN32
|
|
|
|
static void
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
sigTermHandler(SIGNAL_ARGS)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
|
|
|
wantAbort = 1;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is called by both Unix and Windows variants to set up
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
* and run a worker process. Caller should exit the process (or thread)
|
|
|
|
* upon return.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
RunWorker(ArchiveHandle *AH, int pipefd[2])
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Call the setup worker function that's defined in the ArchiveHandle.
|
|
|
|
*/
|
2016-01-13 23:48:33 +01:00
|
|
|
(AH->SetupWorkerPtr) ((Archive *) AH);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
Assert(AH->connection != NULL);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Execute commands until done.
|
|
|
|
*/
|
2016-01-13 23:48:33 +01:00
|
|
|
WaitForCommands(AH, pipefd);
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Thread base function for Windows
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
#ifdef WIN32
|
|
|
|
static unsigned __stdcall
|
|
|
|
init_spawned_worker_win32(WorkerInfo *wi)
|
|
|
|
{
|
|
|
|
ArchiveHandle *AH;
|
|
|
|
int pipefd[2] = {wi->pipeRead, wi->pipeWrite};
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Clone the archive so that we have our own state to work with, and in
|
|
|
|
* particular our own database connection.
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
AH = CloneArchive(wi->AH);
|
|
|
|
|
|
|
|
free(wi);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Run the worker ... */
|
|
|
|
RunWorker(AH, pipefd);
|
|
|
|
|
|
|
|
/* Clean up and exit the thread */
|
2013-03-24 16:27:20 +01:00
|
|
|
DeCloneArchive(AH);
|
|
|
|
_endthreadex(0);
|
|
|
|
return 0;
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function starts a parallel dump or restore by spawning off the worker
|
|
|
|
* processes. For Windows, it creates a number of threads; on Unix the
|
|
|
|
* workers are created with fork().
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
ParallelState *
|
2016-01-13 23:48:33 +01:00
|
|
|
ParallelBackupStart(ArchiveHandle *AH)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
|
|
|
ParallelState *pstate;
|
|
|
|
int i;
|
|
|
|
const size_t slotSize = AH->public.numWorkers * sizeof(ParallelSlot);
|
|
|
|
|
|
|
|
Assert(AH->public.numWorkers > 0);
|
|
|
|
|
|
|
|
/* Ensure stdio state is quiesced before forking */
|
|
|
|
fflush(NULL);
|
|
|
|
|
|
|
|
pstate = (ParallelState *) pg_malloc(sizeof(ParallelState));
|
|
|
|
|
|
|
|
pstate->numWorkers = AH->public.numWorkers;
|
|
|
|
pstate->parallelSlot = NULL;
|
|
|
|
|
|
|
|
if (AH->public.numWorkers == 1)
|
|
|
|
return pstate;
|
|
|
|
|
|
|
|
pstate->parallelSlot = (ParallelSlot *) pg_malloc(slotSize);
|
|
|
|
memset((void *) pstate->parallelSlot, 0, slotSize);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Set the pstate in the shutdown_info. The exit handler uses pstate if
|
|
|
|
* set and falls back to AHX otherwise.
|
|
|
|
*/
|
|
|
|
shutdown_info.pstate = pstate;
|
|
|
|
|
|
|
|
#ifdef WIN32
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Set up thread management state */
|
2013-03-24 16:27:20 +01:00
|
|
|
tMasterThreadId = GetCurrentThreadId();
|
|
|
|
termEvent = CreateEvent(NULL, true, false, "Terminate");
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Make fmtId() and fmtQualifiedId() use thread-local storage */
|
|
|
|
getLocalPQExpBuffer = getThreadLocalPQExpBuffer;
|
2013-03-24 16:27:20 +01:00
|
|
|
#else
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Set up signal handling state */
|
2013-03-24 16:27:20 +01:00
|
|
|
signal(SIGTERM, sigTermHandler);
|
|
|
|
signal(SIGINT, sigTermHandler);
|
|
|
|
signal(SIGQUIT, sigTermHandler);
|
|
|
|
#endif
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Create desired number of workers */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
{
|
|
|
|
#ifdef WIN32
|
|
|
|
WorkerInfo *wi;
|
|
|
|
uintptr_t handle;
|
|
|
|
#else
|
|
|
|
pid_t pid;
|
|
|
|
#endif
|
|
|
|
int pipeMW[2],
|
|
|
|
pipeWM[2];
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Create communication pipes for this worker */
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pgpipe(pipeMW) < 0 || pgpipe(pipeWM) < 0)
|
|
|
|
exit_horribly(modulename,
|
2013-07-12 03:48:09 +02:00
|
|
|
"could not create communication channels: %s\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
strerror(errno));
|
|
|
|
|
|
|
|
pstate->parallelSlot[i].workerStatus = WRKR_IDLE;
|
|
|
|
pstate->parallelSlot[i].args = (ParallelArgs *) pg_malloc(sizeof(ParallelArgs));
|
|
|
|
pstate->parallelSlot[i].args->AH = NULL;
|
|
|
|
pstate->parallelSlot[i].args->te = NULL;
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
|
|
|
|
/* master's ends of the pipes */
|
|
|
|
pstate->parallelSlot[i].pipeRead = pipeWM[PIPE_READ];
|
|
|
|
pstate->parallelSlot[i].pipeWrite = pipeMW[PIPE_WRITE];
|
|
|
|
/* child's ends of the pipes */
|
|
|
|
pstate->parallelSlot[i].pipeRevRead = pipeMW[PIPE_READ];
|
|
|
|
pstate->parallelSlot[i].pipeRevWrite = pipeWM[PIPE_WRITE];
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
#ifdef WIN32
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Create transient structure to pass args to worker function */
|
2013-03-24 16:27:20 +01:00
|
|
|
wi = (WorkerInfo *) pg_malloc(sizeof(WorkerInfo));
|
|
|
|
|
|
|
|
wi->AH = AH;
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
wi->pipeRead = pipeMW[PIPE_READ];
|
|
|
|
wi->pipeWrite = pipeWM[PIPE_WRITE];
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
handle = _beginthreadex(NULL, 0, (void *) &init_spawned_worker_win32,
|
|
|
|
wi, 0, &(pstate->parallelSlot[i].threadId));
|
|
|
|
pstate->parallelSlot[i].hThread = handle;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#else /* !WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
pid = fork();
|
|
|
|
if (pid == 0)
|
|
|
|
{
|
|
|
|
/* we are the worker */
|
|
|
|
int j;
|
2014-04-04 17:26:01 +02:00
|
|
|
int pipefd[2];
|
|
|
|
|
|
|
|
pipefd[0] = pipeMW[PIPE_READ];
|
|
|
|
pipefd[1] = pipeWM[PIPE_WRITE];
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
pstate->parallelSlot[i].pid = getpid();
|
|
|
|
|
|
|
|
/* close read end of Worker -> Master */
|
|
|
|
closesocket(pipeWM[PIPE_READ]);
|
|
|
|
/* close write end of Master -> Worker */
|
|
|
|
closesocket(pipeMW[PIPE_WRITE]);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Close all inherited fds for communication of the master with
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
* previously-forked workers.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
for (j = 0; j < i; j++)
|
|
|
|
{
|
|
|
|
closesocket(pstate->parallelSlot[j].pipeRead);
|
|
|
|
closesocket(pstate->parallelSlot[j].pipeWrite);
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Call CloneArchive on Unix as well as Windows, even though
|
|
|
|
* technically we don't need to because fork() gives us a copy in
|
|
|
|
* our own address space already. But CloneArchive resets the
|
|
|
|
* state information and also clones the database connection which
|
|
|
|
* both seem kinda helpful.
|
|
|
|
*/
|
|
|
|
pstate->parallelSlot[i].args->AH = CloneArchive(AH);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Run the worker ... */
|
|
|
|
RunWorker(pstate->parallelSlot[i].args->AH, pipefd);
|
|
|
|
|
|
|
|
/* We can just exit(0) when done */
|
2013-03-24 16:27:20 +01:00
|
|
|
exit(0);
|
|
|
|
}
|
|
|
|
else if (pid < 0)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
/* fork failed */
|
|
|
|
exit_horribly(modulename,
|
|
|
|
"could not create worker process: %s\n",
|
|
|
|
strerror(errno));
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* In Master after successful fork */
|
|
|
|
pstate->parallelSlot[i].pid = pid;
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
/* close read end of Master -> Worker */
|
|
|
|
closesocket(pipeMW[PIPE_READ]);
|
|
|
|
/* close write end of Worker -> Master */
|
|
|
|
closesocket(pipeWM[PIPE_WRITE]);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
/*
|
|
|
|
* Having forked off the workers, disable SIGPIPE so that master isn't
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* killed if it tries to send a command to a dead worker. We don't want
|
|
|
|
* the workers to inherit this setting, though.
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
*/
|
|
|
|
#ifndef WIN32
|
|
|
|
signal(SIGPIPE, SIG_IGN);
|
|
|
|
#endif
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
return pstate;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Close down a parallel dump or restore.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
ParallelBackupEnd(ArchiveHandle *AH, ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* No work if non-parallel */
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pstate->numWorkers == 1)
|
|
|
|
return;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* There should not be any unfinished jobs */
|
2013-03-24 16:27:20 +01:00
|
|
|
Assert(IsEveryWorkerIdle(pstate));
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Close the sockets so that the workers know they can exit */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
{
|
|
|
|
closesocket(pstate->parallelSlot[i].pipeRead);
|
|
|
|
closesocket(pstate->parallelSlot[i].pipeWrite);
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* Wait for them to exit */
|
2013-03-24 16:27:20 +01:00
|
|
|
WaitForTerminatingWorkers(pstate);
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Unlink pstate from shutdown_info, so the exit handler will again fall
|
|
|
|
* back to closing AH->connection (if connected).
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
shutdown_info.pstate = NULL;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Release state (mere neatnik-ism, since we're about to terminate) */
|
2013-03-24 16:27:20 +01:00
|
|
|
free(pstate->parallelSlot);
|
|
|
|
free(pstate);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Dispatch a job to some free worker (caller must ensure there is one!)
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* te is the TocEntry to be processed, act is the action to be taken on it.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
DispatchJobForTocEntry(ArchiveHandle *AH, ParallelState *pstate, TocEntry *te,
|
|
|
|
T_Action act)
|
|
|
|
{
|
|
|
|
int worker;
|
|
|
|
char *arg;
|
|
|
|
|
|
|
|
/* our caller makes sure that at least one worker is idle */
|
|
|
|
worker = GetIdleWorker(pstate);
|
|
|
|
Assert(worker != NO_SLOT);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Construct and send command string */
|
2013-03-24 16:27:20 +01:00
|
|
|
arg = (AH->MasterStartParallelItemPtr) (AH, te, act);
|
|
|
|
|
|
|
|
sendMessageToWorker(pstate, worker, arg);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* XXX aren't we leaking string here? (no, because it's static. Ick.) */
|
|
|
|
|
|
|
|
/* Remember worker is busy, and which TocEntry it's working on */
|
2013-03-24 16:27:20 +01:00
|
|
|
pstate->parallelSlot[worker].workerStatus = WRKR_WORKING;
|
|
|
|
pstate->parallelSlot[worker].args->te = te;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Find an idle worker and return its slot number.
|
|
|
|
* Return NO_SLOT if none are idle.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
int
|
|
|
|
GetIdleWorker(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pstate->parallelSlot[i].workerStatus == WRKR_IDLE)
|
|
|
|
return i;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
return NO_SLOT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Return true iff every worker is in the WRKR_TERMINATED state.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
HasEveryWorkerTerminated(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pstate->parallelSlot[i].workerStatus != WRKR_TERMINATED)
|
|
|
|
return false;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return true iff every worker is in the WRKR_IDLE state.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
IsEveryWorkerIdle(ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
2013-03-24 16:27:20 +01:00
|
|
|
if (pstate->parallelSlot[i].workerStatus != WRKR_IDLE)
|
|
|
|
return false;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Acquire lock on a table to be dumped by a worker process.
|
|
|
|
*
|
|
|
|
* The master process is already holding an ACCESS SHARE lock. Ordinarily
|
|
|
|
* it's no problem for a worker to get one too, but if anything else besides
|
|
|
|
* pg_dump is running, there's a possible deadlock:
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
|
|
|
* 1) Master dumps the schema and locks all tables in ACCESS SHARE mode.
|
|
|
|
* 2) Another process requests an ACCESS EXCLUSIVE lock (which is not granted
|
|
|
|
* because the master holds a conflicting ACCESS SHARE lock).
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* 3) A worker process also requests an ACCESS SHARE lock to read the table.
|
|
|
|
* The worker is enqueued behind the ACCESS EXCLUSIVE lock request.
|
|
|
|
* 4) Now we have a deadlock, since the master is effectively waiting for
|
|
|
|
* the worker. The server cannot detect that, however.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* To prevent an infinite wait, prior to touching a table in a worker, request
|
|
|
|
* a lock in ACCESS SHARE mode but with NOWAIT. If we don't get the lock,
|
2013-03-24 16:27:20 +01:00
|
|
|
* then we know that somebody else has requested an ACCESS EXCLUSIVE lock and
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* so we have a deadlock. We must fail the backup in that case.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
lockTableForWorker(ArchiveHandle *AH, TocEntry *te)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
|
|
|
Archive *AHX = (Archive *) AH;
|
|
|
|
const char *qualId;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
PQExpBuffer query;
|
2013-03-24 16:27:20 +01:00
|
|
|
PGresult *res;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Nothing to do for BLOBS */
|
|
|
|
if (strcmp(te->desc, "BLOBS") == 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
query = createPQExpBuffer();
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* XXX this is an unbelievably expensive substitute for knowing how to dig
|
|
|
|
* a table name out of a TocEntry.
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
appendPQExpBuffer(query,
|
|
|
|
"SELECT pg_namespace.nspname,"
|
|
|
|
" pg_class.relname "
|
|
|
|
" FROM pg_class "
|
|
|
|
" JOIN pg_namespace on pg_namespace.oid = relnamespace "
|
2014-12-11 21:41:15 +01:00
|
|
|
" WHERE pg_class.oid = %u", te->catalogId.oid);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
res = PQexec(AH->connection, query->data);
|
|
|
|
|
|
|
|
if (!res || PQresultStatus(res) != PGRES_TUPLES_OK)
|
|
|
|
exit_horribly(modulename,
|
2013-07-12 03:48:09 +02:00
|
|
|
"could not get relation name for OID %u: %s\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
te->catalogId.oid, PQerrorMessage(AH->connection));
|
|
|
|
|
|
|
|
resetPQExpBuffer(query);
|
|
|
|
|
|
|
|
qualId = fmtQualifiedId(AHX->remoteVersion,
|
|
|
|
PQgetvalue(res, 0, 0),
|
|
|
|
PQgetvalue(res, 0, 1));
|
|
|
|
|
|
|
|
appendPQExpBuffer(query, "LOCK TABLE %s IN ACCESS SHARE MODE NOWAIT",
|
|
|
|
qualId);
|
|
|
|
PQclear(res);
|
|
|
|
|
|
|
|
res = PQexec(AH->connection, query->data);
|
|
|
|
|
|
|
|
if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)
|
|
|
|
exit_horribly(modulename,
|
2013-07-12 03:48:09 +02:00
|
|
|
"could not obtain lock on relation \"%s\"\n"
|
2014-05-06 18:12:18 +02:00
|
|
|
"This usually means that someone requested an ACCESS EXCLUSIVE lock "
|
|
|
|
"on the table after the pg_dump parent process had gotten the "
|
2013-03-24 16:27:20 +01:00
|
|
|
"initial ACCESS SHARE lock on the table.\n", qualId);
|
|
|
|
|
|
|
|
PQclear(res);
|
|
|
|
destroyPQExpBuffer(query);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* WaitForCommands: main routine for a worker process.
|
|
|
|
*
|
|
|
|
* Read and execute commands from the master until we see EOF on the pipe.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
2016-01-13 23:48:33 +01:00
|
|
|
WaitForCommands(ArchiveHandle *AH, int pipefd[2])
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
|
|
|
char *command;
|
|
|
|
DumpId dumpId;
|
|
|
|
int nBytes;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
char *str;
|
2013-03-24 16:27:20 +01:00
|
|
|
TocEntry *te;
|
|
|
|
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
if (!(command = getMessageFromMaster(pipefd)))
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* EOF ... clean up */
|
2013-03-24 16:27:20 +01:00
|
|
|
PQfinish(AH->connection);
|
|
|
|
AH->connection = NULL;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (messageStartsWith(command, "DUMP "))
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Decode the command */
|
2013-03-24 16:27:20 +01:00
|
|
|
sscanf(command + strlen("DUMP "), "%d%n", &dumpId, &nBytes);
|
|
|
|
Assert(nBytes == strlen(command) - strlen("DUMP "));
|
|
|
|
te = getTocEntryByDumpId(AH, dumpId);
|
|
|
|
Assert(te != NULL);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Acquire lock on this table within the worker's session */
|
|
|
|
lockTableForWorker(AH, te);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Perform the dump command */
|
2016-01-13 23:48:33 +01:00
|
|
|
str = (AH->WorkerJobDumpPtr) (AH, te);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* Return status to master */
|
2013-03-24 16:27:20 +01:00
|
|
|
sendMessageToMaster(pipefd, str);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* we are responsible for freeing the status string */
|
2013-03-24 16:27:20 +01:00
|
|
|
free(str);
|
|
|
|
}
|
|
|
|
else if (messageStartsWith(command, "RESTORE "))
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Decode the command */
|
2013-03-24 16:27:20 +01:00
|
|
|
sscanf(command + strlen("RESTORE "), "%d%n", &dumpId, &nBytes);
|
|
|
|
Assert(nBytes == strlen(command) - strlen("RESTORE "));
|
|
|
|
te = getTocEntryByDumpId(AH, dumpId);
|
|
|
|
Assert(te != NULL);
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Perform the restore command */
|
2013-03-24 16:27:20 +01:00
|
|
|
str = (AH->WorkerJobRestorePtr) (AH, te);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* Return status to master */
|
2013-03-24 16:27:20 +01:00
|
|
|
sendMessageToMaster(pipefd, str);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
|
|
|
/* we are responsible for freeing the status string */
|
2013-03-24 16:27:20 +01:00
|
|
|
free(str);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
exit_horribly(modulename,
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
"unrecognized command received from master: \"%s\"\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
command);
|
2013-07-14 20:35:26 +02:00
|
|
|
|
|
|
|
/* command was pg_malloc'd and we are responsible for free()ing it. */
|
|
|
|
free(command);
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Check for status messages from workers.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* If do_wait is true, wait to get a status message; otherwise, just return
|
|
|
|
* immediately if there is none available.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* When we get a status message, we let MasterEndParallelItemPtr process it,
|
|
|
|
* then save the resulting status code and switch the worker's state to
|
|
|
|
* WRKR_FINISHED. Later, caller must call ReapWorkerStatus() to verify
|
|
|
|
* that the status was "OK" and push the worker back to IDLE state.
|
|
|
|
*
|
|
|
|
* XXX Rube Goldberg would be proud of this API, but no one else should be.
|
|
|
|
*
|
|
|
|
* XXX is it worth checking for more than one status message per call?
|
|
|
|
* It seems somewhat unlikely that multiple workers would finish at exactly
|
|
|
|
* the same time.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
ListenToWorkers(ArchiveHandle *AH, ParallelState *pstate, bool do_wait)
|
|
|
|
{
|
|
|
|
int worker;
|
|
|
|
char *msg;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Try to collect a status message */
|
2013-03-24 16:27:20 +01:00
|
|
|
msg = getMessageFromWorker(pstate, do_wait, &worker);
|
|
|
|
|
|
|
|
if (!msg)
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* If do_wait is true, we must have detected EOF on some socket */
|
2013-03-24 16:27:20 +01:00
|
|
|
if (do_wait)
|
2013-07-12 03:48:09 +02:00
|
|
|
exit_horribly(modulename, "a worker process died unexpectedly\n");
|
2013-03-24 16:27:20 +01:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Process it and update our idea of the worker's status */
|
2013-03-24 16:27:20 +01:00
|
|
|
if (messageStartsWith(msg, "OK "))
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
TocEntry *te = pstate->parallelSlot[worker].args->te;
|
2013-03-24 16:27:20 +01:00
|
|
|
char *statusString;
|
|
|
|
|
|
|
|
if (messageStartsWith(msg, "OK RESTORE "))
|
|
|
|
{
|
|
|
|
statusString = msg + strlen("OK RESTORE ");
|
|
|
|
pstate->parallelSlot[worker].status =
|
|
|
|
(AH->MasterEndParallelItemPtr)
|
|
|
|
(AH, te, statusString, ACT_RESTORE);
|
|
|
|
}
|
|
|
|
else if (messageStartsWith(msg, "OK DUMP "))
|
|
|
|
{
|
|
|
|
statusString = msg + strlen("OK DUMP ");
|
|
|
|
pstate->parallelSlot[worker].status =
|
|
|
|
(AH->MasterEndParallelItemPtr)
|
|
|
|
(AH, te, statusString, ACT_DUMP);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
exit_horribly(modulename,
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
"invalid message received from worker: \"%s\"\n",
|
|
|
|
msg);
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
pstate->parallelSlot[worker].workerStatus = WRKR_FINISHED;
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
else
|
Fix broken error handling in parallel pg_dump/pg_restore.
In the original design for parallel dump, worker processes reported errors
by sending them up to the master process, which would print the messages.
This is unworkably fragile for a couple of reasons: it risks deadlock if a
worker sends an error at an unexpected time, and if the master has already
died for some reason, the user will never get to see the error at all.
Revert that idea and go back to just always printing messages to stderr.
This approach means that if all the workers fail for similar reasons (eg,
bad password or server shutdown), the user will see N copies of that
message, not only one as before. While that's slightly annoying, it's
certainly better than not seeing any message; not to mention that we
shouldn't assume that only the first failure is interesting.
An additional problem in the same area was that the master failed to
disable SIGPIPE (at least until much too late), which meant that sending a
command to an already-dead worker would cause the master to crash silently.
That was bad enough in itself but was made worse by the total reliance on
the master to print errors: even if the worker had reported an error, you
would probably not see it, depending on timing. Instead disable SIGPIPE
right after we've forked the workers, before attempting to send them
anything.
Additionally, the master relies on seeing socket EOF to realize that a
worker has exited prematurely --- but on Windows, there would be no EOF
since the socket is attached to the process that includes both the master
and worker threads, so it remains open. Make archive_close_connection()
close the worker end of the sockets so that this acts more like the Unix
case. It's not perfect, because if a worker thread exits without going
through exit_nicely() the closures won't happen; but that's not really
supposed to happen.
This has been wrong all along, so back-patch to 9.3 where parallel dump
was introduced.
Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
|
|
|
exit_horribly(modulename,
|
|
|
|
"invalid message received from worker: \"%s\"\n",
|
|
|
|
msg);
|
2013-03-24 16:27:20 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Free the string returned from getMessageFromWorker */
|
2013-03-24 16:27:20 +01:00
|
|
|
free(msg);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Check to see if any worker is in WRKR_FINISHED state. If so,
|
|
|
|
* return its command status code into *status, reset it to IDLE state,
|
|
|
|
* and return its slot number. Otherwise return NO_SLOT.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in the master process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
int
|
|
|
|
ReapWorkerStatus(ParallelState *pstate, int *status)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
{
|
|
|
|
if (pstate->parallelSlot[i].workerStatus == WRKR_FINISHED)
|
|
|
|
{
|
|
|
|
*status = pstate->parallelSlot[i].status;
|
|
|
|
pstate->parallelSlot[i].status = 0;
|
|
|
|
pstate->parallelSlot[i].workerStatus = WRKR_IDLE;
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return NO_SLOT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Wait, if necessary, until we have at least one idle worker.
|
|
|
|
* Reap worker status as necessary to move FINISHED workers to IDLE state.
|
|
|
|
*
|
|
|
|
* We assume that no extra processing is required when reaping a finished
|
|
|
|
* command, except for checking that the status was OK (zero).
|
|
|
|
* Caution: that assumption means that this function can only be used in
|
|
|
|
* parallel dump, not parallel restore, because the latter has a more
|
|
|
|
* complex set of rules about handling status.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in the master process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
EnsureIdleWorker(ArchiveHandle *AH, ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int ret_worker;
|
|
|
|
int work_status;
|
|
|
|
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
int nTerm = 0;
|
|
|
|
|
|
|
|
while ((ret_worker = ReapWorkerStatus(pstate, &work_status)) != NO_SLOT)
|
|
|
|
{
|
|
|
|
if (work_status != 0)
|
2013-07-12 03:48:09 +02:00
|
|
|
exit_horribly(modulename, "error processing a parallel work item\n");
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
nTerm++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We need to make sure that we have an idle worker before dispatching
|
|
|
|
* the next item. If nTerm > 0 we already have that (quick check).
|
|
|
|
*/
|
|
|
|
if (nTerm > 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* explicit check for an idle worker */
|
|
|
|
if (GetIdleWorker(pstate) != NO_SLOT)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we have no idle worker, read the result of one or more workers
|
|
|
|
* and loop the loop to call ReapWorkerStatus() on them
|
|
|
|
*/
|
|
|
|
ListenToWorkers(AH, pstate, true);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Wait for all workers to be idle.
|
|
|
|
* Reap worker status as necessary to move FINISHED workers to IDLE state.
|
|
|
|
*
|
|
|
|
* We assume that no extra processing is required when reaping a finished
|
|
|
|
* command, except for checking that the status was OK (zero).
|
|
|
|
* Caution: that assumption means that this function can only be used in
|
|
|
|
* parallel dump, not parallel restore, because the latter has a more
|
|
|
|
* complex set of rules about handling status.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in the master process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
EnsureWorkersFinished(ArchiveHandle *AH, ParallelState *pstate)
|
|
|
|
{
|
|
|
|
int work_status;
|
|
|
|
|
|
|
|
if (!pstate || pstate->numWorkers == 1)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Waiting for the remaining worker processes to finish */
|
|
|
|
while (!IsEveryWorkerIdle(pstate))
|
|
|
|
{
|
|
|
|
if (ReapWorkerStatus(pstate, &work_status) == NO_SLOT)
|
|
|
|
ListenToWorkers(AH, pstate, true);
|
|
|
|
else if (work_status != 0)
|
|
|
|
exit_horribly(modulename,
|
2013-07-12 03:48:09 +02:00
|
|
|
"error processing a parallel work item\n");
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Read one command message from the master, blocking if necessary
|
|
|
|
* until one is available, and return it as a malloc'd string.
|
|
|
|
* On EOF, return NULL.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in worker processes.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
getMessageFromMaster(int pipefd[2])
|
|
|
|
{
|
|
|
|
return readMessageFromPipe(pipefd[PIPE_READ]);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Send a status message to the master.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in worker processes.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
sendMessageToMaster(int pipefd[2], const char *str)
|
|
|
|
{
|
|
|
|
int len = strlen(str) + 1;
|
|
|
|
|
|
|
|
if (pipewrite(pipefd[PIPE_WRITE], str, len) != len)
|
|
|
|
exit_horribly(modulename,
|
2013-07-12 03:48:09 +02:00
|
|
|
"could not write to the communication channel: %s\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
strerror(errno));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Wait until some descriptor in "workerset" becomes readable.
|
|
|
|
* Returns -1 on error, else the number of readable descriptors.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static int
|
|
|
|
select_loop(int maxFd, fd_set *workerset)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
fd_set saveSet = *workerset;
|
|
|
|
|
|
|
|
#ifdef WIN32
|
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Sleep a quarter of a second before checking if we should terminate.
|
|
|
|
*
|
|
|
|
* XXX we're not actually checking for a cancel interrupt ... but we
|
|
|
|
* should be.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
struct timeval tv = {0, 250000};
|
|
|
|
|
|
|
|
*workerset = saveSet;
|
|
|
|
i = select(maxFd + 1, workerset, NULL, NULL, &tv);
|
|
|
|
|
|
|
|
if (i == SOCKET_ERROR && WSAGetLastError() == WSAEINTR)
|
|
|
|
continue;
|
|
|
|
if (i)
|
|
|
|
break;
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#else /* !WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
for (;;)
|
|
|
|
{
|
|
|
|
*workerset = saveSet;
|
|
|
|
i = select(maxFd + 1, workerset, NULL, NULL, NULL);
|
|
|
|
|
|
|
|
/*
|
2015-01-23 19:02:45 +01:00
|
|
|
* If we Ctrl-C the master process, it's likely that we interrupt
|
2013-03-24 16:27:20 +01:00
|
|
|
* select() here. The signal handler will set wantAbort == true and
|
|
|
|
* the shutdown journey starts from here. Note that we'll come back
|
|
|
|
* here later when we tell all workers to terminate and read their
|
|
|
|
* responses. But then we have aborting set to true.
|
|
|
|
*/
|
|
|
|
if (wantAbort && !aborting)
|
|
|
|
exit_horribly(modulename, "terminated by user\n");
|
|
|
|
|
|
|
|
if (i < 0 && errno == EINTR)
|
|
|
|
continue;
|
|
|
|
break;
|
|
|
|
}
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Check for messages from worker processes.
|
|
|
|
*
|
|
|
|
* If a message is available, return it as a malloc'd string, and put the
|
|
|
|
* index of the sending worker in *worker.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* If nothing is available, wait if "do_wait" is true, else return NULL.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* If we detect EOF on any socket, we'll return NULL. It's not great that
|
|
|
|
* that's hard to distinguish from the no-data-available case, but for now
|
|
|
|
* our one caller is okay with that.
|
|
|
|
*
|
|
|
|
* This function is executed in the master process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
getMessageFromWorker(ParallelState *pstate, bool do_wait, int *worker)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
fd_set workerset;
|
|
|
|
int maxFd = -1;
|
|
|
|
struct timeval nowait = {0, 0};
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* construct bitmap of socket descriptors for select() */
|
2013-03-24 16:27:20 +01:00
|
|
|
FD_ZERO(&workerset);
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
{
|
|
|
|
if (pstate->parallelSlot[i].workerStatus == WRKR_TERMINATED)
|
|
|
|
continue;
|
|
|
|
FD_SET(pstate->parallelSlot[i].pipeRead, &workerset);
|
|
|
|
if (pstate->parallelSlot[i].pipeRead > maxFd)
|
|
|
|
maxFd = pstate->parallelSlot[i].pipeRead;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (do_wait)
|
|
|
|
{
|
|
|
|
i = select_loop(maxFd, &workerset);
|
|
|
|
Assert(i != 0);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
if ((i = select(maxFd + 1, &workerset, NULL, NULL, &nowait)) == 0)
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (i < 0)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
exit_horribly(modulename, "select() failed: %s\n", strerror(errno));
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
for (i = 0; i < pstate->numWorkers; i++)
|
|
|
|
{
|
|
|
|
char *msg;
|
|
|
|
|
|
|
|
if (!FD_ISSET(pstate->parallelSlot[i].pipeRead, &workerset))
|
|
|
|
continue;
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Read the message if any. If the socket is ready because of EOF,
|
|
|
|
* we'll return NULL instead (and the socket will stay ready, so the
|
|
|
|
* condition will persist).
|
|
|
|
*
|
|
|
|
* Note: because this is a blocking read, we'll wait if only part of
|
|
|
|
* the message is available. Waiting a long time would be bad, but
|
|
|
|
* since worker status messages are short and are always sent in one
|
|
|
|
* operation, it shouldn't be a problem in practice.
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
msg = readMessageFromPipe(pstate->parallelSlot[i].pipeRead);
|
|
|
|
*worker = i;
|
|
|
|
return msg;
|
|
|
|
}
|
|
|
|
Assert(false);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Send a command message to the specified worker process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This function is executed in the master process.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
sendMessageToWorker(ParallelState *pstate, int worker, const char *str)
|
|
|
|
{
|
|
|
|
int len = strlen(str) + 1;
|
|
|
|
|
|
|
|
if (pipewrite(pstate->parallelSlot[worker].pipeWrite, str, len) != len)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If we're already aborting anyway, don't care if we succeed or not.
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* The child might have gone already. (XXX but if we're aborting
|
|
|
|
* already, why are we here at all?)
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
#ifndef WIN32
|
|
|
|
if (!aborting)
|
|
|
|
#endif
|
|
|
|
exit_horribly(modulename,
|
2014-05-06 18:12:18 +02:00
|
|
|
"could not write to the communication channel: %s\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
strerror(errno));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* Read one message from the specified pipe (fd), blocking if necessary
|
|
|
|
* until one is available, and return it as a malloc'd string.
|
|
|
|
* On EOF, return NULL.
|
|
|
|
*
|
|
|
|
* A "message" on the channel is just a null-terminated string.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static char *
|
|
|
|
readMessageFromPipe(int fd)
|
|
|
|
{
|
|
|
|
char *msg;
|
|
|
|
int msgsize,
|
|
|
|
bufsize;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* In theory, if we let piperead() read multiple bytes, it might give us
|
|
|
|
* back fragments of multiple messages. (That can't actually occur, since
|
|
|
|
* neither master nor workers send more than one message without waiting
|
|
|
|
* for a reply, but we don't wish to assume that here.) For simplicity,
|
|
|
|
* read a byte at a time until we get the terminating '\0'. This method
|
|
|
|
* is a bit inefficient, but since this is only used for relatively short
|
|
|
|
* command and status strings, it shouldn't matter.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
bufsize = 64; /* could be any number */
|
|
|
|
msg = (char *) pg_malloc(bufsize);
|
|
|
|
msgsize = 0;
|
|
|
|
for (;;)
|
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
Assert(msgsize < bufsize);
|
2013-03-24 16:27:20 +01:00
|
|
|
ret = piperead(fd, msg + msgsize, 1);
|
|
|
|
if (ret <= 0)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
break; /* error or connection closure */
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
Assert(ret == 1);
|
|
|
|
|
|
|
|
if (msg[msgsize] == '\0')
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
return msg; /* collected whole message */
|
2013-03-24 16:27:20 +01:00
|
|
|
|
|
|
|
msgsize++;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
if (msgsize == bufsize) /* enlarge buffer if needed */
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
bufsize += 16; /* could be any number */
|
2014-11-18 19:28:06 +01:00
|
|
|
msg = (char *) pg_realloc(msg, bufsize);
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
}
|
2014-01-24 21:10:08 +01:00
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/* Other end has closed the connection */
|
2014-11-18 19:28:06 +01:00
|
|
|
pg_free(msg);
|
2014-01-24 21:10:08 +01:00
|
|
|
return NULL;
|
2013-03-24 16:27:20 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef WIN32
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
/*
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
* This is a replacement version of pipe(2) for Windows which allows the pipe
|
|
|
|
* handles to be used in select().
|
|
|
|
*
|
|
|
|
* Reads and writes on the pipe must go through piperead()/pipewrite().
|
|
|
|
*
|
|
|
|
* For consistency with Unix we declare the returned handles as "int".
|
|
|
|
* This is okay even on WIN64 because system handles are not more than
|
|
|
|
* 32 bits wide, but we do have to do some casting.
|
2013-03-24 16:27:20 +01:00
|
|
|
*/
|
|
|
|
static int
|
|
|
|
pgpipe(int handles[2])
|
|
|
|
{
|
2014-10-17 18:19:05 +02:00
|
|
|
pgsocket s,
|
|
|
|
tmp_sock;
|
2013-03-24 16:27:20 +01:00
|
|
|
struct sockaddr_in serv_addr;
|
|
|
|
int len = sizeof(serv_addr);
|
|
|
|
|
2014-06-16 21:24:38 +02:00
|
|
|
/* We have to use the Unix socket invalid file descriptor value here. */
|
|
|
|
handles[0] = handles[1] = -1;
|
2013-03-24 16:27:20 +01:00
|
|
|
|
2014-06-16 21:24:38 +02:00
|
|
|
/*
|
|
|
|
* setup listen socket
|
|
|
|
*/
|
|
|
|
if ((s = socket(AF_INET, SOCK_STREAM, 0)) == PGINVALID_SOCKET)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not create socket: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
memset((void *) &serv_addr, 0, sizeof(serv_addr));
|
|
|
|
serv_addr.sin_family = AF_INET;
|
|
|
|
serv_addr.sin_port = htons(0);
|
|
|
|
serv_addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
2013-05-29 22:58:43 +02:00
|
|
|
if (bind(s, (SOCKADDR *) &serv_addr, len) == SOCKET_ERROR)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not bind: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
if (listen(s, 1) == SOCKET_ERROR)
|
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not listen: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
2013-05-29 22:58:43 +02:00
|
|
|
if (getsockname(s, (SOCKADDR *) &serv_addr, &len) == SOCKET_ERROR)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: getsockname() failed: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
2014-06-16 21:24:38 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* setup pipe handles
|
|
|
|
*/
|
|
|
|
if ((tmp_sock = socket(AF_INET, SOCK_STREAM, 0)) == PGINVALID_SOCKET)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not create second socket: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
2014-06-16 21:24:38 +02:00
|
|
|
handles[1] = (int) tmp_sock;
|
2013-03-24 16:27:20 +01:00
|
|
|
|
2013-05-29 22:58:43 +02:00
|
|
|
if (connect(handles[1], (SOCKADDR *) &serv_addr, len) == SOCKET_ERROR)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not connect socket: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
closesocket(handles[1]);
|
|
|
|
handles[1] = -1;
|
2013-03-24 16:27:20 +01:00
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
2014-06-16 21:24:38 +02:00
|
|
|
if ((tmp_sock = accept(s, (SOCKADDR *) &serv_addr, &len)) == PGINVALID_SOCKET)
|
2013-03-24 16:27:20 +01:00
|
|
|
{
|
2013-07-12 03:48:09 +02:00
|
|
|
write_msg(modulename, "pgpipe: could not accept connection: error code %d\n",
|
2013-03-24 16:27:20 +01:00
|
|
|
WSAGetLastError());
|
|
|
|
closesocket(handles[1]);
|
2014-06-16 21:24:38 +02:00
|
|
|
handles[1] = -1;
|
2013-03-24 16:27:20 +01:00
|
|
|
closesocket(s);
|
|
|
|
return -1;
|
|
|
|
}
|
2014-06-16 21:24:38 +02:00
|
|
|
handles[0] = (int) tmp_sock;
|
|
|
|
|
2013-03-24 16:27:20 +01:00
|
|
|
closesocket(s);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
/*
|
|
|
|
* Windows implementation of reading from a pipe.
|
|
|
|
*/
|
2013-03-24 16:27:20 +01:00
|
|
|
static int
|
|
|
|
piperead(int s, char *buf, int len)
|
|
|
|
{
|
|
|
|
int ret = recv(s, buf, len, 0);
|
|
|
|
|
|
|
|
if (ret < 0 && WSAGetLastError() == WSAECONNRESET)
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
{
|
|
|
|
/* EOF on the pipe! */
|
2013-03-24 16:27:20 +01:00
|
|
|
ret = 0;
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
}
|
2013-03-24 16:27:20 +01:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
Lots of comment-fixing, and minor cosmetic cleanup, in pg_dump/parallel.c.
The commentary in this file was in extremely sad shape. The author(s)
had clearly never heard of the project convention that a function header
comment should provide an API spec of some sort for that function. Much
of it was flat out wrong, too --- maybe it was accurate when written, but
if so it had not been updated to track subsequent code revisions. Rewrite
and rearrange to try to bring it up to speed, and annotate some of the
places where more work is needed. (I've refrained from actually fixing
anything of substance ... yet.)
Also, rename a couple of functions for more clarity as to what they do,
do some very minor code rearrangement, remove some pointless Asserts,
fix an incorrect Assert in readMessageFromPipe, and add a missing socket
close in one error exit from pgpipe(). The last would be a bug if we
tried to continue after pgpipe() failure, but since we don't, it's just
cosmetic at present.
Although this is only cosmetic, back-patch to 9.3 where parallel.c was
added. It's sufficiently invasive that it'll pose a hazard for future
back-patching if we don't.
Discussion: <25239.1464386067@sss.pgh.pa.us>
2016-05-28 20:02:11 +02:00
|
|
|
#endif /* WIN32 */
|