rdelaage/postgresql
1
0
Fork 0
mirror of https://git.postgresql.org/git/postgresql.git synced 2024-10-04 19:36:53 +02:00
Code Issues Packages Projects Releases Wiki Activity
d81ecb9b20
postgresql/src/bin/pg_dump/parallel.h

88 lines
2.2 KiB
C
Raw Normal View History

Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
/*-------------------------------------------------------------------------
*
* parallel.h
*
* Parallel support header file for the pg_dump archiver
*
Update copyright for 2016 Backpatch certain files through 9.1
2016-01-02 19:33:40 +01:00
* Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
* Portions Copyright (c) 1994, Regents of the University of California
*
* The author is not responsible for loss or damages that may
* result from its use.
*
* IDENTIFICATION
* src/bin/pg_dump/parallel.h
*
*-------------------------------------------------------------------------
*/
Move some pg_dump function around. Move functions used only by pg_dump and pg_restore from dumputils.c to a new file, pg_backup_utils.c. dumputils.c is linked into psql and some programs in bin/scripts, so it seems good to keep it slim. The parallel functionality is moved to parallel.c, as is exit_horribly, because the interesting code in exit_horribly is parallel-related. This refactoring gets rid of the on_exit_msg_func function pointer. It was problematic, because a modern gcc version with -Wmissing-format-attribute complained if it wasn't marked with PF_PRINTF_ATTRIBUTE, but the ancient gcc version that Tom Lane's old HP-UX box has didn't accept that attribute on a function pointer, and gave an error. We still use a similar function pointer trick for getLocalPQBuffer() function, to use a thread-local version of that in parallel mode on Windows, but that dodges the problem because it doesn't take printf-like arguments.
2013-03-27 17:10:40 +01:00
#ifndef PG_DUMP_PARALLEL_H
#define PG_DUMP_PARALLEL_H
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
#include "pg_backup_archiver.h"
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
typedef enum
{
WRKR_TERMINATED = 0,
WRKR_IDLE,
WRKR_WORKING,
WRKR_FINISHED
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
} T_WorkerStatus;
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
/* Arguments needed for a worker process */
typedef struct ParallelArgs
{
Re-pgindent src/bin/pg_dump/*. Seems to have gotten rather messy lately, as a consequence of a couple of large recent commits.
2014-10-17 18:19:05 +02:00
ArchiveHandle *AH;
TocEntry *te;
pgindent run for release 9.3 This is the first run of the Perl-based pgindent script. Also update pgindent instructions.
2013-05-29 22:58:43 +02:00
} ParallelArgs;
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
/* State for each parallel activity slot */
typedef struct ParallelSlot
{
ParallelArgs *args;
T_WorkerStatus workerStatus;
int status;
Fix broken error handling in parallel pg_dump/pg_restore. In the original design for parallel dump, worker processes reported errors by sending them up to the master process, which would print the messages. This is unworkably fragile for a couple of reasons: it risks deadlock if a worker sends an error at an unexpected time, and if the master has already died for some reason, the user will never get to see the error at all. Revert that idea and go back to just always printing messages to stderr. This approach means that if all the workers fail for similar reasons (eg, bad password or server shutdown), the user will see N copies of that message, not only one as before. While that's slightly annoying, it's certainly better than not seeing any message; not to mention that we shouldn't assume that only the first failure is interesting. An additional problem in the same area was that the master failed to disable SIGPIPE (at least until much too late), which meant that sending a command to an already-dead worker would cause the master to crash silently. That was bad enough in itself but was made worse by the total reliance on the master to print errors: even if the worker had reported an error, you would probably not see it, depending on timing. Instead disable SIGPIPE right after we've forked the workers, before attempting to send them anything. Additionally, the master relies on seeing socket EOF to realize that a worker has exited prematurely --- but on Windows, there would be no EOF since the socket is attached to the process that includes both the master and worker threads, so it remains open. Make archive_close_connection() close the worker end of the sockets so that this acts more like the Unix case. It's not perfect, because if a worker thread exits without going through exit_nicely() the closures won't happen; but that's not really supposed to happen. This has been wrong all along, so back-patch to 9.3 where parallel dump was introduced. Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
int pipeRead; /* master's end of the pipes */
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
int pipeWrite;
Fix broken error handling in parallel pg_dump/pg_restore. In the original design for parallel dump, worker processes reported errors by sending them up to the master process, which would print the messages. This is unworkably fragile for a couple of reasons: it risks deadlock if a worker sends an error at an unexpected time, and if the master has already died for some reason, the user will never get to see the error at all. Revert that idea and go back to just always printing messages to stderr. This approach means that if all the workers fail for similar reasons (eg, bad password or server shutdown), the user will see N copies of that message, not only one as before. While that's slightly annoying, it's certainly better than not seeing any message; not to mention that we shouldn't assume that only the first failure is interesting. An additional problem in the same area was that the master failed to disable SIGPIPE (at least until much too late), which meant that sending a command to an already-dead worker would cause the master to crash silently. That was bad enough in itself but was made worse by the total reliance on the master to print errors: even if the worker had reported an error, you would probably not see it, depending on timing. Instead disable SIGPIPE right after we've forked the workers, before attempting to send them anything. Additionally, the master relies on seeing socket EOF to realize that a worker has exited prematurely --- but on Windows, there would be no EOF since the socket is attached to the process that includes both the master and worker threads, so it remains open. Make archive_close_connection() close the worker end of the sockets so that this acts more like the Unix case. It's not perfect, because if a worker thread exits without going through exit_nicely() the closures won't happen; but that's not really supposed to happen. This has been wrong all along, so back-patch to 9.3 where parallel dump was introduced. Report: <2458.1450894615@sss.pgh.pa.us>
2016-05-25 18:39:57 +02:00
int pipeRevRead; /* child's end of the pipes */
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
int pipeRevWrite;
#ifdef WIN32
uintptr_t hThread;
unsigned int threadId;
#else
pid_t pid;
#endif
} ParallelSlot;
#define NO_SLOT (-1)
typedef struct ParallelState
{
int numWorkers;
ParallelSlot *parallelSlot;
} ParallelState;
Move some pg_dump function around. Move functions used only by pg_dump and pg_restore from dumputils.c to a new file, pg_backup_utils.c. dumputils.c is linked into psql and some programs in bin/scripts, so it seems good to keep it slim. The parallel functionality is moved to parallel.c, as is exit_horribly, because the interesting code in exit_horribly is parallel-related. This refactoring gets rid of the on_exit_msg_func function pointer. It was problematic, because a modern gcc version with -Wmissing-format-attribute complained if it wasn't marked with PF_PRINTF_ATTRIBUTE, but the ancient gcc version that Tom Lane's old HP-UX box has didn't accept that attribute on a function pointer, and gave an error. We still use a similar function pointer trick for getLocalPQBuffer() function, to use a thread-local version of that in parallel mode on Windows, but that dodges the problem because it doesn't take printf-like arguments.
2013-03-27 17:10:40 +01:00
#ifdef WIN32
extern bool parallel_init_done;
extern DWORD mainThreadId;
#endif
extern void init_parallel_dump_utils(void);
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
extern int GetIdleWorker(ParallelState *pstate);
extern bool IsEveryWorkerIdle(ParallelState *pstate);
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
extern void ListenToWorkers(ArchiveHandle *AH, ParallelState *pstate, bool do_wait);
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
extern int ReapWorkerStatus(ParallelState *pstate, int *status);
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
extern void EnsureIdleWorker(ArchiveHandle *AH, ParallelState *pstate);
extern void EnsureWorkersFinished(ArchiveHandle *AH, ParallelState *pstate);
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
Access pg_dump's options structs through Archive struct, not directly. Rather than passing around DumpOptions and RestoreOptions as separate arguments, add fields to struct Archive to carry pointers to these objects, and access them through those fields when needed. There already was a RestoreOptions pointer in Archive, though for no obvious reason it was part of the "private" struct rather than out where pg_dump.c could see it. Doing this allows reversion of quite a lot of parameter-addition changes made in commit 0eea8047bf, which is a good thing IMO because this will reduce the code delta between 9.4 and 9.5, probably easing a few future back-patch efforts. Moreover, the previous commit only added a DumpOptions argument to functions that had to have it at the time, which means we could anticipate still more code churn (and more back-patch hazard) as the requirement spread further. I'd hit exactly that problem in my upcoming patch to fix extension membership marking, which is what motivated me to do this.
2016-01-13 23:48:33 +01:00
extern ParallelState *ParallelBackupStart(ArchiveHandle *AH);
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
extern void DispatchJobForTocEntry(ArchiveHandle *AH,
Add parallel pg_dump option. New infrastructure is added which creates a set number of workers (threads on Windows, forked processes on Unix). Jobs are then handed out to these workers by the master process as needed. pg_restore is adjusted to use this new infrastructure in place of the old setup which created a new worker for each step on the fly. Parallel dumps acquire a snapshot clone in order to stay consistent, if available. The parallel option is selected by the -j / --jobs command line parameter of pg_dump. Joachim Wieland, lightly editorialized by Andrew Dunstan.
2013-03-24 16:27:20 +01:00
ParallelState *pstate,
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
TocEntry *te, T_Action act);
extern void ParallelBackupEnd(ArchiveHandle *AH, ParallelState *pstate);
Move some pg_dump function around. Move functions used only by pg_dump and pg_restore from dumputils.c to a new file, pg_backup_utils.c. dumputils.c is linked into psql and some programs in bin/scripts, so it seems good to keep it slim. The parallel functionality is moved to parallel.c, as is exit_horribly, because the interesting code in exit_horribly is parallel-related. This refactoring gets rid of the on_exit_msg_func function pointer. It was problematic, because a modern gcc version with -Wmissing-format-attribute complained if it wasn't marked with PF_PRINTF_ATTRIBUTE, but the ancient gcc version that Tom Lane's old HP-UX box has didn't accept that attribute on a function pointer, and gave an error. We still use a similar function pointer trick for getLocalPQBuffer() function, to use a thread-local version of that in parallel mode on Windows, but that dodges the problem because it doesn't take printf-like arguments.
2013-03-27 17:10:40 +01:00
pg_dump: Reduce use of global variables Most pg_dump.c global variables, which were passed down individually to dumping routines, are now grouped as members of the new DumpOptions struct, which is used as a local variable and passed down into routines that need it. This helps future development efforts; in particular it is said to enable a mode in which a parallel pg_dump run can output multiple streams, and have them restored in parallel. Also take the opportunity to clean up the pg_dump header files somewhat, to avoid circularity. Author: Joachim Wieland, revised by Álvaro Herrera Reviewed by Peter Eisentraut
2014-10-14 20:00:55 +02:00
extern void checkAborting(ArchiveHandle *AH);
Move some pg_dump function around. Move functions used only by pg_dump and pg_restore from dumputils.c to a new file, pg_backup_utils.c. dumputils.c is linked into psql and some programs in bin/scripts, so it seems good to keep it slim. The parallel functionality is moved to parallel.c, as is exit_horribly, because the interesting code in exit_horribly is parallel-related. This refactoring gets rid of the on_exit_msg_func function pointer. It was problematic, because a modern gcc version with -Wmissing-format-attribute complained if it wasn't marked with PF_PRINTF_ATTRIBUTE, but the ancient gcc version that Tom Lane's old HP-UX box has didn't accept that attribute on a function pointer, and gave an error. We still use a similar function pointer trick for getLocalPQBuffer() function, to use a thread-local version of that in parallel mode on Windows, but that dodges the problem because it doesn't take printf-like arguments.
2013-03-27 17:10:40 +01:00
#endif /* PG_DUMP_PARALLEL_H */
Reference in New Issue Copy Permalink