rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/src/bin/pg_dump/pg_backup_archiver.c

4298 lines
105 KiB
C
Raw Normal View History

pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/*-------------------------------------------------------------------------
*
* pg_backup_archiver.c
*
* Private implementation of the archiver routines.
*
* See the headers to pg_restore for more details.
*
* Copyright (c) 2000, Philip Warner
* Rights are granted to use this software in any way so long
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* as this notice is not removed.
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*
* The author is not responsible for loss or damages that may
Make sure all clients have the same understanding of default user name and database. In particular, make script wrappers understand the PGDATABASE environment variable.
2002-08-10 18:57:32 +02:00
* result from its use.
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*
*
* IDENTIFICATION
Remove cvs keywords from all files.
2010-09-20 22:08:53 +02:00
* src/bin/pg_dump/pg_backup_archiver.c
Fix pg_restore to handle the 'set max oid' entry correctly in archives dumped by pg_dump -o. Per bug report posted by Bruce; fix is from Philip Warner, reviewed by Tom Lane.
2002-01-18 18:13:51 +01:00
*
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*-------------------------------------------------------------------------
*/
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
#include "pg_backup_db.h"
Move pg_dump memory routines into pg_dumpmem.c/h and restore common.c with its original functions. The previous function migration would cause too many difficulties in back-patching.
2011-11-27 04:34:36 +01:00
#include "dumpmem.h"
Reimplement pg_dumpall in C. Currently no change in functionality, except that it's more robust, reconnects less often, and is NLS'ed.
2002-08-27 20:57:26 +02:00
#include "dumputils.h"
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Be more wary about mixed-case database names and user names. Get the CREATE DATABASE command right in pg_dump -C case.
2002-02-11 01:18:20 +01:00
#include <ctype.h>
Reimplement pg_dumpall in C. Currently no change in functionality, except that it's more robust, reconnects less often, and is NLS'ed.
2002-08-27 20:57:26 +02:00
#include <unistd.h>
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
#include <sys/stat.h>
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
#include <sys/types.h>
#include <sys/wait.h>
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Make pg_dump and pg_restore handle binary archive formats correctly when using stdout/stdin on Windows.
2005-01-26 20:44:43 +01:00
#ifdef WIN32
#include <io.h>
#endif
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
#include "libpq/libpq-fs.h"
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Special exit values from worker children. We reserve 0 for normal
* success; 1 and other small values should be interpreted as crashes.
*/
#define WORKER_CREATE_DONE 10
#define WORKER_INHIBIT_DATA 11
#define WORKER_IGNORED_ERRORS 12
/*
* Unix uses exit to return result from worker child, so function is void.
* Windows thread result comes via function return.
*/
#ifndef WIN32
#define parallel_restore_result void
#else
#define parallel_restore_result DWORD
#endif
/* IDs for worker children are either PIDs or thread handles */
#ifndef WIN32
#define thandle pid_t
#else
#define thandle HANDLE
#endif
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
/* Arguments needed for a worker child */
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
typedef struct _restore_args
{
ArchiveHandle *AH;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
TocEntry *te;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
} RestoreArgs;
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
/* State for each parallel activity slot */
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
typedef struct _parallel_slot
{
thandle child_id;
RestoreArgs *args;
} ParallelSlot;
#define NO_SLOT (-1)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Suggest use of psql when pg_restore gets a text dump.
2012-01-03 22:02:49 +01:00
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
/* state needed to save/restore an archive's output target */
typedef struct _outputContext
{
void *OF;
int gzOut;
} OutputContext;
Code cleanup: mark some variables with the "const" modifier, when they are initialized with a string literal. Patch from Stefan Huehner.
2007-03-18 17:50:44 +01:00
static const char *modulename = gettext_noop("archiver");
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
/* index array created by fix_dependencies -- only used in parallel restore */
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
static TocEntry **tocsByDumpId; /* index by dumpId - 1 */
static DumpId maxDumpId; /* length of above array */
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const int compression, ArchiveMode mode);
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
static void _getObjectDescription(PQExpBuffer buf, TocEntry *te,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
ArchiveHandle *AH);
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt, bool isData, bool acl_pass);
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
static char *replace_line_endings(const char *str);
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
static void _doSetFixedOutputState(ArchiveHandle *AH);
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
static void _doSetSessionAuth(ArchiveHandle *AH, const char *user);
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
static void _doSetWithOids(ArchiveHandle *AH, const bool withOids);
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
static void _reconnectToDB(ArchiveHandle *AH, const char *dbname);
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
static void _becomeUser(ArchiveHandle *AH, const char *user);
static void _becomeOwner(ArchiveHandle *AH, TocEntry *te);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
static void _selectOutputSchema(ArchiveHandle *AH, const char *schemaName);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
static bool _tocEntryIsACL(TocEntry *te);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
static void _moveBefore(ArchiveHandle *AH, TocEntry *pos, TocEntry *te);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static int _discoverArchiveFormat(ArchiveHandle *AH);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
static int RestoringToDB(ArchiveHandle *AH);
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
static void dump_lo_buf(ArchiveHandle *AH);
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
static void vdie_horribly(ArchiveHandle *AH, const char *modulename,
const char *fmt, va_list ap)
pg_dump: Add GCC noreturn attribute to appropriate functions This is a small help to the compiler and static analyzers.
2012-01-31 19:47:43 +01:00
__attribute__((format(PG_PRINTF_ATTRIBUTE, 3, 0), noreturn));
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
pg_dump: Add some const qualifiers
2012-02-07 22:20:29 +01:00
static void SetOutput(ArchiveHandle *AH, const char *filename, int compression);
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
static OutputContext SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext);
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static int restore_toc_entry(ArchiveHandle *AH, TocEntry *te,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
RestoreOptions *ropt, bool is_parallel);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static void restore_toc_entries_parallel(ArchiveHandle *AH);
static thandle spawn_restore(RestoreArgs *args);
static thandle reap_child(ParallelSlot *slots, int n_slots, int *work_status);
static bool work_in_progress(ParallelSlot *slots, int n_slots);
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
static int get_next_slot(ParallelSlot *slots, int n_slots);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
static void par_list_header_init(TocEntry *l);
static void par_list_append(TocEntry *l, TocEntry *te);
static void par_list_remove(TocEntry *te);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static TocEntry *get_next_work_item(ArchiveHandle *AH,
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
TocEntry *ready_list,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ParallelSlot *slots, int n_slots);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static parallel_restore_result parallel_restore(RestoreArgs *args);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
static void mark_work_done(ArchiveHandle *AH, TocEntry *ready_list,
thandle worker, int status,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ParallelSlot *slots, int n_slots);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static void fix_dependencies(ArchiveHandle *AH);
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static void repoint_table_dependencies(ArchiveHandle *AH,
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
DumpId tableId, DumpId tableDataId);
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
static void identify_locking_dependencies(TocEntry *te);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
TocEntry *ready_list);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
static ArchiveHandle *CloneArchive(ArchiveHandle *AH);
static void DeCloneArchive(ArchiveHandle *AH);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Wrapper functions.
*
* The objective it to make writing new formats and dumpers as simple
* as possible, if necessary at the expense of extra function calls etc.
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*
*/
/* Create a new archive */
/* Public */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Archive *
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
Add 'output file' option for pg_dumpall, especially useful for Win32, where output redirection of child processes (pg_dump) doesn't work. Dave Page
2007-01-25 04:30:43 +01:00
const int compression, ArchiveMode mode)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Add 'output file' option for pg_dumpall, especially useful for Win32, where output redirection of child processes (pg_dump) doesn't work. Dave Page
2007-01-25 04:30:43 +01:00
ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression, mode);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return (Archive *) AH;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/* Open an existing archive */
/* Public */
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
Archive *
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
OpenArchive(const char *FileSpec, const ArchiveFormat fmt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = _allocAH(FileSpec, fmt, 0, archModeRead);
return (Archive *) AH;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
CloseArchive(Archive *AHX)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int res = 0;
ArchiveHandle *AH = (ArchiveHandle *) AHX;
(*AH->ClosePtr) (AH);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Close the output */
if (AH->gzOut)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
res = GZCLOSE(AH->OF);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else if (AH->OF != stdout)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
res = fclose(AH->OF);
if (res != 0)
Add strerror to pg_dump error messages where missing.
2006-05-22 13:21:54 +02:00
die_horribly(AH, modulename, "could not close output file: %s\n",
strerror(errno));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
RestoreArchive(Archive *AHX, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
Actually, all of parallel restore's limitations should be tested earlier. On closer inspection, whining in restore_toc_entries_parallel is really much too late for any user-facing error case. The right place to do it is at the start of RestoreArchive(), before we've done anything interesting (suh as trying to DROP all the targets ...) Back-patch to 8.4, where parallel restore was introduced.
2011-08-29 04:27:48 +02:00
bool parallel_mode;
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
TocEntry *te;
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
teReqs reqs;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
OutputContext sav;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->ropt = ropt;
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
AH->stage = STAGE_INITIALIZING;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Forbid the switch combination --clean --create, which is pointless (why bother dropping individual objects in a just-created database?) as well as dangerous (as the code stands, the drops will be issued in the wrong database, namely the one you were originally connected to).
2001-10-23 23:26:44 +02:00
/*
* Check for nonsensical option combinations.
*
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
* NB: createDB+dropSchema is useless because if you're creating the DB,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* there's no need to drop individual items in it. Moreover, if we tried
* to do that then we'd issue the drops in the database initially
* connected to, not the one we will create, which is very bad...
Forbid the switch combination --clean --create, which is pointless (why bother dropping individual objects in a just-created database?) as well as dangerous (as the code stands, the drops will be issued in the wrong database, namely the one you were originally connected to).
2001-10-23 23:26:44 +02:00
*/
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
if (ropt->createDB && ropt->dropSchema)
Forbid the switch combination --clean --create, which is pointless (why bother dropping individual objects in a just-created database?) as well as dangerous (as the code stands, the drops will be issued in the wrong database, namely the one you were originally connected to).
2001-10-23 23:26:44 +02:00
die_horribly(AH, modulename, "-C and -c are incompatible options\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Throw an error when using -C and -1 at the same time in pg_restore. It's not possible to do CREATE DATABASE inside a transaction, so previously we just got a server error instead. Backpatch to 8.2, which is where the -1 feature appeared.
2009-01-13 12:44:56 +01:00
/*
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
* -C is not compatible with -1, because we can't create a database inside
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* a transaction block.
Throw an error when using -C and -1 at the same time in pg_restore. It's not possible to do CREATE DATABASE inside a transaction, so previously we just got a server error instead. Backpatch to 8.2, which is where the -1 feature appeared.
2009-01-13 12:44:56 +01:00
*/
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
if (ropt->createDB && ropt->single_txn)
Throw an error when using -C and -1 at the same time in pg_restore. It's not possible to do CREATE DATABASE inside a transaction, so previously we just got a server error instead. Backpatch to 8.2, which is where the -1 feature appeared.
2009-01-13 12:44:56 +01:00
die_horribly(AH, modulename, "-C and -1 are incompatible options\n");
Forbid the switch combination --clean --create, which is pointless (why bother dropping individual objects in a just-created database?) as well as dangerous (as the code stands, the drops will be issued in the wrong database, namely the one you were originally connected to).
2001-10-23 23:26:44 +02:00
Actually, all of parallel restore's limitations should be tested earlier. On closer inspection, whining in restore_toc_entries_parallel is really much too late for any user-facing error case. The right place to do it is at the start of RestoreArchive(), before we've done anything interesting (suh as trying to DROP all the targets ...) Back-patch to 8.4, where parallel restore was introduced.
2011-08-29 04:27:48 +02:00
/*
* If we're going to do parallel restore, there are some restrictions.
*/
parallel_mode = (ropt->number_of_jobs > 1 && ropt->useDB);
if (parallel_mode)
{
/* We haven't got round to making this work for all archive formats */
if (AH->ClonePtr == NULL || AH->ReopenPtr == NULL)
die_horribly(AH, modulename, "parallel restore is not supported with this archive file format\n");
/* Doesn't work if the archive represents dependencies as OIDs */
if (AH->version < K_VERS_1_8)
die_horribly(AH, modulename, "parallel restore is not supported with archives made by pre-8.0 pg_dump\n");
/*
* It's also not gonna work if we can't reopen the input file, so
* let's try that immediately.
*/
(AH->ReopenPtr) (AH);
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Make sure we won't need (de)compression we haven't got
*/
#ifndef HAVE_LIBZ
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
if (AH->compression != 0 && AH->PrintTocDataPtr !=NULL)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, false);
if (te->hadDumper && (reqs & REQ_DATA) != 0)
die_horribly(AH, modulename, "cannot restore from compressed archive (compression not supported in this installation)\n");
}
}
#endif
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
* If we're using a DB connection, then connect it.
*/
if (ropt->useDB)
{
More message munging and localization for pg_dump, especially the --verbose messages, which had not been considered so far. Output to the terminal should okay now; comments written into the dump are still English only, which may or may not be the desirable thing.
2001-07-03 22:21:50 +02:00
ahlog(AH, 1, "connecting to database for restore\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->version < K_VERS_1_3)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "direct database connections are not supported in pre-1.3 archives\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
- Fixed CONSTRAINT TRIGGER dump to record tgconstrelid properly - pgsql v7.0 compatbility
2001-04-25 09:03:20 +02:00
/* XXX Should get this from the archive */
AHX->minRemoteVersion = 070100;
AHX->maxRemoteVersion = 999999;
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
ConnectDatabase(AHX, ropt->dbname,
ropt->pghost, ropt->pgport, ropt->username,
Add a -w/--no-password option that prevents all password prompts to all programs that have a -W/--password option. In passing, remove the ancient PSQL_ALWAYS_GET_PASSWORDS compile option.
2009-02-26 17:02:39 +01:00
ropt->promptPassword);
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* If we're talking to the DB directly, don't send comments since they
* obscure SQL when displaying errors
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
*/
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
AH->noTocComments = 1;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
* are found then we set the dataOnly flag.
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* We could scan for wanted TABLE entries, but that is not the same as
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
* dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
*/
if (!ropt->dataOnly)
{
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
int impliedDataOnly = 1;
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
reqs = _tocEntryRequired(te, ropt, true);
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
if ((reqs & REQ_SCHEMA) != 0)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{ /* It's schema, and it's wanted */
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
impliedDataOnly = 0;
break;
}
}
if (impliedDataOnly)
{
ropt->dataOnly = impliedDataOnly;
More message munging and localization for pg_dump, especially the --verbose messages, which had not been considered so far. Output to the terminal should okay now; comments written into the dump are still English only, which may or may not be the desirable thing.
2001-07-03 22:21:50 +02:00
ahlog(AH, 1, "implied data-only restore\n");
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
- Only disable triggers in DataOnly (or implied data-only) restores. - Change -U option to -L to allow -U to specify username in future. (pg_restore)
2001-03-06 05:08:04 +01:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Setup the output file if necessary.
pgindent run for 8.3.
2007-11-15 22:14:46 +01:00
*/
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
sav = SaveOutput(AH);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (ropt->filename || ropt->compression)
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
SetOutput(AH, ropt->filename, ropt->compression);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
pgindent run.
2003-08-04 02:43:34 +02:00
Have pg_dump (-v) verbose mode output the pg_dump and server versions in text output mode, like we do in custom output mode. Jim Cox
2010-02-23 22:48:32 +01:00
if (AH->public.verbose)
{
Fix patch for printing backend and pg_dump versions so that it works in a desirable fashion in archive-dump cases, ie you should get the pg_dump version not the pg_restore version.
2010-02-24 03:42:55 +01:00
if (AH->archiveRemoteVersion)
ahprintf(AH, "-- Dumped from database version %s\n",
AH->archiveRemoteVersion);
if (AH->archiveDumpVersion)
ahprintf(AH, "-- Dumped by pg_dump version %s\n",
AH->archiveDumpVersion);
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
dumpTimestamp(AH, "Started on", AH->createDate);
Fix patch for printing backend and pg_dump versions so that it works in a desirable fashion in archive-dump cases, ie you should get the pg_dump version not the pg_restore version.
2010-02-24 03:42:55 +01:00
}
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
IMHO, --single-transaction should wrap *all* the commands in BEGIN/COMMIT, not just some of them.
2006-02-13 22:30:19 +01:00
if (ropt->single_txn)
Repair --single-transaction patch so it handles blobs correctly. Simon Riggs
2006-02-15 00:30:43 +01:00
{
if (AH->connection)
StartTransaction(AH);
else
ahprintf(AH, "BEGIN;\n\n");
}
IMHO, --single-transaction should wrap *all* the commands in BEGIN/COMMIT, not just some of them.
2006-02-13 22:30:19 +01:00
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
/*
* Establish important parameter values right away.
*/
_doSetFixedOutputState(AH);
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
AH->stage = STAGE_PROCESSING;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/*
* Drop the items at the start, in reverse order
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (ropt->dropSchema)
{
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
for (te = AH->toc->prev; te != AH->toc; te = te->prev)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
AH->currentTE = te;
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* We want anything that's selected and has a dropStmt */
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
if (((reqs & (REQ_SCHEMA | REQ_DATA)) != 0) && te->dropStmt)
More message munging and localization for pg_dump, especially the --verbose messages, which had not been considered so far. Output to the terminal should okay now; comments written into the dump are still English only, which may or may not be the desirable thing.
2001-07-03 22:21:50 +02:00
{
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
/* Select owner and schema as necessary */
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
_becomeOwner(AH, te);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
_selectOutputSchema(AH, te->namespace);
When dropping the schema, reconnect as owner of each object.
2001-01-27 13:35:29 +01:00
/* Drop it */
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
ahprintf(AH, "%s", te->dropStmt);
}
}
Move resetting of currSchema to a spot where it's not vulnerable to getting missed in possible future partial-drop scenarios. Improve comment. Per report from Zoltan Boszormenyi.
2007-11-24 21:26:49 +01:00
/*
* _selectOutputSchema may have set currSchema to reflect the effect
* of a "SET search_path" command it emitted. However, by now we may
* have dropped that schema; or it might not have existed in the first
* place. In either case the effective value of search_path will not
* be what we think. Forcibly reset currSchema so that we will
* re-establish the search_path setting when needed (after creating
* the schema).
*
* If we treated users as pg_dump'able objects then we'd need to reset
* currUser here too.
*/
if (AH->currSchema)
free(AH->currSchema);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
AH->currSchema = NULL;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
* In serial mode, we now process each non-ACL TOC entry.
*
* In parallel mode, turn control over to the parallel-restore logic.
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
Actually, all of parallel restore's limitations should be tested earlier. On closer inspection, whining in restore_toc_entries_parallel is really much too late for any user-facing error case. The right place to do it is at the start of RestoreArchive(), before we've done anything interesting (suh as trying to DROP all the targets ...) Back-patch to 8.4, where parallel restore was introduced.
2011-08-29 04:27:48 +02:00
if (parallel_mode)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
restore_toc_entries_parallel(AH);
else
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
(void) restore_toc_entry(AH, te, ropt, false);
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
/*
* Scan TOC again to output ownership commands and ACLs
*/
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
{
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
AH->currentTE = te;
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* Both schema and data objects might now have ownership/ACLs */
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
if ((reqs & (REQ_SCHEMA | REQ_DATA)) != 0)
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
{
Message styling
2004-10-22 18:04:35 +02:00
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
te->desc, te->tag);
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
_printTocEntry(AH, te, ropt, false, true);
}
}
IMHO, --single-transaction should wrap *all* the commands in BEGIN/COMMIT, not just some of them.
2006-02-13 22:30:19 +01:00
if (ropt->single_txn)
Repair --single-transaction patch so it handles blobs correctly. Simon Riggs
2006-02-15 00:30:43 +01:00
{
if (AH->connection)
CommitTransaction(AH);
else
ahprintf(AH, "COMMIT;\n\n");
}
Add psql option: -1 or --single-transaction Simon Riggs
2006-02-12 05:04:32 +01:00
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
if (AH->public.verbose)
dumpTimestamp(AH, "Completed on", time(NULL));
Put 'dump complete' message in the right place, so it comes out where it's supposed to when --file option is used.
2005-03-18 18:32:55 +01:00
ahprintf(AH, "--\n-- PostgreSQL database dump complete\n--\n\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
Modify pg_dump so that the preferred dump order is by name within object types, rather than by OID. This should help ensure consistent dump output from databases that are logically the same but have different histories, per recent discussion about 'diffing' databases. The patch is bulky because of renaming of fields, but not very complicated. Also, do some tweaking to cause BLOB restoration to be done in a better order, and clean up pg_restore's textual output to exactly match pg_dump.
2004-03-03 22:28:55 +01:00
* Clean up & we're done.
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
AH->stage = STAGE_FINALIZING;
Modify pg_dump so that the preferred dump order is by name within object types, rather than by OID. This should help ensure consistent dump output from databases that are logically the same but have different histories, per recent discussion about 'diffing' databases. The patch is bulky because of renaming of fields, but not very complicated. Also, do some tweaking to cause BLOB restoration to be done in a better order, and clean up pg_restore's textual output to exactly match pg_dump.
2004-03-03 22:28:55 +01:00
if (ropt->filename || ropt->compression)
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
RestoreOutput(AH, sav);
Modify pg_dump so that the preferred dump order is by name within object types, rather than by OID. This should help ensure consistent dump output from databases that are logically the same but have different histories, per recent discussion about 'diffing' databases. The patch is bulky because of renaming of fields, but not very complicated. Also, do some tweaking to cause BLOB restoration to be done in a better order, and clean up pg_restore's textual output to exactly match pg_dump.
2004-03-03 22:28:55 +01:00
if (ropt->useDB)
Invent on_exit_nicely for pg_dump. Per recent discussions on pgsql-hackers regarding parallel pg_dump.
2012-02-16 17:49:20 +01:00
DisconnectDatabase(&AH->public);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Restore a single TOC item. Used in both parallel and non-parallel restore;
* is_parallel is true if we are in a worker child process.
*
* Returns 0 normally, but WORKER_CREATE_DONE or WORKER_INHIBIT_DATA if
* the parallel parent has to make the corresponding status update.
*/
static int
restore_toc_entry(ArchiveHandle *AH, TocEntry *te,
RestoreOptions *ropt, bool is_parallel)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int retval = 0;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
teReqs reqs;
bool defnDumped;
AH->currentTE = te;
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, false);
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
}
defnDumped = false;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
_printTocEntry(AH, te, ropt, false, false);
defnDumped = true;
if (strcmp(te->desc, "TABLE") == 0)
{
if (AH->lastErrorTE == te)
{
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* We failed to create the table. If
* --no-data-for-failed-tables was given, mark the
* corresponding TABLE DATA to be ignored.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* In the parallel case this must be done in the parent, so we
* just set the return value.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (ropt->noDataForFailedTables)
{
if (is_parallel)
retval = WORKER_INHIBIT_DATA;
else
inhibit_data_for_failed_table(AH, te);
}
}
else
{
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* We created the table successfully. Mark the corresponding
* TABLE DATA for possible truncation.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* In the parallel case this must be done in the parent, so we
* just set the return value.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (is_parallel)
retval = WORKER_CREATE_DONE;
else
mark_create_done(AH, te);
}
}
/* If we created a DB, connect to it... */
if (strcmp(te->desc, "DATABASE") == 0)
{
ahlog(AH, 1, "connecting to new database \"%s\"\n", te->tag);
_reconnectToDB(AH, te->tag);
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
ropt->dbname = pg_strdup(te->tag);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
}
/*
* If we have a data component, then process it
*/
if ((reqs & REQ_DATA) != 0)
{
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* hadDumper will be set if there is genuine data component for this
* node. Otherwise, we need to check the defn field for statements
* that need to be executed in data-only restores.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (te->hadDumper)
{
/*
* If we can output the data, then restore it.
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
_printTocEntry(AH, te, ropt, true, false);
if (strcmp(te->desc, "BLOBS") == 0 ||
strcmp(te->desc, "BLOB COMMENTS") == 0)
{
ahlog(AH, 1, "restoring %s\n", te->desc);
_selectOutputSchema(AH, "pg_catalog");
(*AH->PrintTocDataPtr) (AH, te, ropt);
}
else
{
_disableTriggersIfNecessary(AH, te, ropt);
/* Select owner and schema as necessary */
_becomeOwner(AH, te);
_selectOutputSchema(AH, te->namespace);
ahlog(AH, 1, "restoring data for table \"%s\"\n",
te->tag);
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* In parallel restore, if we created the table earlier in
* the run then we wrap the COPY in a transaction and
* precede it with a TRUNCATE. If archiving is not on
* this prevents WAL-logging the COPY. This obtains a
* speedup similar to that from using single_txn mode in
* non-parallel restores.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (is_parallel && te->created)
{
/*
* Parallel restore is always talking directly to a
* server, so no need to see if we should issue BEGIN.
*/
StartTransaction(AH);
/*
* If the server version is >= 8.4, make sure we issue
* TRUNCATE with ONLY so that child tables are not
* wiped.
*/
ahprintf(AH, "TRUNCATE TABLE %s%s;\n\n",
(PQserverVersion(AH->connection) >= 80400 ?
"ONLY " : ""),
fmtId(te->tag));
}
/*
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
* If we have a copy statement, use it.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (te->copyStmt && strlen(te->copyStmt) > 0)
{
ahprintf(AH, "%s", te->copyStmt);
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
AH->outputKind = OUTPUT_COPYDATA;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
else
AH->outputKind = OUTPUT_OTHERDATA;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
(*AH->PrintTocDataPtr) (AH, te, ropt);
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
/*
* Terminate COPY if needed.
*/
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
if (AH->outputKind == OUTPUT_COPYDATA &&
RestoringToDB(AH))
EndDBCopyMode(AH, te);
AH->outputKind = OUTPUT_SQLCMDS;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* close out the transaction started above */
if (is_parallel && te->created)
CommitTransaction(AH);
_enableTriggersIfNecessary(AH, te, ropt);
}
}
}
else if (!defnDumped)
{
/* If we haven't already dumped the defn part, do so now */
ahlog(AH, 1, "executing %s %s\n", te->desc, te->tag);
_printTocEntry(AH, te, ropt, false, false);
}
}
return retval;
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
* Allocate a new RestoreOptions block.
* This is mainly so we can initialize it, but also for future expansion,
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
RestoreOptions *
NewRestoreOptions(void)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
RestoreOptions *opts;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
opts = (RestoreOptions *) pg_calloc(1, sizeof(RestoreOptions));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Add a --role option to pg_dump, pg_dumpall, and pg_restore. This allows performing dumps and restores in accordance with a security policy that forbids logging in directly as superuser, but instead specifies that you should log into an admin account and then SET ROLE to the superuser. In passing, clean up some ugly and mostly-broken code for quoting shell arguments in pg_dumpall. Benedek László, with some help from Tom Lane
2009-01-05 17:54:37 +01:00
/* set any fields that shouldn't default to zeroes */
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
opts->format = archUnknown;
Add a -w/--no-password option that prevents all password prompts to all programs that have a -W/--password option. In passing, remove the ancient PSQL_ALWAYS_GET_PASSWORDS compile option.
2009-02-26 17:02:39 +01:00
opts->promptPassword = TRI_DEFAULT;
Add --section option to pg_dump and pg_restore. Valid values are --pre-data, data and post-data. The option can be given more than once. --schema-only is equivalent to --section=pre-data --section=post-data. --data-only is equivalent to --section=data. Andrew Dunstan, reviewed by Joachim Wieland and Josh Berkus.
2011-12-17 01:09:38 +01:00
opts->dumpSections = DUMP_UNSECTIONED;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
return opts;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
/* This hack is only needed in a data-only restore */
if (!ropt->dataOnly || !ropt->disable_triggers)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
return;
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
/*
pgindent run.
2002-09-04 22:31:48 +02:00
* Become superuser if possible, since they are the only ones who can
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
* disable constraint triggers. If -S was not given, assume the initial
* user identity is a superuser. (XXX would it be better to become the
* table owner?)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
*/
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
_becomeUser(AH, ropt->superuser);
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
/*
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
* Disable them.
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
*/
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
_selectOutputSchema(AH, te->namespace);
- Only disable triggers for the table being loaded - disable triggers prior to BLOB load - better query for loading blob xrefs - Fixes to error handling code
2000-08-05 11:57:46 +02:00
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
ahprintf(AH, "ALTER TABLE %s DISABLE TRIGGER ALL;\n\n",
fmtId(te->tag));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
/* This hack is only needed in a data-only restore */
if (!ropt->dataOnly || !ropt->disable_triggers)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
return;
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
/*
pgindent run.
2002-09-04 22:31:48 +02:00
* Become superuser if possible, since they are the only ones who can
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
* disable constraint triggers. If -S was not given, assume the initial
* user identity is a superuser. (XXX would it be better to become the
* table owner?)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
*/
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
_becomeUser(AH, ropt->superuser);
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
/*
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
* Enable them.
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
*/
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
_selectOutputSchema(AH, te->namespace);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
Add ALTER TABLE ENABLE/DISABLE TRIGGER commands. Change pg_dump to use these instead of its previous hack of changing pg_class.reltriggers. Documentation is lacking, will add that later. Patch by Satoshi Nagayasu, review and some extra work by Tom Lane.
2005-08-24 00:40:47 +02:00
ahprintf(AH, "ALTER TABLE %s ENABLE TRIGGER ALL;\n\n",
fmtId(te->tag));
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/*
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
* This is a routine that is part of the dumper interface, hence the 'Archive*' parameter.
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*/
/* Public */
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t
WriteData(Archive *AHX, const void *data, size_t dLen)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (!AH->currToc)
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
die_horribly(AH, modulename, "internal error -- WriteData cannot be called outside the context of a DataDumper routine\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return (*AH->WriteDataPtr) (AH, data, dLen);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Create a new TOC entry. The TOC was designed as a TOC, but is now the
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
* repository for all metadata. But the name has stuck.
*/
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
ArchiveEntry(Archive *AHX,
CatalogId catalogId, DumpId dumpId,
const char *tag,
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
const char *namespace,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
const char *tablespace,
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
const char *owner, bool withOids,
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
const char *desc, teSection section,
const char *defn,
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
const char *dropStmt, const char *copyStmt,
const DumpId *deps, int nDeps,
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
DataDumperPtr dumpFn, void *dumpArg)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *newToc;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
newToc = (TocEntry *) pg_calloc(1, sizeof(TocEntry));
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
AH->tocCount++;
if (dumpId > AH->maxDumpId)
AH->maxDumpId = dumpId;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
newToc->prev = AH->toc->prev;
newToc->next = AH->toc;
AH->toc->prev->next = newToc;
AH->toc->prev = newToc;
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
newToc->catalogId = catalogId;
newToc->dumpId = dumpId;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
newToc->section = section;
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
newToc->tag = pg_strdup(tag);
newToc->namespace = namespace ? pg_strdup(namespace) : NULL;
newToc->tablespace = tablespace ? pg_strdup(tablespace) : NULL;
newToc->owner = pg_strdup(owner);
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
newToc->withOids = withOids;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
newToc->desc = pg_strdup(desc);
newToc->defn = pg_strdup(defn);
newToc->dropStmt = pg_strdup(dropStmt);
newToc->copyStmt = copyStmt ? pg_strdup(copyStmt) : NULL;
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
if (nDeps > 0)
{
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
newToc->dependencies = (DumpId *) pg_malloc(nDeps * sizeof(DumpId));
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
memcpy(newToc->dependencies, deps, nDeps * sizeof(DumpId));
newToc->nDeps = nDeps;
}
else
{
newToc->dependencies = NULL;
newToc->nDeps = 0;
}
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
newToc->dataDumper = dumpFn;
newToc->dataDumperArg = dumpArg;
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
newToc->hadDumper = dumpFn ? true : false;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
newToc->formatData = NULL;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (AH->ArchiveEntryPtr !=NULL)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
(*AH->ArchiveEntryPtr) (AH, newToc);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
PrintTOCSummary(Archive *AHX, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
TocEntry *te;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
OutputContext sav;
char *fmtName;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
sav = SaveOutput(AH);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (ropt->filename)
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
SetOutput(AH, ropt->filename, 0 /* no compression */ );
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
ahprintf(AH, ";\n; Archive created at %s", ctime(&AH->createDate));
ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression: %d\n",
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->archdbname, AH->tocCount, AH->compression);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
switch (AH->format)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
case archCustom:
fmtName = "CUSTOM";
break;
case archTar:
fmtName = "TAR";
break;
default:
fmtName = "UNKNOWN";
}
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
ahprintf(AH, "; Dump Version: %d.%d-%d\n", AH->vmaj, AH->vmin, AH->vrev);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
ahprintf(AH, "; Format: %s\n", fmtName);
Suppress compile warnings on OS X.
2002-10-27 03:52:10 +01:00
ahprintf(AH, "; Integer: %d bytes\n", (int) AH->intSize);
ahprintf(AH, "; Offset: %d bytes\n", (int) AH->offSize);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
if (AH->archiveRemoteVersion)
ahprintf(AH, "; Dumped from database version: %s\n",
AH->archiveRemoteVersion);
if (AH->archiveDumpVersion)
ahprintf(AH, "; Dumped by pg_dump version: %s\n",
AH->archiveDumpVersion);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
ahprintf(AH, ";\n;\n; Selected TOC Entries:\n;\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
/* We should print DATABASE entries whether or not -C was specified */
ropt->createDB = 1;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (ropt->verbose || _tocEntryRequired(te, ropt, true) != 0)
Include schema name in pg_restore -l output, per request from Phil Endecott.
2004-10-08 17:03:26 +02:00
ahprintf(AH, "%d; %u %u %s %s %s %s\n", te->dumpId,
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te->catalogId.tableoid, te->catalogId.oid,
Include schema name in pg_restore -l output, per request from Phil Endecott.
2004-10-08 17:03:26 +02:00
te->desc, te->namespace ? te->namespace : "-",
te->tag, te->owner);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (ropt->verbose && te->nDeps > 0)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ahprintf(AH, ";\tdepends on:");
for (i = 0; i < te->nDeps; i++)
ahprintf(AH, " %d", te->dependencies[i]);
ahprintf(AH, "\n");
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (ropt->filename)
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
RestoreOutput(AH, sav);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/***********
* BLOB Archival
***********/
/* Called by a dumper to signal start of a BLOB */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
StartBlob(Archive *AHX, Oid oid)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (!AH->StartBlobPtr)
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
die_horribly(AH, modulename, "large-object output not supported in chosen format\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
(*AH->StartBlobPtr) (AH, AH->currToc, oid);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return 1;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
/* Called by a dumper to signal end of a BLOB */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
EndBlob(Archive *AHX, Oid oid)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->EndBlobPtr)
(*AH->EndBlobPtr) (AH, AH->currToc, oid);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return 1;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
/**********
* BLOB Restoration
**********/
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Called by a format handler before any blobs are restored
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
StartRestoreBlobs(ArchiveHandle *AH)
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
{
Repair --single-transaction patch so it handles blobs correctly. Simon Riggs
2006-02-15 00:30:43 +01:00
if (!AH->ropt->single_txn)
{
if (AH->connection)
StartTransaction(AH);
else
ahprintf(AH, "BEGIN;\n\n");
}
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
AH->blobCount = 0;
}
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Called by a format handler after all blobs are restored
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
EndRestoreBlobs(ArchiveHandle *AH)
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
{
Repair --single-transaction patch so it handles blobs correctly. Simon Riggs
2006-02-15 00:30:43 +01:00
if (!AH->ropt->single_txn)
{
if (AH->connection)
CommitTransaction(AH);
else
ahprintf(AH, "COMMIT;\n\n");
}
Modify pg_dump so that the preferred dump order is by name within object types, rather than by OID. This should help ensure consistent dump output from databases that are logically the same but have different histories, per recent discussion about 'diffing' databases. The patch is bulky because of renaming of fields, but not very complicated. Also, do some tweaking to cause BLOB restoration to be done in a better order, and clean up pg_restore's textual output to exactly match pg_dump.
2004-03-03 22:28:55 +01:00
Gettext plural support In the backend, I changed only a handful of exemplary or important-looking instances to make use of the plural support; there is probably more work there. For the rest of the source, this should cover all relevant cases.
2009-03-26 23:26:08 +01:00
ahlog(AH, 1, ngettext("restored %d large object\n",
"restored %d large objects\n",
AH->blobCount),
AH->blobCount);
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
* Called by a format handler to initiate restoration of a blob
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
Make pg_dump/pg_restore --clean options drop large objects too. In passing, make invocations of lo_xxx functions a bit more schema-safe. Itagaki Takahiro
2009-07-21 23:46:10 +02:00
StartRestoreBlob(ArchiveHandle *AH, Oid oid, bool drop)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
bool old_blob_style = (AH->version < K_VERS_1_12);
More message munging and localization for pg_dump, especially the --verbose messages, which had not been considered so far. Output to the terminal should okay now; comments written into the dump are still English only, which may or may not be the desirable thing.
2001-07-03 22:21:50 +02:00
Oid loOid;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
AH->blobCount++;
Speed improvement for large object restore. Mario Weilguni
2002-04-24 04:21:04 +02:00
/* Initialize the LO Buffer */
AH->lo_buf_used = 0;
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
ahlog(AH, 2, "restoring large object with OID %u\n", oid);
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* With an old archive we must do drop and create logic here */
if (old_blob_style && drop)
Additional fixes for large object access control. Use pg_largeobject_metadata.oid instead of pg_largeobject.loid to enumerate existing large objects in pg_dump, pg_restore, and contrib modules.
2009-12-14 01:39:11 +01:00
DropBlobIfExists(AH, oid);
Make pg_dump/pg_restore --clean options drop large objects too. In passing, make invocations of lo_xxx functions a bit more schema-safe. Itagaki Takahiro
2009-07-21 23:46:10 +02:00
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
if (AH->connection)
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
{
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (old_blob_style)
{
loOid = lo_create(AH->connection, oid);
if (loOid == 0 || loOid != oid)
Also print the libpq error message when lo_create or lo_open fails
2010-03-18 21:00:51 +01:00
die_horribly(AH, modulename, "could not create large object %u: %s",
oid, PQerrorMessage(AH->connection));
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
}
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
AH->loFd = lo_open(AH->connection, oid, INV_WRITE);
if (AH->loFd == -1)
Also print the libpq error message when lo_create or lo_open fails
2010-03-18 21:00:51 +01:00
die_horribly(AH, modulename, "could not open large object %u: %s",
oid, PQerrorMessage(AH->connection));
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
}
else
{
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (old_blob_style)
ahprintf(AH, "SELECT pg_catalog.lo_open(pg_catalog.lo_create('%u'), %d);\n",
oid, INV_WRITE);
else
ahprintf(AH, "SELECT pg_catalog.lo_open('%u', %d);\n",
oid, INV_WRITE);
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->writingBlob = 1;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
EndRestoreBlob(ArchiveHandle *AH, Oid oid)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
Fix some more not-schema-aware queries in pg_dump. Also fix some places that would do the wrong thing with BLOB OIDs exceeding 2G.
2002-05-29 03:38:56 +02:00
if (AH->lo_buf_used > 0)
{
/* Write remaining bytes from the LO buffer */
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
dump_lo_buf(AH);
Fix some more not-schema-aware queries in pg_dump. Also fix some places that would do the wrong thing with BLOB OIDs exceeding 2G.
2002-05-29 03:38:56 +02:00
}
Speed improvement for large object restore. Mario Weilguni
2002-04-24 04:21:04 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->writingBlob = 0;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
if (AH->connection)
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
{
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
lo_close(AH->connection, AH->loFd);
AH->loFd = -1;
}
else
{
Make pg_dump/pg_restore --clean options drop large objects too. In passing, make invocations of lo_xxx functions a bit more schema-safe. Itagaki Takahiro
2009-07-21 23:46:10 +02:00
ahprintf(AH, "SELECT pg_catalog.lo_close(0);\n\n");
Added long-standing transaction when restoring BLOBS (uses commit every BLOB_BATCH_SIZE) Prevent dumping of languages from template1.
2000-10-31 15:20:30 +01:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/***********
* Sorting and Reordering
***********/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
SortTocFromFile(Archive *AHX, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH = (ArchiveHandle *) AHX;
FILE *fh;
Fix SortTocFromFile() to cope with lines that are too long for its buffer. The original coding supposed that a dump TOC file could never contain lines longer than 1K. The folly of that was exposed by a recent report from Per-Olov Esgard. We only really need to see the first dozen or two bytes of each line, since we're just trying to read off the numeric ID at the start of the line; so there's no need for a particularly huge buffer. What there is a need for is logic to not process continuation bufferloads. Back-patch to all supported branches, since it's always been like this.
2011-04-07 17:40:23 +02:00
char buf[100];
bool incomplete_line;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Allocate space for the 'wanted' array, and init it */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
ropt->idWanted = (bool *) pg_malloc(sizeof(bool) * AH->maxDumpId);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
memset(ropt->idWanted, 0, sizeof(bool) * AH->maxDumpId);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Setup the file */
fh = fopen(ropt->tocFile, PG_BINARY_R);
if (!fh)
Make pg_dump and friends consistently report both the filename and the errno string when complaining of fopen failures. Per gripe from Bob Pawley, it's not always instantly obvious to the user which name we tried to open.
2007-10-28 22:55:52 +01:00
die_horribly(AH, modulename, "could not open TOC file \"%s\": %s\n",
ropt->tocFile, strerror(errno));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix SortTocFromFile() to cope with lines that are too long for its buffer. The original coding supposed that a dump TOC file could never contain lines longer than 1K. The folly of that was exposed by a recent report from Per-Olov Esgard. We only really need to see the first dozen or two bytes of each line, since we're just trying to read off the numeric ID at the start of the line; so there's no need for a particularly huge buffer. What there is a need for is logic to not process continuation bufferloads. Back-patch to all supported branches, since it's always been like this.
2011-04-07 17:40:23 +02:00
incomplete_line = false;
Guard against duplicate IDs in input file in SortTocFromFile(). Per report from Brian Hackett.
2005-05-17 19:30:29 +02:00
while (fgets(buf, sizeof(buf), fh) != NULL)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Fix SortTocFromFile() to cope with lines that are too long for its buffer. The original coding supposed that a dump TOC file could never contain lines longer than 1K. The folly of that was exposed by a recent report from Per-Olov Esgard. We only really need to see the first dozen or two bytes of each line, since we're just trying to read off the numeric ID at the start of the line; so there's no need for a particularly huge buffer. What there is a need for is logic to not process continuation bufferloads. Back-patch to all supported branches, since it's always been like this.
2011-04-07 17:40:23 +02:00
bool prev_incomplete_line = incomplete_line;
int buflen;
char *cmnt;
char *endptr;
DumpId id;
TocEntry *te;
/*
* Some lines in the file might be longer than sizeof(buf). This is
* no problem, since we only care about the leading numeric ID which
* can be at most a few characters; but we have to skip continuation
* bufferloads when processing a long line.
*/
buflen = strlen(buf);
if (buflen > 0 && buf[buflen - 1] == '\n')
incomplete_line = false;
else
incomplete_line = true;
if (prev_incomplete_line)
continue;
Guard against duplicate IDs in input file in SortTocFromFile(). Per report from Brian Hackett.
2005-05-17 19:30:29 +02:00
/* Truncate line at comment, if any */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
cmnt = strchr(buf, ';');
if (cmnt != NULL)
cmnt[0] = '\0';
Guard against duplicate IDs in input file in SortTocFromFile(). Per report from Brian Hackett.
2005-05-17 19:30:29 +02:00
/* Ignore if all blank */
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
if (strspn(buf, " \t\r\n") == strlen(buf))
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
continue;
Guard against duplicate IDs in input file in SortTocFromFile(). Per report from Brian Hackett.
2005-05-17 19:30:29 +02:00
/* Get an ID, check it's valid and not already seen */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
id = strtol(buf, &endptr, 10);
Guard against duplicate IDs in input file in SortTocFromFile(). Per report from Brian Hackett.
2005-05-17 19:30:29 +02:00
if (endptr == buf || id <= 0 || id > AH->maxDumpId ||
ropt->idWanted[id - 1])
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
write_msg(modulename, "WARNING: line ignored: %s\n", buf);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
continue;
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Find TOC entry */
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te = getTocEntryByDumpId(AH, id);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (!te)
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
die_horribly(AH, modulename, "could not find entry for ID %d\n",
id);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/* Mark it wanted */
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
ropt->idWanted[id - 1] = true;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/*
* Move each item to the end of the list as it is selected, so that
* they are placed in the desired order. Any unwanted items will end
* up at the front of the list, which may seem unintuitive but it's
* what we need. In an ordinary serial restore that makes no
* difference, but in a parallel restore we need to mark unrestored
* items' dependencies as satisfied before we start examining
* restorable items. Otherwise they could have surprising
* side-effects on the order in which restorable items actually get
* restored.
*/
_moveBefore(AH, AH->toc, te);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (fclose(fh) != 0)
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
die_horribly(AH, modulename, "could not close TOC file: %s\n",
strerror(errno));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Code review for --no-data-for-failed-tables patch. Instead of trashing one of the program's core data structures, make use of the existing ability to selectively exclude TOC items by ID. Slightly more code but much less likely to create future maintenance problems.
2006-10-15 01:07:22 +02:00
/*
* Set up a dummy ID filter that selects all dump IDs
*/
void
InitDummyWantedList(Archive *AHX, RestoreOptions *ropt)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
/* Allocate space for the 'wanted' array, and init it to 1's */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
ropt->idWanted = (bool *) pg_malloc(sizeof(bool) * AH->maxDumpId);
Code review for --no-data-for-failed-tables patch. Instead of trashing one of the program's core data structures, make use of the existing ability to selectively exclude TOC items by ID. Slightly more code but much less likely to create future maintenance problems.
2006-10-15 01:07:22 +02:00
memset(ropt->idWanted, 1, sizeof(bool) * AH->maxDumpId);
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/**********************
* 'Convenience functions that look like standard IO functions
* for writing data when in dump mode.
**********************/
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
archputs(const char *s, Archive *AH)
{
return WriteData(AH, s, strlen(s));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/* Public */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
archprintf(Archive *AH, const char *fmt,...)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
char *p = NULL;
va_list ap;
int bSize = strlen(fmt) + 256;
int cnt = -1;
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* This is paranoid: deal with the possibility that vsnprintf is willing
* to ignore trailing null or returns > 0 even if string does not fit. It
* may be the case that it returns cnt = bufsize
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
*/
while (cnt < 0 || cnt >= (bSize - 1))
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (p != NULL)
free(p);
Fix pg_dump crashes caused by bogus use of va_start/va_end (only seen on some platforms, which is not too surprising considering how platform specific these macros must be).
2001-02-23 23:52:32 +01:00
bSize *= 2;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
p = (char *) pg_malloc(bSize);
Fix pg_dump crashes caused by bogus use of va_start/va_end (only seen on some platforms, which is not too surprising considering how platform specific these macros must be).
2001-02-23 23:52:32 +01:00
va_start(ap, fmt);
cnt = vsnprintf(p, bSize, fmt, ap);
va_end(ap);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
WriteData(AH, p, cnt);
free(p);
return cnt;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/*******************************
* Stuff below here should be 'private' to the archiver routines
*******************************/
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
static void
pg_dump: Add some const qualifiers
2012-02-07 22:20:29 +01:00
SetOutput(ArchiveHandle *AH, const char *filename, int compression)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
int fn;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (filename)
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
fn = -1;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else if (AH->FH)
fn = fileno(AH->FH);
else if (AH->fSpec)
{
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
fn = -1;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
filename = AH->fSpec;
}
else
fn = fileno(stdout);
/* If compression explicitly requested, use gzopen */
Please find attached a patch for the pg_dump directory which addresses: - The problems Jan reported - incompatibility with configure (now uses HAVE_LIBZ instead of HAVE_ZLIB) - a problem in auto-detecting archive file format on piped archives Philip Warner
2000-07-06 20:39:39 +02:00
#ifdef HAVE_LIBZ
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (compression != 0)
{
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
char fmode[10];
/* Don't use PG_BINARY_x since this is zlib */
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
sprintf(fmode, "wb%d", compression);
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
if (fn >= 0)
AH->OF = gzdopen(dup(fn), fmode);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else
AH->OF = gzopen(filename, fmode);
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
AH->gzOut = 1;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
#endif
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
{ /* Use fopen */
Add 'output file' option for pg_dumpall, especially useful for Win32, where output redirection of child processes (pg_dump) doesn't work. Dave Page
2007-01-25 04:30:43 +01:00
if (AH->mode == archModeAppend)
{
if (fn >= 0)
AH->OF = fdopen(dup(fn), PG_BINARY_A);
else
AH->OF = fopen(filename, PG_BINARY_A);
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else
Add 'output file' option for pg_dumpall, especially useful for Win32, where output redirection of child processes (pg_dump) doesn't work. Dave Page
2007-01-25 04:30:43 +01:00
{
if (fn >= 0)
AH->OF = fdopen(dup(fn), PG_BINARY_W);
else
AH->OF = fopen(filename, PG_BINARY_W);
}
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
AH->gzOut = 0;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
if (!AH->OF)
Make pg_dump and friends consistently report both the filename and the errno string when complaining of fopen failures. Per gripe from Bob Pawley, it's not always instantly obvious to the user which name we tried to open.
2007-10-28 22:55:52 +01:00
{
if (filename)
die_horribly(AH, modulename, "could not open output file \"%s\": %s\n",
filename, strerror(errno));
else
die_horribly(AH, modulename, "could not open output file: %s\n",
strerror(errno));
}
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
}
static OutputContext
SaveOutput(ArchiveHandle *AH)
{
OutputContext sav;
sav.OF = AH->OF;
sav.gzOut = AH->gzOut;
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return sav;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Mark a few functions as static or NOT_USED.
2006-07-18 19:42:01 +02:00
static void
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
RestoreOutput(ArchiveHandle *AH, OutputContext savedContext)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int res;
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->gzOut)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
res = GZCLOSE(AH->OF);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
res = fclose(AH->OF);
if (res != 0)
Complain if pg_restore is given both -d and -f options; this suggests the user is confused about whether -f is input or output file.
2003-10-20 23:05:12 +02:00
die_horribly(AH, modulename, "could not close output file: %s\n",
strerror(errno));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Suppress possibly-uninitialized-variable warnings from gcc 4.5. It appears that gcc 4.5 can issue such warnings for whole structs, not just scalar variables as in the past. Refactor some pg_dump code slightly so that the OutputContext local variables are always initialized, even if they won't be used. It's cheap enough to not be worth worrying about.
2011-01-22 23:56:42 +01:00
AH->gzOut = savedContext.gzOut;
AH->OF = savedContext.OF;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* Print formatted text to the output file (usually stdout).
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
ahprintf(ArchiveHandle *AH, const char *fmt,...)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
char *p = NULL;
va_list ap;
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
int bSize = strlen(fmt) + 256; /* Usually enough */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int cnt = -1;
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* This is paranoid: deal with the possibility that vsnprintf is willing
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
* to ignore trailing null or returns > 0 even if string does not fit.
* It may be the case that it returns cnt = bufsize.
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
*/
while (cnt < 0 || cnt >= (bSize - 1))
Fix pg_dump crashes caused by bogus use of va_start/va_end (only seen on some platforms, which is not too surprising considering how platform specific these macros must be).
2001-02-23 23:52:32 +01:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (p != NULL)
free(p);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
bSize *= 2;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
p = (char *) pg_malloc(bSize);
Fix pg_dump crashes caused by bogus use of va_start/va_end (only seen on some platforms, which is not too surprising considering how platform specific these macros must be).
2001-02-23 23:52:32 +01:00
va_start(ap, fmt);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
cnt = vsnprintf(p, bSize, fmt, ap);
Fix pg_dump crashes caused by bogus use of va_start/va_end (only seen on some platforms, which is not too surprising considering how platform specific these macros must be).
2001-02-23 23:52:32 +01:00
va_end(ap);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
ahwrite(p, 1, cnt, AH);
free(p);
return cnt;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
ahlog(ArchiveHandle *AH, int level, const char *fmt,...)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
va_list ap;
if (AH->debugLevel < level && (!AH->public.verbose || level > 1))
return;
va_start(ap, fmt);
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vwrite_msg(NULL, fmt, ap);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
va_end(ap);
}
- no longer put ACLs at end of dump - connect as appropriate user in pg_restore with db connection - dump owner of rule in pg_dump
2000-07-24 08:24:26 +02:00
/*
* Single place for logic which says 'We are restoring to a direct DB connection'.
*/
Mark a few functions as static or NOT_USED.
2006-07-18 19:42:01 +02:00
static int
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
RestoringToDB(ArchiveHandle *AH)
- no longer put ACLs at end of dump - connect as appropriate user in pg_restore with db connection - dump owner of rule in pg_dump
2000-07-24 08:24:26 +02:00
{
return (AH->ropt && AH->ropt->useDB && AH->connection);
}
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
/*
* Dump the current contents of the LO data buffer while writing a BLOB
*/
static void
dump_lo_buf(ArchiveHandle *AH)
{
if (AH->connection)
{
size_t res;
res = lo_write(AH->connection, AH->loFd, AH->lo_buf, AH->lo_buf_used);
Gettext plural support In the backend, I changed only a handful of exemplary or important-looking instances to make use of the plural support; there is probably more work there. For the rest of the source, this should cover all relevant cases.
2009-03-26 23:26:08 +01:00
ahlog(AH, 5, ngettext("wrote %lu byte of large object data (result = %lu)\n",
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
"wrote %lu bytes of large object data (result = %lu)\n",
Gettext plural support In the backend, I changed only a handful of exemplary or important-looking instances to make use of the plural support; there is probably more work there. For the rest of the source, this should cover all relevant cases.
2009-03-26 23:26:08 +01:00
AH->lo_buf_used),
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
(unsigned long) AH->lo_buf_used, (unsigned long) res);
if (res != AH->lo_buf_used)
die_horribly(AH, modulename,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
"could not write to large object (result: %lu, expected: %lu)\n",
(unsigned long) res, (unsigned long) AH->lo_buf_used);
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
}
else
{
Fix pg_dump to do the right thing when escaping the contents of large objects. The previous implementation got it right in most cases but failed in one: if you pg_dump into an archive with standard_conforming_strings enabled, then pg_restore to a script file (not directly to a database), the script will set standard_conforming_strings = on but then emit large object data as nonstandardly-escaped strings. At the moment the code is made to emit hex-format bytea strings when dumping to a script file. We might want to change to old-style escaping for backwards compatibility, but that would be slower and bulkier. If we do, it's just a matter of reimplementing appendByteaLiteral(). This has been broken for a long time, but given the lack of field complaints I'm not going to worry about back-patching.
2009-08-04 23:56:09 +02:00
PQExpBuffer buf = createPQExpBuffer();
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
Fix pg_dump to do the right thing when escaping the contents of large objects. The previous implementation got it right in most cases but failed in one: if you pg_dump into an archive with standard_conforming_strings enabled, then pg_restore to a script file (not directly to a database), the script will set standard_conforming_strings = on but then emit large object data as nonstandardly-escaped strings. At the moment the code is made to emit hex-format bytea strings when dumping to a script file. We might want to change to old-style escaping for backwards compatibility, but that would be slower and bulkier. If we do, it's just a matter of reimplementing appendByteaLiteral(). This has been broken for a long time, but given the lack of field complaints I'm not going to worry about back-patching.
2009-08-04 23:56:09 +02:00
appendByteaLiteralAHX(buf,
(const unsigned char *) AH->lo_buf,
AH->lo_buf_used,
AH);
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
/* Hack: turn off writingBlob so ahwrite doesn't recurse to here */
AH->writingBlob = 0;
Fix pg_dump to do the right thing when escaping the contents of large objects. The previous implementation got it right in most cases but failed in one: if you pg_dump into an archive with standard_conforming_strings enabled, then pg_restore to a script file (not directly to a database), the script will set standard_conforming_strings = on but then emit large object data as nonstandardly-escaped strings. At the moment the code is made to emit hex-format bytea strings when dumping to a script file. We might want to change to old-style escaping for backwards compatibility, but that would be slower and bulkier. If we do, it's just a matter of reimplementing appendByteaLiteral(). This has been broken for a long time, but given the lack of field complaints I'm not going to worry about back-patching.
2009-08-04 23:56:09 +02:00
ahprintf(AH, "SELECT pg_catalog.lowrite(0, %s);\n", buf->data);
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
AH->writingBlob = 1;
Fix pg_dump to do the right thing when escaping the contents of large objects. The previous implementation got it right in most cases but failed in one: if you pg_dump into an archive with standard_conforming_strings enabled, then pg_restore to a script file (not directly to a database), the script will set standard_conforming_strings = on but then emit large object data as nonstandardly-escaped strings. At the moment the code is made to emit hex-format bytea strings when dumping to a script file. We might want to change to old-style escaping for backwards compatibility, but that would be slower and bulkier. If we do, it's just a matter of reimplementing appendByteaLiteral(). This has been broken for a long time, but given the lack of field complaints I'm not going to worry about back-patching.
2009-08-04 23:56:09 +02:00
destroyPQExpBuffer(buf);
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
}
AH->lo_buf_used = 0;
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
/*
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
* Write buffer to the output file (usually stdout). This is used for
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* outputting 'restore' scripts etc. It is even possible for an archive
* format to create a custom output routine to 'fake' a restore if it
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
* wants to generate a script (see TAR output).
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
ahwrite(const void *ptr, size_t size, size_t nmemb, ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t res;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->writingBlob)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
size_t remaining = size * nmemb;
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
while (AH->lo_buf_used + remaining > AH->lo_buf_size)
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
{
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
size_t avail = AH->lo_buf_size - AH->lo_buf_used;
memcpy((char *) AH->lo_buf + AH->lo_buf_used, ptr, avail);
ptr = (const void *) ((const char *) ptr + avail);
remaining -= avail;
AH->lo_buf_used += avail;
dump_lo_buf(AH);
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
}
pg_dump can now dump large objects even in plain-text output mode, by using the recently added lo_create() function. The restore logic in pg_restore is greatly simplified as well, since there's no need anymore to try to adjust database references to match a new set of blob OIDs.
2005-06-21 22:45:44 +02:00
memcpy((char *) AH->lo_buf + AH->lo_buf_used, ptr, remaining);
AH->lo_buf_used += remaining;
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
return size * nmemb;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else if (AH->gzOut)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
{
Remove many -Wcast-qual warnings This addresses only those cases that are easy to fix by adding or moving a const qualifier or removing an unnecessary cast. There are many more complicated cases remaining.
2011-09-11 20:54:32 +02:00
res = GZWRITE(ptr, size, nmemb, AH->OF);
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
if (res != (nmemb * size))
Add strerror to pg_dump error messages where missing.
2006-05-22 13:21:54 +02:00
die_horribly(AH, modulename, "could not write to output file: %s\n", strerror(errno));
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
return res;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else if (AH->CustomOutPtr)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
res = AH->CustomOutPtr (AH, ptr, size * nmemb);
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
if (res != (nmemb * size))
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "could not write to custom output routine\n");
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
return res;
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
else
{
/*
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
* If we're doing a restore, and it's direct to DB, and we're
* connected then send it to the DB.
*/
- no longer put ACLs at end of dump - connect as appropriate user in pg_restore with db connection - dump owner of rule in pg_dump
2000-07-24 08:24:26 +02:00
if (RestoringToDB(AH))
Fix pg_restore's direct-to-database mode for standard_conforming_strings. pg_backup_db.c contained a mini SQL lexer with which it tried to identify boundaries between SQL commands, but that code was not designed to cope with standard_conforming_strings, and would get the wrong answer if a backslash immediately precedes a closing single quote in such a string, as per report from Julian Mehnle. The bug only affects direct-to-database restores from archive files made with standard_conforming_strings = on. Rather than complicating the code some more to try to fix that, let's just rip it all out. The only reason it was needed was to cope with COPY data embedded into ordinary archive entries, which was a layout that was used only for about the first three weeks of the archive format's existence, and never in any production release of pg_dump. Instead, just rely on the archive file layout to tell us whether we're printing COPY data or not. This bug represents a data corruption hazard in all releases in which standard_conforming_strings can be turned on, ie 8.2 and later, so back-patch to all supported branches.
2011-07-28 20:06:57 +02:00
return ExecuteSqlCommandBuf(AH, (const char *) ptr, size * nmemb);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
else
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
{
Remove many -Wcast-qual warnings This addresses only those cases that are easy to fix by adding or moving a const qualifier or removing an unnecessary cast. There are many more complicated cases remaining.
2011-09-11 20:54:32 +02:00
res = fwrite(ptr, size, nmemb, AH->OF);
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
if (res != nmemb)
Add strerror to pg_dump error messages where missing.
2006-05-22 13:21:54 +02:00
die_horribly(AH, modulename, "could not write to output file: %s\n",
strerror(errno));
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
return res;
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
/* Report a fatal error and exit(1) */
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
static void
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vdie_horribly(ArchiveHandle *AH, const char *modulename,
const char *fmt, va_list ap)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
{
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vwrite_msg(modulename, fmt, ap);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run.
2003-08-04 02:43:34 +02:00
if (AH)
{
Prevent core dump when die_horribly() is called with null AH pointer. Problem reported and fixed by Oliver Elphick.
2003-01-27 01:23:38 +01:00
if (AH->public.verbose)
write_msg(NULL, "*** aborted because of error\n");
Invent on_exit_nicely for pg_dump. Per recent discussions on pgsql-hackers regarding parallel pg_dump.
2012-02-16 17:49:20 +01:00
DisconnectDatabase(&AH->public);
Prevent core dump when die_horribly() is called with null AH pointer. Problem reported and fixed by Oliver Elphick.
2003-01-27 01:23:38 +01:00
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Invent on_exit_nicely for pg_dump. Per recent discussions on pgsql-hackers regarding parallel pg_dump.
2012-02-16 17:49:20 +01:00
exit_nicely(1);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
/* As above, but with variable arg list */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(ArchiveHandle *AH, const char *modulename, const char *fmt,...)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
va_list ap;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
va_start(ap, fmt);
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vdie_horribly(AH, modulename, fmt, ap);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
va_end(ap);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pg_dump: Further reduce reliance on global variables. This is another round of refactoring to make things simpler for parallel pg_dump. pg_dump.c now issues SQL queries through the relevant Archive object, rather than relying on the global variable g_conn. This commit isn't quite enough to get rid of g_conn entirely, but it makes a big dent in its utilization and, along the way, manages to be slightly less code than before.
2012-02-07 16:07:02 +01:00
/* As above, but with a complaint about a particular query. */
void
die_on_query_failure(ArchiveHandle *AH, const char *modulename,
const char *query)
{
write_msg(modulename, "query failed: %s",
PQerrorMessage(AH->connection));
die_horribly(AH, modulename, "query was: %s\n", query);
}
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
/* on some error, we may decide to go on... */
void
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
warn_or_die_horribly(ArchiveHandle *AH,
const char *modulename, const char *fmt,...)
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
{
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
va_list ap;
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
switch (AH->stage)
{
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
case STAGE_NONE:
/* Do nothing special */
break;
case STAGE_INITIALIZING:
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (AH->stage != AH->lastErrorStage)
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
write_msg(modulename, "Error while INITIALIZING:\n");
break;
case STAGE_PROCESSING:
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (AH->stage != AH->lastErrorStage)
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
write_msg(modulename, "Error while PROCESSING TOC:\n");
break;
case STAGE_FINALIZING:
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (AH->stage != AH->lastErrorStage)
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
write_msg(modulename, "Error while FINALIZING:\n");
break;
}
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
if (AH->currentTE != NULL && AH->currentTE != AH->lastErrorTE)
{
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
write_msg(modulename, "Error from TOC entry %d; %u %u %s %s %s\n",
AH->currentTE->dumpId,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
AH->currentTE->catalogId.tableoid, AH->currentTE->catalogId.oid,
AH->currentTE->desc, AH->currentTE->tag, AH->currentTE->owner);
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
}
AH->lastErrorStage = AH->stage;
AH->lastErrorTE = AH->currentTE;
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
va_start(ap, fmt);
> Please find attached a submission to add a "exit on error" option to > pg_restore, as it seems that some people have scripts that rely on the > previous "abort on error" default behavior when restoring data with a > direct connection. > > Fabien Coelho
2004-08-20 06:20:23 +02:00
if (AH->public.exit_on_error)
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vdie_horribly(AH, modulename, fmt, ap);
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
else
{
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
vwrite_msg(modulename, fmt, ap);
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
AH->public.n_errors++;
}
va_end(ap);
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
#ifdef NOT_USED
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
_moveAfter(ArchiveHandle *AH, TocEntry *pos, TocEntry *te)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/* Unlink te from list */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
te->prev->next = te->next;
te->next->prev = te->prev;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/* and insert it after "pos" */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
te->prev = pos;
te->next = pos->next;
pos->next->prev = te;
pos->next = te;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
#endif
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
_moveBefore(ArchiveHandle *AH, TocEntry *pos, TocEntry *te)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/* Unlink te from list */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
te->prev->next = te->next;
te->next->prev = te->prev;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
/* and insert it before "pos" */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
te->prev = pos->prev;
te->next = pos;
pos->prev->next = te;
pos->prev = te;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static TocEntry *
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
getTocEntryByDumpId(ArchiveHandle *AH, DumpId id)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
TocEntry *te;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
if (te->dumpId == id)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return te;
}
return NULL;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
teReqs
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
TocEntry *te = getTocEntryByDumpId(AH, id);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (!te)
return 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
return _tocEntryRequired(te, ropt, true);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
size_t
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
WriteOffset(ArchiveHandle *AH, pgoff_t o, int wasSet)
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
{
pgindent run.
2003-08-04 02:43:34 +02:00
int off;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
/* Save the flag */
(*AH->WriteBytePtr) (AH, wasSet);
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
/* Write out pgoff_t smallest byte first, prevents endian mismatch */
for (off = 0; off < sizeof(pgoff_t); off++)
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
{
pgindent run.
2003-08-04 02:43:34 +02:00
(*AH->WriteBytePtr) (AH, o & 0xFF);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
o >>= 8;
}
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
return sizeof(pgoff_t) + 1;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
}
int
pgindent run for 8.3.
2007-11-15 22:14:46 +01:00
ReadOffset(ArchiveHandle *AH, pgoff_t * o)
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
{
pgindent run.
2003-08-04 02:43:34 +02:00
int i;
int off;
int offsetFlg;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
/* Initialize to zero */
*o = 0;
/* Check for old version */
if (AH->version < K_VERS_1_7)
{
/* Prior versions wrote offsets using WriteInt */
i = ReadInt(AH);
/* -1 means not set */
if (i < 0)
pgindent run.
2003-08-04 02:43:34 +02:00
return K_OFFSET_POS_NOT_SET;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
else if (i == 0)
pgindent run.
2003-08-04 02:43:34 +02:00
return K_OFFSET_NO_DATA;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
/* Cast to pgoff_t because it was written as an int. */
*o = (pgoff_t) i;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
return K_OFFSET_POS_SET;
}
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Read the flag indicating the state of the data pointer. Check if valid
* and die if not.
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
*
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
2005-11-22 19:17:34 +01:00
* This used to be handled by a negative or zero pointer, now we use an
* extra byte specifically for the state.
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
*/
offsetFlg = (*AH->ReadBytePtr) (AH) & 0xFF;
switch (offsetFlg)
{
case K_OFFSET_POS_NOT_SET:
case K_OFFSET_NO_DATA:
case K_OFFSET_POS_SET:
pgindent run.
2003-08-04 02:43:34 +02:00
break;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
default:
Add strerror to pg_dump error messages where missing.
2006-05-22 13:21:54 +02:00
die_horribly(AH, modulename, "unexpected data offset flag %d\n", offsetFlg);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
}
/*
* Read the bytes
*/
for (off = 0; off < AH->offSize; off++)
{
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
if (off < sizeof(pgoff_t))
*o |= ((pgoff_t) ((*AH->ReadBytePtr) (AH))) << (off * 8);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
else
{
if ((*AH->ReadBytePtr) (AH) != 0)
pgindent run.
2003-08-04 02:43:34 +02:00
die_horribly(AH, modulename, "file offset in dump file is too large\n");
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
}
}
return offsetFlg;
}
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
WriteInt(ArchiveHandle *AH, int i)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int b;
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* This is a bit yucky, but I don't want to make the binary format very
* dependent on representation, and not knowing much about it, I write out
* a sign byte. If you change this, don't forget to change the file
* version #, and modify readInt to read the new format AS WELL AS the old
* formats.
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
*/
/* SIGN byte */
if (i < 0)
{
(*AH->WriteBytePtr) (AH, 1);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
i = -i;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
(*AH->WriteBytePtr) (AH, 0);
for (b = 0; b < AH->intSize; b++)
{
(*AH->WriteBytePtr) (AH, i & 0xFF);
Fix some more not-schema-aware queries in pg_dump. Also fix some places that would do the wrong thing with BLOB OIDs exceeding 2G.
2002-05-29 03:38:56 +02:00
i >>= 8;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
return AH->intSize + 1;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int
ReadInt(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int res = 0;
int bv,
b;
int sign = 0; /* Default positive */
int bitShift = 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->version > K_VERS_1_0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/* Read a sign byte */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
sign = (*AH->ReadBytePtr) (AH);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
for (b = 0; b < AH->intSize; b++)
{
bv = (*AH->ReadBytePtr) (AH) & 0xFF;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (bv != 0)
res = res + (bv << bitShift);
bitShift += 8;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (sign)
res = -res;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return res;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
WriteStr(ArchiveHandle *AH, const char *c)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t res;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (c)
{
res = WriteInt(AH, strlen(c));
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
res += (*AH->WriteBufPtr) (AH, c, strlen(c));
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
else
res = WriteInt(AH, -1);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return res;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
char *
ReadStr(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
char *buf;
int l;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
l = ReadInt(AH);
Fix pg_restore to guard against unexpected EOF while reading an archive file. Per report and partial patch from Chad Wagner.
2007-08-06 03:38:15 +02:00
if (l < 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
buf = NULL;
else
{
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
buf = (char *) pg_malloc(l + 1);
Fix pg_restore to guard against unexpected EOF while reading an archive file. Per report and partial patch from Chad Wagner.
2007-08-06 03:38:15 +02:00
if ((*AH->ReadBufPtr) (AH, (void *) buf, l) != l)
die_horribly(AH, modulename, "unexpected end of file\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
buf[l] = '\0';
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return buf;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Silence compiler warning.
2000-12-07 03:52:27 +01:00
static int
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
_discoverArchiveFormat(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
FILE *fh;
char sig[6]; /* More than enough */
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
size_t cnt;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int wantClose = 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
#if 0
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
write_msg(modulename, "attempting to ascertain archive format\n");
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
#endif
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->lookahead)
free(AH->lookahead);
AH->lookaheadSize = 512;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->lookahead = pg_calloc(1, 512);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->lookaheadLen = 0;
AH->lookaheadPos = 0;
Please find attached a patch for the pg_dump directory which addresses: - The problems Jan reported - incompatibility with configure (now uses HAVE_LIBZ instead of HAVE_ZLIB) - a problem in auto-detecting archive file format on piped archives Philip Warner
2000-07-06 20:39:39 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->fSpec)
{
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
struct stat st;
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
wantClose = 1;
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
/*
* Check if the specified archive is a directory. If so, check if
* there's a "toc.dat" (or "toc.dat.gz") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
char buf[MAXPGPATH];
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
if (snprintf(buf, MAXPGPATH, "%s/toc.dat", AH->fSpec) >= MAXPGPATH)
die_horribly(AH, modulename, "directory name too long: \"%s\"\n",
AH->fSpec);
if (stat(buf, &st) == 0 && S_ISREG(st.st_mode))
{
AH->format = archDirectory;
return AH->format;
}
#ifdef HAVE_LIBZ
if (snprintf(buf, MAXPGPATH, "%s/toc.dat.gz", AH->fSpec) >= MAXPGPATH)
die_horribly(AH, modulename, "directory name too long: \"%s\"\n",
AH->fSpec);
if (stat(buf, &st) == 0 && S_ISREG(st.st_mode))
{
AH->format = archDirectory;
return AH->format;
}
#endif
die_horribly(AH, modulename, "directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)\n",
AH->fSpec);
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
fh = NULL; /* keep compiler quiet */
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
}
else
{
fh = fopen(AH->fSpec, PG_BINARY_R);
if (!fh)
die_horribly(AH, modulename, "could not open input file \"%s\": %s\n",
AH->fSpec, strerror(errno));
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
Make pg_dump and friends consistently report both the filename and the errno string when complaining of fopen failures. Per gripe from Bob Pawley, it's not always instantly obvious to the user which name we tried to open.
2007-10-28 22:55:52 +01:00
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
fh = stdin;
Make pg_dump and friends consistently report both the filename and the errno string when complaining of fopen failures. Per gripe from Bob Pawley, it's not always instantly obvious to the user which name we tried to open.
2007-10-28 22:55:52 +01:00
if (!fh)
die_horribly(AH, modulename, "could not open input file: %s\n",
strerror(errno));
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
cnt = fread(sig, 1, 5, fh);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (cnt != 5)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
{
if (ferror(fh))
die_horribly(AH, modulename, "could not read input file: %s\n", strerror(errno));
else
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
die_horribly(AH, modulename, "input file is too short (read %lu, expected 5)\n",
(unsigned long) cnt);
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Save it, just in case we need it later */
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
strncpy(&AH->lookahead[0], sig, 5);
AH->lookaheadLen = 5;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (strncmp(sig, "PGDMP", 5) == 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
/*
* Finish reading (most of) a custom-format header.
*
* NB: this code must agree with ReadHead().
*/
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->vmaj = fgetc(fh);
AH->vmin = fgetc(fh);
/* Save these too... */
AH->lookahead[AH->lookaheadLen++] = AH->vmaj;
AH->lookahead[AH->lookaheadLen++] = AH->vmin;
/* Check header version; varies from V1.0 */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->vmaj > 1 || ((AH->vmaj == 1) && (AH->vmin > 0))) /* Version > 1.0 */
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->vrev = fgetc(fh);
AH->lookahead[AH->lookaheadLen++] = AH->vrev;
}
else
AH->vrev = 0;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
/* Make a convenient integer <maj><min><rev>00 */
AH->version = ((AH->vmaj * 256 + AH->vmin) * 256 + AH->vrev) * 256 + 0;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->intSize = fgetc(fh);
AH->lookahead[AH->lookaheadLen++] = AH->intSize;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
if (AH->version >= K_VERS_1_7)
{
AH->offSize = fgetc(fh);
AH->lookahead[AH->lookaheadLen++] = AH->offSize;
}
else
AH->offSize = AH->intSize;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->format = fgetc(fh);
AH->lookahead[AH->lookaheadLen++] = AH->format;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
Suggest use of psql when pg_restore gets a text dump.
2012-01-03 22:02:49 +01:00
* *Maybe* we have a tar archive format file or a text dump ...
* So, read first 512 byte header...
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
cnt = fread(&AH->lookahead[AH->lookaheadLen], 1, 512 - AH->lookaheadLen, fh);
AH->lookaheadLen += cnt;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Suggest use of psql when pg_restore gets a text dump.
2012-01-03 22:02:49 +01:00
if (AH->lookaheadLen >= strlen(TEXT_DUMPALL_HEADER) &&
(strncmp(AH->lookahead, TEXT_DUMP_HEADER, strlen(TEXT_DUMP_HEADER)) == 0 ||
strncmp(AH->lookahead, TEXT_DUMPALL_HEADER, strlen(TEXT_DUMPALL_HEADER)) == 0))
{
/* looks like it's probably a text format dump. so suggest they try psql */
die_horribly(AH, modulename, "input file appears to be a text format dump. Please use psql.\n");
}
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->lookaheadLen != 512)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "input file does not appear to be a valid archive (too short?)\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (!isValidTarHeader(AH->lookahead))
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "input file does not appear to be a valid archive\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->format = archTar;
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* If we can't seek, then mark the header as read */
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
if (fseeko(fh, 0, SEEK_SET) != 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* NOTE: Formats that use the lookahead buffer can unset this in their
* Init routine.
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
AH->readHeader = 1;
}
else
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->lookaheadLen = 0; /* Don't bother since we've reset the file */
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Close the file */
if (wantClose)
- Check ntuples == 1 for various SELECT statements. - Fix handling of --tables=* (multiple tables never worked properly, AFAICT) - strdup() the current user in DB routines - Check results of IO routines more carefully. - Check results of PQ routines more carefully. Have not fixed index output yet.
2001-01-12 05:32:07 +01:00
if (fclose(fh) != 0)
Add strerror to pg_dump error messages where missing.
2006-05-22 13:21:54 +02:00
die_horribly(AH, modulename, "could not close input file: %s\n",
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
strerror(errno));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return AH->format;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
/*
* Allocate an archive handle
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static ArchiveHandle *
_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const int compression, ArchiveMode mode)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
ArchiveHandle *AH;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
#if 0
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
write_msg(modulename, "allocating AH for %s, format %d\n", FileSpec, fmt);
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
#endif
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH = (ArchiveHandle *) pg_calloc(1, sizeof(ArchiveHandle));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
/* AH->debugLevel = 100; */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->vmaj = K_VERS_MAJOR;
AH->vmin = K_VERS_MINOR;
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
AH->vrev = K_VERS_REV;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* Make a convenient integer <maj><min><rev>00 */
AH->version = ((AH->vmaj * 256 + AH->vmin) * 256 + AH->vrev) * 256 + 0;
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* initialize for backwards compatible string processing */
Fix the inadvertent libpq ABI breakage discovered by Martin Pitt: the renumbering of encoding IDs done between 8.2 and 8.3 turns out to break 8.2 initdb and psql if they are run with an 8.3beta1 libpq.so. For the moment we can rearrange the order of enum pg_enc to keep the same number for everything except PG_JOHAB, which isn't a problem since there are no direct references to it in the 8.2 programs anyway. (This does force initdb unfortunately.) Going forward, we want to fix things so that encoding IDs can be changed without an ABI break, and this commit includes the changes needed to allow libpq's encoding IDs to be treated as fully independent of the backend's. The main issue is that libpq clients should not include pg_wchar.h or otherwise assume they know the specific values of libpq's encoding IDs, since they might encounter version skew between pg_wchar.h and the libpq.so they are using. To fix, have libpq officially export functions needed for encoding name<=>ID conversion and validity checking; it was doing this anyway unofficially. It's still the case that we can't renumber backend encoding IDs until the next bump in libpq's major version number, since doing so will break the 8.2-era client programs. However the code is now prepared to avoid this type of problem in future. Note that initdb is no longer a libpq client: we just pull in the two source files we need directly. The patch also fixes a few places that were being sloppy about checking for an unrecognized encoding name.
2007-10-13 22:18:42 +02:00
AH->public.encoding = 0; /* PG_SQL_ASCII */
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
AH->public.std_strings = false;
/* sql error handling */
AH->public.exit_on_error = true;
AH->public.n_errors = 0;
Fix patch for printing backend and pg_dump versions so that it works in a desirable fashion in archive-dump cases, ie you should get the pg_dump version not the pg_restore version.
2010-02-24 03:42:55 +01:00
AH->archiveDumpVersion = PG_VERSION;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->createDate = time(NULL);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->intSize = sizeof(int);
Fix pg_dump on win32 to properly dump files larger than 2Gb when using binary dump formats.
2007-02-19 16:05:06 +01:00
AH->offSize = sizeof(pgoff_t);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (FileSpec)
{
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->fSpec = pg_strdup(FileSpec);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/*
* Not used; maybe later....
*
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
* AH->workDir = pg_strdup(FileSpec); for(i=strlen(FileSpec) ; i > 0 ;
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
2005-11-22 19:17:34 +01:00
* i--) if (AH->workDir[i-1] == '/')
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->fSpec = NULL;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
AH->currUser = NULL; /* unknown */
AH->currSchema = NULL; /* ditto */
AH->currTablespace = NULL; /* ditto */
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
AH->currWithOids = -1; /* force SET */
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->toc = (TocEntry *) pg_calloc(1, sizeof(TocEntry));
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->toc->next = AH->toc;
AH->toc->prev = AH->toc;
AH->mode = mode;
AH->compression = compression;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse));
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Open stdout with no compression for AH output handle */
AH->gzOut = 0;
AH->OF = stdout;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Make pg_dump and pg_restore handle binary archive formats correctly when using stdout/stdin on Windows.
2005-01-26 20:44:43 +01:00
/*
* On Windows, we need to use binary mode to read/write non-text archive
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* formats. Force stdin/stdout into binary mode if that is what we are
* using.
Make pg_dump and pg_restore handle binary archive formats correctly when using stdout/stdin on Windows.
2005-01-26 20:44:43 +01:00
*/
#ifdef WIN32
Avoid changing stdin/stdout to binary mode on Windows unless that is really the source or destination of the archive. I think this will resolve recent complaints that password prompting is broken in pg_restore on Windows. Note that password prompting and reading from stdin is an unworkable combination on Windows ... but that was true anyway.
2005-09-11 02:36:14 +02:00
if (fmt != archNull &&
(AH->fSpec == NULL || strcmp(AH->fSpec, "") == 0))
Make pg_dump and pg_restore handle binary archive formats correctly when using stdout/stdin on Windows.
2005-01-26 20:44:43 +01:00
{
if (mode == archModeWrite)
setmode(fileno(stdout), O_BINARY);
else
setmode(fileno(stdin), O_BINARY);
}
#endif
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (fmt == archUnknown)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->format = _discoverArchiveFormat(AH);
else
AH->format = fmt;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Add a -w/--no-password option that prevents all password prompts to all programs that have a -W/--password option. In passing, remove the ancient PSQL_ALWAYS_GET_PASSWORDS compile option.
2009-02-26 17:02:39 +01:00
AH->promptPassword = TRI_DEFAULT;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
switch (AH->format)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
case archCustom:
InitArchiveFmt_Custom(AH);
break;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
case archNull:
InitArchiveFmt_Null(AH);
break;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Add 'directory' format to pg_dump. The new directory format is compatible with the 'tar' format, in that untarring a tar format archive produces a valid directory format archive. Joachim Wieland and Heikki Linnakangas
2011-01-23 22:10:15 +01:00
case archDirectory:
InitArchiveFmt_Directory(AH);
break;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
case archTar:
InitArchiveFmt_Tar(AH);
break;
default:
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
die_horribly(AH, modulename, "unrecognized file format \"%d\"\n", fmt);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return AH;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
WriteDataChunks(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
TocEntry *te;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
StartDataPtr startPtr;
EndDataPtr endPtr;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
if (te->dataDumper != NULL)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->currToc = te;
/* printf("Writing data for %d (%x)\n", te->id, te); */
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (strcmp(te->desc, "BLOBS") == 0)
{
startPtr = AH->StartBlobsPtr;
endPtr = AH->EndBlobsPtr;
}
else
{
startPtr = AH->StartDataPtr;
endPtr = AH->EndDataPtr;
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (startPtr != NULL)
(*startPtr) (AH, te);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* printf("Dumper arg for %d is %x\n", te->id, te->dataDumperArg);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
*/
/*
* The user-provided DataDumper routine needs to call
* AH->WriteData
*/
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
(*te->dataDumper) ((Archive *) AH, te->dataDumperArg);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (endPtr != NULL)
(*endPtr) (AH, te);
AH->currToc = NULL;
}
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
WriteToc(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
TocEntry *te;
char workbuf[32];
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
int i;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* printf("%d TOC Entries to save\n", AH->tocCount); */
WriteInt(AH, AH->tocCount);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
WriteInt(AH, te->dumpId);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
WriteInt(AH, te->dataDumper ? 1 : 0);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
/* OID is recorded as a string for historical reasons */
sprintf(workbuf, "%u", te->catalogId.tableoid);
WriteStr(AH, workbuf);
sprintf(workbuf, "%u", te->catalogId.oid);
WriteStr(AH, workbuf);
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
WriteStr(AH, te->tag);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
WriteStr(AH, te->desc);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
WriteInt(AH, te->section);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
WriteStr(AH, te->defn);
WriteStr(AH, te->dropStmt);
WriteStr(AH, te->copyStmt);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
WriteStr(AH, te->namespace);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
WriteStr(AH, te->tablespace);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
WriteStr(AH, te->owner);
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
WriteStr(AH, te->withOids ? "true" : "false");
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
/* Dump list of dependencies */
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
for (i = 0; i < te->nDeps; i++)
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
sprintf(workbuf, "%d", te->dependencies[i]);
WriteStr(AH, workbuf);
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
}
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
WriteStr(AH, NULL); /* Terminate List */
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->WriteExtraTocPtr)
(*AH->WriteExtraTocPtr) (AH, te);
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
ReadToc(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
int i;
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
char *tmp;
DumpId *deps;
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
int depIdx;
int depSize;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
TocEntry *te;
Add --section option to pg_dump and pg_restore. Valid values are --pre-data, data and post-data. The option can be given more than once. --schema-only is equivalent to --section=pre-data --section=post-data. --data-only is equivalent to --section=data. Andrew Dunstan, reviewed by Joachim Wieland and Josh Berkus.
2011-12-17 01:09:38 +01:00
bool in_post_data = false;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->tocCount = ReadInt(AH);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
AH->maxDumpId = 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
for (i = 0; i < AH->tocCount; i++)
{
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
te = (TocEntry *) pg_calloc(1, sizeof(TocEntry));
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te->dumpId = ReadInt(AH);
if (te->dumpId > AH->maxDumpId)
AH->maxDumpId = te->dumpId;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
/* Sanity check */
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
if (te->dumpId <= 0)
die_horribly(AH, modulename,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
"entry ID %d out of range -- perhaps a corrupt TOC\n",
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te->dumpId);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
te->hadDumper = ReadInt(AH);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
if (AH->version >= K_VERS_1_8)
{
tmp = ReadStr(AH);
sscanf(tmp, "%u", &te->catalogId.tableoid);
free(tmp);
}
else
te->catalogId.tableoid = InvalidOid;
tmp = ReadStr(AH);
sscanf(tmp, "%u", &te->catalogId.oid);
free(tmp);
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
te->tag = ReadStr(AH);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
te->desc = ReadStr(AH);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (AH->version >= K_VERS_1_11)
{
te->section = ReadInt(AH);
}
else
{
/*
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
* Rules for pre-8.4 archives wherein pg_dump hasn't classified
* the entries into sections. This list need not cover entry
* types added later than 8.4.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (strcmp(te->desc, "COMMENT") == 0 ||
Create an ALTER DEFAULT PRIVILEGES command, which allows users to adjust the privileges that will be applied to subsequently-created objects. Such adjustments are always per owning role, and can be restricted to objects created in particular schemas too. A notable benefit is that users can override the traditional default privilege settings, eg, the PUBLIC EXECUTE privilege traditionally granted by default for functions. Petr Jelinek
2009-10-05 21:24:49 +02:00
strcmp(te->desc, "ACL") == 0 ||
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
strcmp(te->desc, "ACL LANGUAGE") == 0)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
te->section = SECTION_NONE;
else if (strcmp(te->desc, "TABLE DATA") == 0 ||
strcmp(te->desc, "BLOBS") == 0 ||
strcmp(te->desc, "BLOB COMMENTS") == 0)
te->section = SECTION_DATA;
else if (strcmp(te->desc, "CONSTRAINT") == 0 ||
strcmp(te->desc, "CHECK CONSTRAINT") == 0 ||
strcmp(te->desc, "FK CONSTRAINT") == 0 ||
strcmp(te->desc, "INDEX") == 0 ||
strcmp(te->desc, "RULE") == 0 ||
strcmp(te->desc, "TRIGGER") == 0)
te->section = SECTION_POST_DATA;
else
te->section = SECTION_PRE_DATA;
}
Add --section option to pg_dump and pg_restore. Valid values are --pre-data, data and post-data. The option can be given more than once. --schema-only is equivalent to --section=pre-data --section=post-data. --data-only is equivalent to --section=data. Andrew Dunstan, reviewed by Joachim Wieland and Josh Berkus.
2011-12-17 01:09:38 +01:00
/* will stay true even for SECTION_NONE items */
if (te->section == SECTION_POST_DATA)
in_post_data = true;
te->inPostData = in_post_data;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
te->defn = ReadStr(AH);
te->dropStmt = ReadStr(AH);
if (AH->version >= K_VERS_1_3)
te->copyStmt = ReadStr(AH);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (AH->version >= K_VERS_1_6)
te->namespace = ReadStr(AH);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
if (AH->version >= K_VERS_1_10)
te->tablespace = ReadStr(AH);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
te->owner = ReadStr(AH);
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
if (AH->version >= K_VERS_1_9)
{
if (strcmp(ReadStr(AH), "true") == 0)
te->withOids = true;
else
te->withOids = false;
}
else
te->withOids = true;
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
/* Read TOC entry dependencies */
if (AH->version >= K_VERS_1_5)
{
depSize = 100;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
deps = (DumpId *) pg_malloc(sizeof(DumpId) * depSize);
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
depIdx = 0;
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
for (;;)
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
{
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
tmp = ReadStr(AH);
if (!tmp)
break; /* end of list */
Fix erroneous space calculation leading to core dump in dumpProcLangs, per report from Olivier Prenant. Also fix off-by-one space calculation in ReadToc; this woould not have hurt us until we had more than 100 dependencies for a single object, but wrong is wrong.
2003-05-04 00:18:59 +02:00
if (depIdx >= depSize)
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
{
depSize *= 2;
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
deps = (DumpId *) pg_realloc(deps, sizeof(DumpId) * depSize);
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
}
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
sscanf(tmp, "%d", &deps[depIdx]);
free(tmp);
depIdx++;
}
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
if (depIdx > 0) /* We have a non-null entry */
{
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
deps = (DumpId *) pg_realloc(deps, sizeof(DumpId) * depIdx);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te->dependencies = deps;
te->nDeps = depIdx;
}
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
else
Fix erroneous space calculation leading to core dump in dumpProcLangs, per report from Olivier Prenant. Also fix off-by-one space calculation in ReadToc; this woould not have hurt us until we had more than 100 dependencies for a single object, but wrong is wrong.
2003-05-04 00:18:59 +02:00
{
free(deps);
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
te->dependencies = NULL;
te->nDeps = 0;
Fix erroneous space calculation leading to core dump in dumpProcLangs, per report from Olivier Prenant. Also fix off-by-one space calculation in ReadToc; this woould not have hurt us until we had more than 100 dependencies for a single object, but wrong is wrong.
2003-05-04 00:18:59 +02:00
}
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
}
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
else
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
{
te->dependencies = NULL;
te->nDeps = 0;
}
Patch to put rudimentary dependency support into pg_dump. This addresses the UDT/function order problem. - Rudimentary support for dependencies in archives. Uses dependencies to modify the OID used in sorting TOC entries. This will NOT handle multi-level dependencies, but will manage simple relationships like UDTs & their functions. - Treat OIDs with more respect (avoid using ints, use macros for conversion & comparison).
2001-04-01 07:42:51 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->ReadExtraTocPtr)
(*AH->ReadExtraTocPtr) (AH, te);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
ahlog(AH, 3, "read TOC entry %d (ID %d) for %s %s\n",
i, te->dumpId, te->desc, te->tag);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* link completed entry into TOC circular list */
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
te->prev = AH->toc->prev;
AH->toc->prev->next = te;
AH->toc->prev = te;
te->next = AH->toc;
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* special processing immediately upon read for some items */
if (strcmp(te->desc, "ENCODING") == 0)
processEncodingEntry(AH, te);
else if (strcmp(te->desc, "STDSTRINGS") == 0)
processStdStringsEntry(AH, te);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
static void
processEncodingEntry(ArchiveHandle *AH, TocEntry *te)
{
/* te->defn should have the form SET client_encoding = 'foo'; */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
char *defn = pg_strdup(te->defn);
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
char *ptr1;
char *ptr2 = NULL;
int encoding;
ptr1 = strchr(defn, '\'');
if (ptr1)
ptr2 = strchr(++ptr1, '\'');
if (ptr2)
{
*ptr2 = '\0';
encoding = pg_char_to_encoding(ptr1);
if (encoding < 0)
die_horribly(AH, modulename, "unrecognized encoding \"%s\"\n",
ptr1);
AH->public.encoding = encoding;
}
else
die_horribly(AH, modulename, "invalid ENCODING item: %s\n",
te->defn);
free(defn);
}
static void
processStdStringsEntry(ArchiveHandle *AH, TocEntry *te)
{
/* te->defn should have the form SET standard_conforming_strings = 'x'; */
char *ptr1;
ptr1 = strchr(te->defn, '\'');
if (ptr1 && strncmp(ptr1, "'on'", 4) == 0)
AH->public.std_strings = true;
else if (ptr1 && strncmp(ptr1, "'off'", 5) == 0)
AH->public.std_strings = false;
else
die_horribly(AH, modulename, "invalid STDSTRINGS item: %s\n",
te->defn);
}
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
static teReqs
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Fix breakage created by addition of separate 'acl pass' in pg_dump. Also clean up incredibly poor style in TocIDRequired() usage.
2005-01-25 23:44:31 +01:00
teReqs res = REQ_ALL;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
strcmp(te->desc, "STDSTRINGS") == 0)
Use E'' strings internally only when standard_conforming_strings = 'off'. This allows pg_dump output with standard_conforming_strings = 'on' to generate proper strings that can be loaded into other databases without the backslash doubling we typically do. I have added the dumping of the standard_conforming_strings value to pg_dump. I also added standard backslash handling for plpgsql.
2006-05-27 01:48:54 +02:00
return 0;
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* If it's an ACL, maybe ignore it */
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if ((!include_acls || ropt->aclsSkip) && _tocEntryIsACL(te))
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Add a SECURITY LABEL command. This is intended as infrastructure to support integration with label-based mandatory access control systems such as SE-Linux. Further changes (mostly hooks) will be needed, but this is a big chunk of it. KaiGai Kohei and Robert Haas
2010-09-28 02:55:27 +02:00
/* If it's security labels, maybe ignore it */
Rename pg_dump --no-security-label to --no-security-labels Other similar options also use the plural form.
2011-05-19 22:20:11 +02:00
if (ropt->no_security_labels && strcmp(te->desc, "SECURITY LABEL") == 0)
Add a SECURITY LABEL command. This is intended as infrastructure to support integration with label-based mandatory access control systems such as SE-Linux. Further changes (mostly hooks) will be needed, but this is a big chunk of it. KaiGai Kohei and Robert Haas
2010-09-28 02:55:27 +02:00
return 0;
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* Ignore DATABASE entry unless we should create it */
Ensure that pg_restore -l will output DATABASE entries whether or not -C is specified. Per bug report from Russell Smith and ensuing discussion. Since this is a corner case behavioral change, I'm going to be conservative and not back-patch it. In passing, also rename the RestoreOptions field for the -C switch to something less generic than "create".
2010-05-15 23:41:16 +02:00
if (!ropt->createDB && strcmp(te->desc, "DATABASE") == 0)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
return 0;
Add --section option to pg_dump and pg_restore. Valid values are --pre-data, data and post-data. The option can be given more than once. --schema-only is equivalent to --section=pre-data --section=post-data. --data-only is equivalent to --section=data. Andrew Dunstan, reviewed by Joachim Wieland and Josh Berkus.
2011-12-17 01:09:38 +01:00
/* skip (all but) post data section as required */
/* table data is filtered if necessary lower down */
if (ropt->dumpSections != DUMP_UNSECTIONED)
{
if (!(ropt->dumpSections & DUMP_POST_DATA) && te->inPostData)
return 0;
if (!(ropt->dumpSections & DUMP_PRE_DATA) && ! te->inPostData && strcmp(te->desc, "TABLE DATA") != 0)
return 0;
}
Fix pg_restore -n option to do what the man page says it does. The original coding only worked if one of the selTypes restriction options was also given. Per report from Nick Johnson.
2006-04-13 00:18:48 +02:00
/* Check options for selective dump/restore */
if (ropt->schemaNames)
{
/* If no namespace is specified, it means all. */
if (!te->namespace)
return 0;
if (strcmp(ropt->schemaNames, te->namespace) != 0)
return 0;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (ropt->selTypes)
{
Add GRANT ON SEQUENCE syntax to support sequence-only permissions. Continue to support GRANT ON [TABLE] for sequences for backward compatibility; issue warning for invalid sequence permissions. [Backward compatibility warning message.] Add USAGE permission for sequences that allows only currval() and nextval(), not setval(). Mention object name in grant/revoke warnings because of possible multi-object operations.
2006-01-21 03:16:21 +01:00
if (strcmp(te->desc, "TABLE") == 0 ||
strcmp(te->desc, "TABLE DATA") == 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
{
if (!ropt->selTable)
return 0;
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
if (ropt->tableNames && strcmp(ropt->tableNames, te->tag) != 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else if (strcmp(te->desc, "INDEX") == 0)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (!ropt->selIndex)
return 0;
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
if (ropt->indexNames && strcmp(ropt->indexNames, te->tag) != 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else if (strcmp(te->desc, "FUNCTION") == 0)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (!ropt->selFunction)
return 0;
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
if (ropt->functionNames && strcmp(ropt->functionNames, te->tag) != 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else if (strcmp(te->desc, "TRIGGER") == 0)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (!ropt->selTrigger)
return 0;
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
if (ropt->triggerNames && strcmp(ropt->triggerNames, te->tag) != 0)
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
else
return 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
New pgindent run with fixes suggested by Tom. Patch manually reviewed, initdb/regression tests pass.
2001-11-05 18:46:40 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Check if we had a dataDumper. Indicates if the entry is schema or data
New pgindent run with fixes suggested by Tom. Patch manually reviewed, initdb/regression tests pass.
2001-11-05 18:46:40 +01:00
*/
if (!te->hadDumper)
{
/*
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* Special Case: If 'SEQUENCE SET' or anything to do with BLOBs, then
* it is considered a data entry. We don't need to check for the
* BLOBS entry or old-style BLOB COMMENTS, because they will have
* hadDumper = true ... but we do need to check new-style BLOB
* comments.
New pgindent run with fixes suggested by Tom. Patch manually reviewed, initdb/regression tests pass.
2001-11-05 18:46:40 +01:00
*/
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (strcmp(te->desc, "SEQUENCE SET") == 0 ||
strcmp(te->desc, "BLOB") == 0 ||
(strcmp(te->desc, "ACL") == 0 &&
strncmp(te->tag, "LARGE OBJECT ", 13) == 0) ||
(strcmp(te->desc, "COMMENT") == 0 &&
Add a SECURITY LABEL command. This is intended as infrastructure to support integration with label-based mandatory access control systems such as SE-Linux. Further changes (mostly hooks) will be needed, but this is a big chunk of it. KaiGai Kohei and Robert Haas
2010-09-28 02:55:27 +02:00
strncmp(te->tag, "LARGE OBJECT ", 13) == 0) ||
(strcmp(te->desc, "SECURITY LABEL") == 0 &&
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
strncmp(te->tag, "LARGE OBJECT ", 13) == 0))
New pgindent run with fixes suggested by Tom. Patch manually reviewed, initdb/regression tests pass.
2001-11-05 18:46:40 +01:00
res = res & REQ_DATA;
else
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
res = res & ~REQ_DATA;
}
Fix bug in sequence dumping using new setval function
2000-10-13 02:43:31 +02:00
You might want to run pg_indent on pg_backup_db.c however ... tom lane
2002-01-18 20:17:05 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Special case: <Init> type with <Max OID> tag; this is obsolete and we
* always ignore it.
You might want to run pg_indent on pg_backup_db.c however ... tom lane
2002-01-18 20:17:05 +01:00
*/
More clearly document in pg_dump when we are dealing with an object name as it appears in the schema dump, and index tags.
2002-07-04 17:35:07 +02:00
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
Solve the problem of OID collisions by probing for duplicate OIDs whenever we generate a new OID. This prevents occasional duplicate-OID errors that can otherwise occur once the OID counter has wrapped around. Duplicate relfilenode values are also checked for when creating new physical files. Per my recent proposal.
2005-08-12 03:36:05 +02:00
return 0;
Fix pg_restore to handle the 'set max oid' entry correctly in archives dumped by pg_dump -o. Per bug report posted by Bruce; fix is from Philip Warner, reviewed by Tom Lane.
2002-01-18 18:13:51 +01:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Mask it if we only want schema */
if (ropt->schemaOnly)
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
res = res & REQ_SCHEMA;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
/* Mask it we only want data */
- Don't dump COMMENTs in data-only dumps - Fix view dumping SQL for V7.0 - Fix bug when getting view oid with long view names - Treat SEQUENCE SET TOC entries as data entries rather than schema entries. - Make allowance for data entries that did not have a data dumper routine (eg. SEQUENCE SET)
2001-05-12 03:03:59 +02:00
if (ropt->dataOnly)
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
res = res & REQ_DATA;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
This patch fixes a bunch of spelling mistakes in comments throughout the PostgreSQL source code. Neil Conway
2003-03-10 23:28:22 +01:00
/* Mask it if we don't have a schema contribution */
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (!te->defn || strlen(te->defn) == 0)
- Fix compiler warning in pg_restore - Fix handling of {data/schema}-only restores when using a full backup file; prior version was restoring schema in data-only restores. Added enum to make code easier to understand.
2001-11-04 05:05:36 +01:00
res = res & ~REQ_SCHEMA;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Code review for --no-data-for-failed-tables patch. Instead of trashing one of the program's core data structures, make use of the existing ability to selectively exclude TOC items by ID. Slightly more code but much less likely to create future maintenance problems.
2006-10-15 01:07:22 +02:00
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
return 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
return res;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/*
* Identify TOC entries that are ACLs.
*/
static bool
_tocEntryIsACL(TocEntry *te)
{
/* "ACL LANGUAGE" was a crock emitted only in PG 7.4 */
if (strcmp(te->desc, "ACL") == 0 ||
strcmp(te->desc, "ACL LANGUAGE") == 0 ||
strcmp(te->desc, "DEFAULT ACL") == 0)
return true;
return false;
}
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
/*
* Issue SET commands for parameters that we want to have set the same way
* at all times during execution of a restore script.
*/
static void
_doSetFixedOutputState(ArchiveHandle *AH)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
/* Disable statement_timeout in archive for pg_restore/psql */
add ; missing due to my carelessness
2008-05-04 10:32:21 +02:00
ahprintf(AH, "SET statement_timeout = 0;\n");
Place statement timeout more appropriately, per gripe from Tom.
2008-05-04 05:46:08 +02:00
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* Select the correct character set encoding */
ahprintf(AH, "SET client_encoding = '%s';\n",
pg_encoding_to_char(AH->public.encoding));
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
/* Select the correct string literal syntax */
ahprintf(AH, "SET standard_conforming_strings = %s;\n",
AH->public.std_strings ? "on" : "off");
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
Add a --role option to pg_dump, pg_dumpall, and pg_restore. This allows performing dumps and restores in accordance with a security policy that forbids logging in directly as superuser, but instead specifies that you should log into an admin account and then SET ROLE to the superuser. In passing, clean up some ugly and mostly-broken code for quoting shell arguments in pg_dumpall. Benedek László, with some help from Tom Lane
2009-01-05 17:54:37 +01:00
/* Select the role to be used during restore */
if (AH->ropt && AH->ropt->use_role)
ahprintf(AH, "SET ROLE %s;\n", fmtId(AH->ropt->use_role));
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
/* Make sure function checking is disabled */
ahprintf(AH, "SET check_function_bodies = false;\n");
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
/* Avoid annoying notices etc */
ahprintf(AH, "SET client_min_messages = warning;\n");
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
if (!AH->public.std_strings)
ahprintf(AH, "SET escape_string_warning = off;\n");
They are two different problems; the TOC entry is important for any multiline command or to rerun the command easily later. Whereas displaying the failed SQL command is a matter of fixing the error messages. The latter is complicated by failed COPY commands which, with die-on-errors off, results in the data being processed as a command, so dumping the command will dump all of the data. In the case of long commands, should the whole command be dumped? eg. (eg. several pages of function definition). In the case of the COPY command, I'm not sure what to do. Obviously, it would be best to avoid sending the data, but the data and command are combined (from memory). Also, the 'data' may be in the form of INSERT statements. Attached patch produces the first 125 chars of the command: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC Entry 26; 1255 16449270 FUNCTION plpgsql_call_handler() pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_call_handler" already exists with same argument types Command was: CREATE FUNCTION plpgsql_call_handler() RETURNS language_handler AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_call_han... pg_restore: [archiver (db)] Error from TOC Entry 27; 1255 16449271 FUNCTION plpgsql_validator(oid) pjw pg_restore: [archiver (db)] could not execute query: ERROR: function "plpgsql_validator" already exists with same argument types Command was: CREATE FUNCTION plpgsql_validator(oid) RETURNS void AS '/var/lib/pgsql-8.0b1/lib/plpgsql', 'plpgsql_validator' LANGU... Philip Warner
2004-08-20 22:00:34 +02:00
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
ahprintf(AH, "\n");
}
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
/*
* Issue a SET SESSION AUTHORIZATION command. Caller is responsible
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
* for updating state if appropriate. If user is NULL or an empty string,
* the specification DEFAULT will be used.
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
*/
static void
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
_doSetSessionAuth(ArchiveHandle *AH, const char *user)
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
{
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
PQExpBuffer cmd = createPQExpBuffer();
pgindent run.
2002-09-04 22:31:48 +02:00
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
appendPQExpBuffer(cmd, "SET SESSION AUTHORIZATION ");
pgindent run.
2002-09-04 22:31:48 +02:00
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
/*
* SQL requires a string literal here. Might as well be correct.
*/
if (user && *user)
Fix up pg_dump to do string escaping fully correctly for client encoding and standard_conforming_strings; likewise for the other client programs that need it. As per previous discussion, a pg_dump dump now conforms to the standard_conforming_strings setting of the source database. We don't use E'' syntax in the dump, thereby improving portability of the SQL. I added a SET escape_strings_warning = off command to keep the dumps from getting a lot of back-chatter from that.
2006-05-28 23:13:54 +02:00
appendStringLiteralAHX(cmd, user, AH);
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
else
appendPQExpBuffer(cmd, "DEFAULT");
appendPQExpBuffer(cmd, ";");
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (RestoringToDB(AH))
{
PGresult *res;
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
res = PQexec(AH->connection, cmd->data);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
/* NOT warn_or_die_horribly... use -O instead to skip this. */
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
die_horribly(AH, modulename, "could not set session user to \"%s\": %s",
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
user, PQerrorMessage(AH->connection));
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
PQclear(res);
}
else
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
ahprintf(AH, "%s\n\n", cmd->data);
destroyPQExpBuffer(cmd);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
}
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
/*
* Issue a SET default_with_oids command. Caller is responsible
* for updating state if appropriate.
*/
static void
_doSetWithOids(ArchiveHandle *AH, const bool withOids)
{
PQExpBuffer cmd = createPQExpBuffer();
appendPQExpBuffer(cmd, "SET default_with_oids = %s;", withOids ?
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
"true" : "false");
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
if (RestoringToDB(AH))
{
PGresult *res;
res = PQexec(AH->connection, cmd->data);
if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
warn_or_die_horribly(AH, modulename,
Please find attached a small patch so that "pg_restore" ignores some sql errors. This is the second submission, which integrates Tom comments about localisation and exit code. I also added some comments about one sql command which is not ignored. Fabien COELHO
2004-04-22 04:39:10 +02:00
"could not set default_with_oids: %s",
PQerrorMessage(AH->connection));
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
PQclear(res);
}
else
ahprintf(AH, "%s\n\n", cmd->data);
destroyPQExpBuffer(cmd);
}
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
/*
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
* Issue the commands to connect to the specified database.
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
*
* If we're currently restoring right into a database, this will
pgindent run.
2002-09-04 22:31:48 +02:00
* actually establish a connection. Otherwise it puts a \connect into
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
* the script output.
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
*
* NULL dbname implies reconnecting to the current DB (pretty useless).
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
_reconnectToDB(ArchiveHandle *AH, const char *dbname)
- no longer put ACLs at end of dump - connect as appropriate user in pg_restore with db connection - dump owner of rule in pg_dump
2000-07-24 08:24:26 +02:00
{
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
if (RestoringToDB(AH))
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
ReconnectToServer(AH, dbname, NULL);
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
else
Be more wary about mixed-case database names and user names. Get the CREATE DATABASE command right in pg_dump -C case.
2002-02-11 01:18:20 +01:00
{
PQExpBuffer qry = createPQExpBuffer();
Fix some problems with restoring databases owned by non-superusers, as per bug #1249; and remove the last vestiges of using \connect to change authorization.
2004-09-10 22:05:18 +02:00
appendPQExpBuffer(qry, "\\connect %s\n\n",
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
dbname ? fmtId(dbname) : "-");
GCC 4.0 includes a new warning option, -Wformat-literal, that emits a warning when a variable is used as a format string for printf() and similar functions (if the variable is derived from untrusted data, it could include unexpected formatting sequences). This emits too many warnings to be enabled by default, but it does flag a few dubious constructs in the Postgres tree. This patch fixes up the obvious variants: functions that are passed a variable format string but no additional arguments. Most of these are harmless (e.g. the ruleutils stuff), but there is at least one actual bug here: if you create a trigger named "%sfoo", pg_dump will read uninitialized memory and fail to dump the trigger correctly.
2005-04-30 10:08:51 +02:00
ahprintf(AH, "%s", qry->data);
Be more wary about mixed-case database names and user names. Get the CREATE DATABASE command right in pg_dump -C case.
2002-02-11 01:18:20 +01:00
destroyPQExpBuffer(qry);
}
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* NOTE: currUser keeps track of what the imaginary session user in our
* script is. It's now effectively reset to the original userID.
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
2001-10-25 07:50:21 +02:00
*/
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
if (AH->currUser)
free(AH->currUser);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
AH->currUser = NULL;
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* don't assume we still know the output schema, tablespace, etc either */
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
if (AH->currSchema)
free(AH->currSchema);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
AH->currSchema = NULL;
if (AH->currTablespace)
free(AH->currTablespace);
AH->currTablespace = NULL;
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
AH->currWithOids = -1;
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
Cause pg_dump to emit a 'SET client_encoding' command at the start of any restore operation, thereby ensuring that dumped data is interpreted the same way it was dumped even if the target database has a different encoding. Per suggestions from Pavel Stehule and others. Also, simplify scheme for handling check_function_bodies ... we may as well just set that at the head of the script.
2004-02-24 04:35:19 +01:00
/* re-establish fixed state */
_doSetFixedOutputState(AH);
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
}
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
/*
* Become the specified user, and update state to avoid redundant commands
*
* NULL or empty argument is taken to mean restoring the session default
*/
static void
_becomeUser(ArchiveHandle *AH, const char *user)
{
if (!user)
user = ""; /* avoid null pointers */
if (AH->currUser && strcmp(AH->currUser, user) == 0)
return; /* no need to do anything */
_doSetSessionAuth(AH, user);
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* NOTE: currUser keeps track of what the imaginary session user in our
* script is
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
*/
if (AH->currUser)
free(AH->currUser);
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->currUser = pg_strdup(user);
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
}
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
/*
Fix various instances of "the the". Two of these were pointed out by Erik Rijkers; the rest I found.
2010-04-24 01:21:44 +02:00
* Become the owner of the given TOC entry object. If
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
* changes in ownership are not allowed, this doesn't do anything.
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
static void
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
_becomeOwner(ArchiveHandle *AH, TocEntry *te)
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
{
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
if (AH->ropt && (AH->ropt->noOwner || !AH->ropt->use_setsessauth))
- Added --create, --no-owner, --superuser, --no-reconnect (pg_dump & pg_restore) - Added code to dump 'Create Schema' statement (pg_dump) - Don't bother to disable/enable triggers if we don't have a superuser (pg_restore) - Cleaned up code for reconnecting to database. - Force a reconnect as superuser before enabling/disabling triggers. - Added & Removed --throttle (pg_dump) - Fixed minor bug in language dumping code: expbuffres were not being reset. - Fixed version number initialization in _allocAH (pg_backup_archiver.c) - Added second connection when restoring BLOBs to allow temp. table to survive (db reconnection causes temp tables to be lost).
2000-08-01 17:51:45 +02:00
return;
pg_dump/pg_restore now always use SET SESSION AUTHORIZATION, not \connect, to control object ownership. The use-set-session-authorization and no-reconnect switches are obsolete (still accepted on the command line, but they don't do anything). This is a precursor to fixing handling of CREATE SCHEMA, which will be a separate commit.
2003-09-24 00:48:53 +02:00
_becomeUser(AH, te->owner);
- no longer put ACLs at end of dump - connect as appropriate user in pg_restore with db connection - dump owner of rule in pg_dump
2000-07-24 08:24:26 +02:00
}
Add option to output SET SESSION AUTHORIZATION commands rather than \connect, to avoid possible password prompts and such, at the drawback of having to have superuser access.
2001-08-22 22:23:24 +02:00
Use the new GUC variable default_with_oids in pg_dump, rather than using WITH/WITHOUT OIDS in dump files. This makes dump files more portable. I have updated the pg_dump version so old binary dumps will load fine. Pre-7.5 dumps use WITHOUT OIDS in SQL were needed, so they should be fine.
2004-03-24 04:06:08 +01:00
/*
* Set the proper default_with_oids value for the table.
*/
static void
_setWithOids(ArchiveHandle *AH, TocEntry *te)
{
if (AH->currWithOids != te->withOids)
{
_doSetWithOids(AH, te->withOids);
AH->currWithOids = te->withOids;
}
}
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
/*
* Issue the commands to select the specified schema as the current schema
* in the target database.
*/
static void
_selectOutputSchema(ArchiveHandle *AH, const char *schemaName)
{
Rework pg_dump namespace search criteria so that dumping of user objects having names conflicting with system objects will work --- the search path is now user-schema, pg_catalog rather than implicitly the other way around. Note this requires being careful to explicitly qualify references to system names whenever pg_catalog is not first in the search path. Also, add support for dumping ACLs of schemas.
2002-05-29 00:26:57 +02:00
PQExpBuffer qry;
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (!schemaName || *schemaName == '\0' ||
Fix a couple of rather-pointless-but-easily-fixed Coverity warnings. Per Martijn van Oosterhout.
2006-04-19 18:02:17 +02:00
(AH->currSchema && strcmp(AH->currSchema, schemaName) == 0))
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
return; /* no need to do anything */
Rework pg_dump namespace search criteria so that dumping of user objects having names conflicting with system objects will work --- the search path is now user-schema, pg_catalog rather than implicitly the other way around. Note this requires being careful to explicitly qualify references to system names whenever pg_catalog is not first in the search path. Also, add support for dumping ACLs of schemas.
2002-05-29 00:26:57 +02:00
qry = createPQExpBuffer();
appendPQExpBuffer(qry, "SET search_path = %s",
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
fmtId(schemaName));
Rework pg_dump namespace search criteria so that dumping of user objects having names conflicting with system objects will work --- the search path is now user-schema, pg_catalog rather than implicitly the other way around. Note this requires being careful to explicitly qualify references to system names whenever pg_catalog is not first in the search path. Also, add support for dumping ACLs of schemas.
2002-05-29 00:26:57 +02:00
if (strcmp(schemaName, "pg_catalog") != 0)
appendPQExpBuffer(qry, ", pg_catalog");
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (RestoringToDB(AH))
{
PGresult *res;
res = PQexec(AH->connection, qry->data);
if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
warn_or_die_horribly(AH, modulename,
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
"could not set search_path to \"%s\": %s",
schemaName, PQerrorMessage(AH->connection));
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
PQclear(res);
}
else
Rework pg_dump namespace search criteria so that dumping of user objects having names conflicting with system objects will work --- the search path is now user-schema, pg_catalog rather than implicitly the other way around. Note this requires being careful to explicitly qualify references to system names whenever pg_catalog is not first in the search path. Also, add support for dumping ACLs of schemas.
2002-05-29 00:26:57 +02:00
ahprintf(AH, "%s;\n\n", qry->data);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
if (AH->currSchema)
free(AH->currSchema);
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->currSchema = pg_strdup(schemaName);
Rework pg_dump namespace search criteria so that dumping of user objects having names conflicting with system objects will work --- the search path is now user-schema, pg_catalog rather than implicitly the other way around. Note this requires being careful to explicitly qualify references to system names whenever pg_catalog is not first in the search path. Also, add support for dumping ACLs of schemas.
2002-05-29 00:26:57 +02:00
destroyPQExpBuffer(qry);
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
}
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
/*
* Issue the commands to select the specified tablespace as the current one
* in the target database.
*/
static void
_selectTablespace(ArchiveHandle *AH, const char *tablespace)
{
PQExpBuffer qry;
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
const char *want,
*have;
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
Support a --no-tablespaces option in pg_dump/pg_dumpall/pg_restore, so that dumps can be loaded into databases without the same tablespaces that the source had. The option acts by suppressing all "SET default_tablespace" commands, and also CREATE TABLESPACE commands in pg_dumpall's case. Gavin Roy, with documentation and minor fixes by me.
2008-03-20 18:36:58 +01:00
/* do nothing in --no-tablespaces mode */
if (AH->ropt->noTablespace)
return;
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
have = AH->currTablespace;
want = tablespace;
/* no need to do anything for non-tablespace object */
if (!want)
return;
if (have && strcmp(want, have) == 0)
return; /* no need to do anything */
qry = createPQExpBuffer();
if (strcmp(want, "") == 0)
{
/* We want the tablespace to be the database's default */
appendPQExpBuffer(qry, "SET default_tablespace = ''");
}
else
{
/* We want an explicit tablespace */
appendPQExpBuffer(qry, "SET default_tablespace = %s", fmtId(want));
}
if (RestoringToDB(AH))
{
PGresult *res;
res = PQexec(AH->connection, qry->data);
if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
warn_or_die_horribly(AH, modulename,
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
"could not set default_tablespace to %s: %s",
fmtId(want), PQerrorMessage(AH->connection));
PQclear(res);
}
else
ahprintf(AH, "%s;\n\n", qry->data);
if (AH->currTablespace)
free(AH->currTablespace);
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
AH->currTablespace = pg_strdup(want);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
destroyPQExpBuffer(qry);
}
First pass at schema-fying pg_dump/pg_restore. Much to do still, but the basic capability seems to work.
2002-05-11 00:36:27 +02:00
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
/*
* Extract an object description for a TOC entry, and append it to buf.
*
* This is not quite as general as it may seem, since it really only
* handles constructing the right thing to put into ALTER ... OWNER TO.
*
* The whole thing is pretty grotty, but we are kind of stuck since the
* information used is all that's available in older dump files.
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
*/
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
static void
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
_getObjectDescription(PQExpBuffer buf, TocEntry *te, ArchiveHandle *AH)
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
{
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
const char *type = te->desc;
/* Use ALTER TABLE for views and sequences */
Add GRANT ON SEQUENCE syntax to support sequence-only permissions. Continue to support GRANT ON [TABLE] for sequences for backward compatibility; issue warning for invalid sequence permissions. [Backward compatibility warning message.] Add USAGE permission for sequences that allows only currval() and nextval(), not setval(). Mention object name in grant/revoke warnings because of possible multi-object operations.
2006-01-21 03:16:21 +01:00
if (strcmp(type, "VIEW") == 0 || strcmp(type, "SEQUENCE") == 0)
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
type = "TABLE";
/* objects named by a schema and name */
DDL support for collations - collowner field - CREATE COLLATION - ALTER COLLATION - DROP COLLATION - COMMENT ON COLLATION - integration with extensions - pg_dump support for the above - dependency management - psql tab completion - psql \dO command
2011-02-12 14:54:13 +01:00
if (strcmp(type, "COLLATION") == 0 ||
strcmp(type, "CONVERSION") == 0 ||
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strcmp(type, "DOMAIN") == 0 ||
strcmp(type, "TABLE") == 0 ||
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
strcmp(type, "TYPE") == 0 ||
Basic foreign table support. Foreign tables are a core component of SQL/MED. This commit does not provide a working SQL/MED infrastructure, because foreign tables cannot yet be queried. Support for foreign table scans will need to be added in a future patch. However, this patch creates the necessary system catalog structure, syntax support, and support for ancillary operations such as COMMENT and SECURITY LABEL. Shigeru Hanada, heavily revised by Robert Haas
2011-01-02 05:48:11 +01:00
strcmp(type, "FOREIGN TABLE") == 0 ||
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
strcmp(type, "TEXT SEARCH DICTIONARY") == 0 ||
strcmp(type, "TEXT SEARCH CONFIGURATION") == 0)
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
{
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
appendPQExpBuffer(buf, "%s ", type);
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
if (te->namespace && te->namespace[0]) /* is null pre-7.3 */
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
appendPQExpBuffer(buf, "%s.", fmtId(te->namespace));
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Pre-7.3 pg_dump would sometimes (not always) put a fmtId'd name
* into te->tag for an index. This check is heuristic, so make its
* scope as narrow as possible.
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
*/
if (AH->version < K_VERS_1_7 &&
te->tag[0] == '"' &&
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
te->tag[strlen(te->tag) - 1] == '"' &&
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
strcmp(type, "INDEX") == 0)
appendPQExpBuffer(buf, "%s", te->tag);
else
appendPQExpBuffer(buf, "%s", fmtId(te->tag));
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
return;
}
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
/* objects named by just a name */
if (strcmp(type, "DATABASE") == 0 ||
Allow non-superuser database owners to create procedural languages. A DBA is allowed to create a language in his database if it's marked "tmpldbacreate" in pg_pltemplate. The factory default is that this is set for all standard trusted languages, but of course a superuser may adjust the settings. In service of this, add the long-foreseen owner column to pg_language; renaming, dropping, and altering owner of a PL now follow normal ownership rules instead of being superuser-only. Jeremy Drake, with some editorialization by Tom Lane.
2007-03-26 18:58:41 +02:00
strcmp(type, "PROCEDURAL LANGUAGE") == 0 ||
SQL/MED catalog manipulation facilities This doesn't do any remote or external things yet, but it gives modules like plproxy and dblink a standardized and future-proof system for managing their connection information. Martin Pihlak and Peter Eisentraut
2008-12-19 17:25:19 +01:00
strcmp(type, "SCHEMA") == 0 ||
strcmp(type, "FOREIGN DATA WRAPPER") == 0 ||
strcmp(type, "SERVER") == 0 ||
strcmp(type, "USER MAPPING") == 0)
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
{
appendPQExpBuffer(buf, "%s %s", type, fmtId(te->tag));
return;
}
Make pg_dump output more portable and more pleasing to look at. The -n and -N options were removed. Quoting is now smart enough to supply quotes if and only if necessary. Numerical types are now printed without quotes, except in cases of special values such as NaN. Boolean values printed as true and false. Most string literals now do not escape whitespace characters (newlines, etc.) for portability. SET SESSION AUTHORIZATION argument is a string literal, to follow SQL. Made commands output by pg_dump use consistent spacing and indentation.
2002-08-18 11:36:26 +02:00
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
/* BLOBs just have a name, but it's numeric so must not use fmtId */
if (strcmp(type, "BLOB") == 0)
{
appendPQExpBuffer(buf, "LARGE OBJECT %s", te->tag);
return;
}
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
/*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* These object types require additional decoration. Fortunately, the
* information needed is exactly what's in the DROP command.
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
*/
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
if (strcmp(type, "AGGREGATE") == 0 ||
strcmp(type, "FUNCTION") == 0 ||
strcmp(type, "OPERATOR") == 0 ||
Update pg_dump to support dumping operator families.
2007-01-23 18:54:50 +01:00
strcmp(type, "OPERATOR CLASS") == 0 ||
strcmp(type, "OPERATOR FAMILY") == 0)
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
{
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
/* Chop "DROP " off the front and make a modifiable copy */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
char *first = pg_strdup(te->dropStmt + 5);
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
char *last;
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
/* point to last character in string */
last = first + strlen(first) - 1;
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
/* Strip off any ';' or '\n' at the end */
while (last >= first && (*last == '\n' || *last == ';'))
last--;
*(last + 1) = '\0';
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
appendPQExpBufferStr(buf, first);
Pgindent run for 8.0.
2004-08-29 07:07:03 +02:00
free(first);
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
return;
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
}
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
write_msg(modulename, "WARNING: don't know how to set owner for object type %s\n",
type);
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
}
static void
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
_printTocEntry(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt, bool isData, bool acl_pass)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
/* ACLs are dumped only during acl pass */
if (acl_pass)
{
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (!_tocEntryIsACL(te))
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
return;
}
else
{
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (_tocEntryIsACL(te))
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
return;
}
/*
* Avoid dumping the public schema, as it will already be created ...
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* unless we are using --clean mode, in which case it's been deleted and
Prevent pg_dump from dumping the comment (if any) on the 'public' schema. This is to avoid uselessly requiring superuser permissions to restore the dump without errors. Pretty grotty, but no better alternative seems available, at least not in the near term.
2008-01-14 20:27:41 +01:00
* we'd better recreate it. Likewise for its comment, if any.
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
*/
Prevent pg_dump from dumping the comment (if any) on the 'public' schema. This is to avoid uselessly requiring superuser permissions to restore the dump without errors. Pretty grotty, but no better alternative seems available, at least not in the near term.
2008-01-14 20:27:41 +01:00
if (!ropt->dropSchema)
{
if (strcmp(te->desc, "SCHEMA") == 0 &&
strcmp(te->tag, "public") == 0)
return;
Add comment about why pg_dump doesn't dump the public schema comment.
2008-09-06 01:53:42 +02:00
/* The comment restore would require super-user privs, so avoid it. */
Prevent pg_dump from dumping the comment (if any) on the 'public' schema. This is to avoid uselessly requiring superuser permissions to restore the dump without errors. Pretty grotty, but no better alternative seems available, at least not in the near term.
2008-01-14 20:27:41 +01:00
if (strcmp(te->desc, "COMMENT") == 0 &&
strcmp(te->tag, "SCHEMA public") == 0)
return;
}
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
/* Select owner, schema, and tablespace as necessary */
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
_becomeOwner(AH, te);
_selectOutputSchema(AH, te->namespace);
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
_selectTablespace(AH, te->tablespace);
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
/* Set up OID mode too */
if (strcmp(te->desc, "TABLE") == 0)
_setWithOids(AH, te);
/* Emit header comment for item */
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
if (!AH->noTocComments)
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
{
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
const char *pfx;
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
char *sanitized_name;
char *sanitized_schema;
char *sanitized_owner;
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
if (isData)
pfx = "Data for ";
else
pfx = "";
ahprintf(AH, "--\n");
if (AH->public.verbose)
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
{
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
ahprintf(AH, "-- TOC entry %d (class %u OID %u)\n",
te->dumpId, te->catalogId.tableoid, te->catalogId.oid);
if (te->nDeps > 0)
{
int i;
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
ahprintf(AH, "-- Dependencies:");
for (i = 0; i < te->nDeps; i++)
ahprintf(AH, " %d", te->dependencies[i]);
ahprintf(AH, "\n");
}
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
}
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
/*
* Zap any line endings embedded in user-supplied fields, to prevent
* corruption of the dump (which could, in the worst case, present an
* SQL injection vulnerability if someone were to incautiously load a
* dump containing objects with maliciously crafted names).
*/
sanitized_name = replace_line_endings(te->tag);
if (te->namespace)
sanitized_schema = replace_line_endings(te->namespace);
else
sanitized_schema = pg_strdup("-");
if (!ropt->noOwner)
sanitized_owner = replace_line_endings(te->owner);
else
sanitized_owner = pg_strdup("-");
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
ahprintf(AH, "-- %sName: %s; Type: %s; Schema: %s; Owner: %s",
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
pfx, sanitized_name, te->desc, sanitized_schema,
sanitized_owner);
free(sanitized_name);
free(sanitized_schema);
free(sanitized_owner);
Support a --no-tablespaces option in pg_dump/pg_dumpall/pg_restore, so that dumps can be loaded into databases without the same tablespaces that the source had. The option acts by suppressing all "SET default_tablespace" commands, and also CREATE TABLESPACE commands in pg_dumpall's case. Gavin Roy, with documentation and minor fixes by me.
2008-03-20 18:36:58 +01:00
if (te->tablespace && !ropt->noTablespace)
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
{
char *sanitized_tablespace;
sanitized_tablespace = replace_line_endings(te->tablespace);
ahprintf(AH, "; Tablespace: %s", sanitized_tablespace);
free(sanitized_tablespace);
}
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
ahprintf(AH, "\n");
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
if (AH->PrintExtraTocPtr !=NULL)
Fix obviously-utterly-untested noTocComments code.
2004-08-30 21:44:14 +02:00
(*AH->PrintExtraTocPtr) (AH, te);
ahprintf(AH, "--\n\n");
Massive overhaul of pg_dump: make use of dependency information from pg_depend to determine a safe dump order. Defaults and check constraints can be emitted either as part of a table or domain definition, or separately if that's needed to break a dependency loop. Lots of old half-baked code for controlling dump order removed.
2003-12-06 04:00:16 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
/*
* Actually print the definition.
*
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* Really crude hack for suppressing AUTHORIZATION clause that old pg_dump
* versions put into CREATE SCHEMA. We have to do this when --no-owner
* mode is selected. This is ugly, but I see no other good way ...
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
*/
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
if (ropt->noOwner && strcmp(te->desc, "SCHEMA") == 0)
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
{
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
ahprintf(AH, "CREATE SCHEMA %s;\n\n\n", fmtId(te->tag));
Issue CREATE SCHEMA as the originally connecting user, with an AUTHORIZATION clause to specify the desired owner. This allows a superuser to restore schemas owned by users without CREATE-SCHEMA permissions (ie, schemas originally created by a superuser using AUTHORIZATION). --no-owner can be specified to suppress the AUTHORIZATION clause if need be.
2003-09-24 01:31:52 +02:00
}
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
else
Issue CREATE SCHEMA as the originally connecting user, with an AUTHORIZATION clause to specify the desired owner. This allows a superuser to restore schemas owned by users without CREATE-SCHEMA permissions (ie, schemas originally created by a superuser using AUTHORIZATION). --no-owner can be specified to suppress the AUTHORIZATION clause if need be.
2003-09-24 01:31:52 +02:00
{
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
if (strlen(te->defn) > 0)
ahprintf(AH, "%s\n\n", te->defn);
Change pg_dump to use ALTER OWNER commands instead of SET SESSION AUTHORIZATION commands by default. Move all GRANT and REVOKE commands to the end of the dump to avoid restore failures in several situations. Bring back --use-set-session-authorization option to get previous SET behaviour Christopher Kings-Lyne
2004-07-13 05:00:17 +02:00
}
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
/*
* If we aren't using SET SESSION AUTH to determine ownership, we must
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
* instead issue an ALTER OWNER command. We assume that anything without
* a DROP command is not a separately ownable object. All the categories
* with DROP commands must appear in one list or the other.
Fix breakage with PUBLIC schema. Try to untwist the remarkably contorted logic a little bit.
2004-08-13 23:37:28 +02:00
*/
if (!ropt->noOwner && !ropt->use_setsessauth &&
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strlen(te->owner) > 0 && strlen(te->dropStmt) > 0)
{
if (strcmp(te->desc, "AGGREGATE") == 0 ||
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
strcmp(te->desc, "BLOB") == 0 ||
DDL support for collations - collowner field - CREATE COLLATION - ALTER COLLATION - DROP COLLATION - COMMENT ON COLLATION - integration with extensions - pg_dump support for the above - dependency management - psql tab completion - psql \dO command
2011-02-12 14:54:13 +01:00
strcmp(te->desc, "COLLATION") == 0 ||
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strcmp(te->desc, "CONVERSION") == 0 ||
strcmp(te->desc, "DATABASE") == 0 ||
strcmp(te->desc, "DOMAIN") == 0 ||
strcmp(te->desc, "FUNCTION") == 0 ||
strcmp(te->desc, "OPERATOR") == 0 ||
strcmp(te->desc, "OPERATOR CLASS") == 0 ||
Update pg_dump to support dumping operator families.
2007-01-23 18:54:50 +01:00
strcmp(te->desc, "OPERATOR FAMILY") == 0 ||
Allow non-superuser database owners to create procedural languages. A DBA is allowed to create a language in his database if it's marked "tmpldbacreate" in pg_pltemplate. The factory default is that this is set for all standard trusted languages, but of course a superuser may adjust the settings. In service of this, add the long-foreseen owner column to pg_language; renaming, dropping, and altering owner of a PL now follow normal ownership rules instead of being superuser-only. Jeremy Drake, with some editorialization by Tom Lane.
2007-03-26 18:58:41 +02:00
strcmp(te->desc, "PROCEDURAL LANGUAGE") == 0 ||
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strcmp(te->desc, "SCHEMA") == 0 ||
strcmp(te->desc, "TABLE") == 0 ||
strcmp(te->desc, "TYPE") == 0 ||
strcmp(te->desc, "VIEW") == 0 ||
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
strcmp(te->desc, "SEQUENCE") == 0 ||
Basic foreign table support. Foreign tables are a core component of SQL/MED. This commit does not provide a working SQL/MED infrastructure, because foreign tables cannot yet be queried. Support for foreign table scans will need to be added in a future patch. However, this patch creates the necessary system catalog structure, syntax support, and support for ancillary operations such as COMMENT and SECURITY LABEL. Shigeru Hanada, heavily revised by Robert Haas
2011-01-02 05:48:11 +01:00
strcmp(te->desc, "FOREIGN TABLE") == 0 ||
Tsearch2 functionality migrates to core. The bulk of this work is by Oleg Bartunov and Teodor Sigaev, but I did a lot of editorializing, so anything that's broken is probably my fault. Documentation is nonexistent as yet, but let's land the patch so we can get some portability testing done.
2007-08-21 03:11:32 +02:00
strcmp(te->desc, "TEXT SEARCH DICTIONARY") == 0 ||
SQL/MED catalog manipulation facilities This doesn't do any remote or external things yet, but it gives modules like plproxy and dblink a standardized and future-proof system for managing their connection information. Martin Pihlak and Peter Eisentraut
2008-12-19 17:25:19 +01:00
strcmp(te->desc, "TEXT SEARCH CONFIGURATION") == 0 ||
strcmp(te->desc, "FOREIGN DATA WRAPPER") == 0 ||
strcmp(te->desc, "SERVER") == 0)
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
{
PQExpBuffer temp = createPQExpBuffer();
appendPQExpBuffer(temp, "ALTER ");
New routine _getObjectDescription() failed to cope with some aspects of pre-7.3 pg_dump archive files: namespace isn't there, and in some cases te->tag may already be quotified. Per report from Alan Pevec and followup testing.
2005-01-23 01:03:54 +01:00
_getObjectDescription(temp, te, AH);
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
appendPQExpBuffer(temp, " OWNER TO %s;", fmtId(te->owner));
ahprintf(AH, "%s\n\n", temp->data);
destroyPQExpBuffer(temp);
}
else if (strcmp(te->desc, "CAST") == 0 ||
strcmp(te->desc, "CHECK CONSTRAINT") == 0 ||
Cause ALTER INDEX OWNER to generate a warning and do nothing, rather than erroring out as it has done for the last couple weeks. Document that this form is now ignored because indexes can't usefully have different owners from their parent tables. Fix pg_dump to not generate ALTER OWNER commands for indexes.
2005-08-22 21:40:37 +02:00
strcmp(te->desc, "CONSTRAINT") == 0 ||
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strcmp(te->desc, "DEFAULT") == 0 ||
strcmp(te->desc, "FK CONSTRAINT") == 0 ||
Cause ALTER INDEX OWNER to generate a warning and do nothing, rather than erroring out as it has done for the last couple weeks. Document that this form is now ignored because indexes can't usefully have different owners from their parent tables. Fix pg_dump to not generate ALTER OWNER commands for indexes.
2005-08-22 21:40:37 +02:00
strcmp(te->desc, "INDEX") == 0 ||
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
strcmp(te->desc, "RULE") == 0 ||
SQL/MED catalog manipulation facilities This doesn't do any remote or external things yet, but it gives modules like plproxy and dblink a standardized and future-proof system for managing their connection information. Martin Pihlak and Peter Eisentraut
2008-12-19 17:25:19 +01:00
strcmp(te->desc, "TRIGGER") == 0 ||
strcmp(te->desc, "USER MAPPING") == 0)
Clean up pg_dump's handling of ownership for indexes (including pkey/unique constraint indexes) and schemas. Per report from Michael Fuhr.
2005-01-11 06:14:13 +01:00
{
/* these object types don't have separate owners */
}
else
{
write_msg(modulename, "WARNING: don't know how to set owner for object type %s\n",
te->desc);
}
Issue CREATE SCHEMA as the originally connecting user, with an AUTHORIZATION clause to specify the desired owner. This allows a superuser to restore schemas owned by users without CREATE-SCHEMA permissions (ie, schemas originally created by a superuser using AUTHORIZATION). --no-owner can be specified to suppress the AUTHORIZATION clause if need be.
2003-09-24 01:31:52 +02:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Fix incorrect tracking of session authorization in the presence of ACLs that contain rights granted by non-owners. Per bug report from Nishad Prakash.
2004-07-19 23:02:17 +02:00
/*
* If it's an ACL entry, it might contain SET SESSION AUTHORIZATION
Standard pgindent run for 8.1.
2005-10-15 04:49:52 +02:00
* commands, so we can no longer assume we know the current auth setting.
Fix incorrect tracking of session authorization in the presence of ACLs that contain rights granted by non-owners. Per bug report from Nishad Prakash.
2004-07-19 23:02:17 +02:00
*/
Fix up pg_dump's treatment of large object ownership and ACLs. We now emit a separate archive entry for each BLOB, and use pg_dump's standard methods for dealing with its ownership, ACL if any, and comment if any. This means that switches like --no-owner and --no-privileges do what they're supposed to. Preliminary testing says that performance is still reasonable even with many blobs, though we'll have to see how that shakes out in the field. KaiGai Kohei, revised by me
2010-02-18 02:29:10 +01:00
if (acl_pass)
Fix incorrect tracking of session authorization in the presence of ACLs that contain rights granted by non-owners. Per bug report from Nishad Prakash.
2004-07-19 23:02:17 +02:00
{
if (AH->currUser)
free(AH->currUser);
AH->currUser = NULL;
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Convert newlines to spaces in names written in pg_dump comments. pg_dump was incautious about sanitizing object names that are emitted within SQL comments in its output script. A name containing a newline would at least render the script syntactically incorrect. Maliciously crafted object names could present a SQL injection risk when the script is reloaded. Reported by Heikki Linnakangas, patch by Robert Haas Security: CVE-2012-0868
2012-02-23 21:53:09 +01:00
/*
* Sanitize a string to be included in an SQL comment, by replacing any
* newlines with spaces.
*/
static char *
replace_line_endings(const char *str)
{
char *result;
char *s;
result = pg_strdup(str);
for (s = result; *s != '\0'; s++)
{
if (*s == '\n' || *s == '\r')
*s = ' ';
}
return result;
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
WriteHead(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
struct tm crtm;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
(*AH->WriteBufPtr) (AH, "PGDMP", 5); /* Magic code */
(*AH->WriteBytePtr) (AH, AH->vmaj);
(*AH->WriteBytePtr) (AH, AH->vmin);
(*AH->WriteBytePtr) (AH, AH->vrev);
(*AH->WriteBytePtr) (AH, AH->intSize);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
(*AH->WriteBytePtr) (AH, AH->offSize);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
(*AH->WriteBytePtr) (AH, AH->format);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Please find attached a patch for the pg_dump directory which addresses: - The problems Jan reported - incompatibility with configure (now uses HAVE_LIBZ instead of HAVE_ZLIB) - a problem in auto-detecting archive file format on piped archives Philip Warner
2000-07-06 20:39:39 +02:00
#ifndef HAVE_LIBZ
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->compression != 0)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
write_msg(modulename, "WARNING: requested compression not available in this "
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
"installation -- archive will be uncompressed\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->compression = 0;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
#endif
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
WriteInt(AH, AH->compression);
crtm = *localtime(&AH->createDate);
WriteInt(AH, crtm.tm_sec);
WriteInt(AH, crtm.tm_min);
WriteInt(AH, crtm.tm_hour);
WriteInt(AH, crtm.tm_mday);
WriteInt(AH, crtm.tm_mon);
WriteInt(AH, crtm.tm_year);
WriteInt(AH, crtm.tm_isdst);
Make sure all clients have the same understanding of default user name and database. In particular, make script wrappers understand the PGDATABASE environment variable.
2002-08-10 18:57:32 +02:00
WriteStr(AH, PQdb(AH->connection));
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
WriteStr(AH, AH->public.remoteVersionStr);
WriteStr(AH, PG_VERSION);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
void
ReadHead(ArchiveHandle *AH)
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
{
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
char tmpMag[7];
int fmt;
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
struct tm crtm;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
/*
* If we haven't already read the header, do so.
*
pgindent run for 9.0, second run
2010-07-06 21:19:02 +02:00
* NB: this code must agree with _discoverArchiveFormat(). Maybe find a
* way to unify the cases?
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
*/
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (!AH->readHeader)
{
Fix pg_restore to guard against unexpected EOF while reading an archive file. Per report and partial patch from Chad Wagner.
2007-08-06 03:38:15 +02:00
if ((*AH->ReadBufPtr) (AH, tmpMag, 5) != 5)
die_horribly(AH, modulename, "unexpected end of file\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (strncmp(tmpMag, "PGDMP", 5) != 0)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "did not find magic string in file header\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->vmaj = (*AH->ReadBytePtr) (AH);
AH->vmin = (*AH->ReadBytePtr) (AH);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->vmaj > 1 || ((AH->vmaj == 1) && (AH->vmin > 0))) /* Version > 1.0 */
AH->vrev = (*AH->ReadBytePtr) (AH);
else
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->vrev = 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->version = ((AH->vmaj * 256 + AH->vmin) * 256 + AH->vrev) * 256 + 0;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->version < K_VERS_1_0 || AH->version > K_VERS_MAX)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "unsupported version (%d.%d) in file header\n",
AH->vmaj, AH->vmin);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->intSize = (*AH->ReadBytePtr) (AH);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->intSize > 32)
Enable large file support. Use off_t and size_t in pg_dump to handle file offset arithmetic correctly.
2002-08-20 19:54:45 +02:00
die_horribly(AH, modulename, "sanity check on integer size (%lu) failed\n",
(unsigned long) AH->intSize);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->intSize > sizeof(int))
Wording cleanup for error messages. Also change can't -> cannot. Standard English uses "may", "can", and "might" in different ways: may - permission, "You may borrow my rake." can - ability, "I can lift that log." might - possibility, "It might rain today." Unfortunately, in conversational English, their use is often mixed, as in, "You may use this variable to do X", when in fact, "can" is a better choice. Similarly, "It may crash" is better stated, "It might crash".
2007-02-01 20:10:30 +01:00
write_msg(modulename, "WARNING: archive was made on a machine with larger integers, some operations might fail\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
if (AH->version >= K_VERS_1_7)
pgindent run.
2003-08-04 02:43:34 +02:00
AH->offSize = (*AH->ReadBytePtr) (AH);
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
else
pgindent run.
2003-08-04 02:43:34 +02:00
AH->offSize = AH->intSize;
Allow 8-byte off_t to properly pg_dump, from Philip Warner with mods by Bruce.
2002-10-22 21:15:23 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
fmt = (*AH->ReadBytePtr) (AH);
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->format != fmt)
National language support for pg_dump and pg_restore. Combined with big message clean up.
2001-06-27 23:21:37 +02:00
die_horribly(AH, modulename, "expected format (%d) differs from format found in file (%d)\n",
AH->format, fmt);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->version >= K_VERS_1_2)
{
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->version < K_VERS_1_4)
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
AH->compression = (*AH->ReadBytePtr) (AH);
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
else
AH->compression = ReadInt(AH);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
}
else
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
AH->compression = Z_DEFAULT_COMPRESSION;
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
Please find attached a patch for the pg_dump directory which addresses: - The problems Jan reported - incompatibility with configure (now uses HAVE_LIBZ instead of HAVE_ZLIB) - a problem in auto-detecting archive file format on piped archives Philip Warner
2000-07-06 20:39:39 +02:00
#ifndef HAVE_LIBZ
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->compression != 0)
Apply message style guide to frontend programs.
2003-07-23 10:47:41 +02:00
write_msg(modulename, "WARNING: archive is compressed, but this installation does not support compression -- no data will be available\n");
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
#endif
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
if (AH->version >= K_VERS_1_4)
{
crtm.tm_sec = ReadInt(AH);
crtm.tm_min = ReadInt(AH);
crtm.tm_hour = ReadInt(AH);
crtm.tm_mday = ReadInt(AH);
crtm.tm_mon = ReadInt(AH);
crtm.tm_year = ReadInt(AH);
crtm.tm_isdst = ReadInt(AH);
AH->archdbname = ReadStr(AH);
AH->createDate = mktime(&crtm);
pgindent run. Make it all clean.
2001-03-22 05:01:46 +01:00
if (AH->createDate == (time_t) -1)
More message munging and localization for pg_dump, especially the --verbose messages, which had not been considered so far. Output to the terminal should okay now; comments written into the dump are still English only, which may or may not be the desirable thing.
2001-07-03 22:21:50 +02:00
write_msg(modulename, "WARNING: invalid creation date in header\n");
- Support for BLOB output from pg_dump and input via pg_restore - Support for direct DB connection in pg_restore - Fixes in support for --insert flag - pg_dump now outputs in modified OID order - various other bug fixes
2000-07-21 13:40:08 +02:00
}
Update pg_dump to use SET DEFAULT_TABLESPACE instead of explicit tablespace clauses; this should improve compatibility of dump files. Philip Warner, some rework by Tom Lane.
2004-11-06 20:36:02 +01:00
if (AH->version >= K_VERS_1_10)
{
AH->archiveRemoteVersion = ReadStr(AH);
AH->archiveDumpVersion = ReadStr(AH);
}
pg_dump patch from Philip Warner
2000-07-04 16:25:28 +02:00
}
Allow fseeko in pg_dump only if fseeko() will work for all supported file sizes.
2002-10-25 03:33:17 +02:00
/*
* checkSeek
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
* check to see if ftell/fseek can be performed.
Allow fseeko in pg_dump only if fseeko() will work for all supported file sizes.
2002-10-25 03:33:17 +02:00
*/
bool
checkSeek(FILE *fp)
{
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
pgoff_t tpos;
/*
pgindent run for 9.0, second run
2010-07-06 21:19:02 +02:00
* If pgoff_t is wider than long, we must have "real" fseeko and not an
* emulation using fseek. Otherwise report no seek capability.
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
*/
#ifndef HAVE_FSEEKO
if (sizeof(pgoff_t) > sizeof(long))
Allow fseeko in pg_dump only if fseeko() will work for all supported file sizes.
2002-10-25 03:33:17 +02:00
return false;
#endif
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
/* Check that ftello works on this file */
errno = 0;
tpos = ftello(fp);
if (errno)
return false;
/*
pgindent run for 9.0, second run
2010-07-06 21:19:02 +02:00
* Check that fseeko(SEEK_SET) works, too. NB: we used to try to test
Improve pg_dump's checkSeek() function to verify the functioning of ftello as well as fseeko, and to not assume that fseeko(fp, 0, SEEK_CUR) proves anything. Also improve some related comments. Per my observation that the SEEK_CUR test didn't actually work on some platforms, and subsequent discussion with Robert Haas. Back-patch to 8.4. In earlier releases it's not that important whether we get the hasSeek test right, but with parallel restore it matters.
2010-06-28 04:07:02 +02:00
* this with fseeko(fp, 0, SEEK_CUR). But some platforms treat that as a
* successful no-op even on files that are otherwise unseekable.
*/
if (fseeko(fp, tpos, SEEK_SET) != 0)
return false;
return true;
Allow fseeko in pg_dump only if fseeko() will work for all supported file sizes.
2002-10-25 03:33:17 +02:00
}
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
/*
* dumpTimestamp
*/
static void
dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim)
{
char buf[256];
Suppress timezone (%Z) part of timestamp display when running on Windows, because on that platform strftime produces localized zone names in varying encodings. Even though it's only in a comment, this can cause encoding errors when reloading the dump script. Per suggestion from Andreas Seltenreich. Also, suppress %Z on Windows in the %s escape of log_line_prefix ... not sure why this one is different from the other two, but it shouldn't be.
2006-11-21 23:19:46 +01:00
/*
* We don't print the timezone on Win32, because the names are long and
* localized, which means they may contain characters in various random
pgindent run for 8.3.
2007-11-15 22:14:46 +01:00
* encodings; this has been seen to cause encoding errors when reading the
* dump script.
Suppress timezone (%Z) part of timestamp display when running on Windows, because on that platform strftime produces localized zone names in varying encodings. Even though it's only in a comment, this can cause encoding errors when reloading the dump script. Per suggestion from Andreas Seltenreich. Also, suppress %Z on Windows in the %s escape of log_line_prefix ... not sure why this one is different from the other two, but it shouldn't be.
2006-11-21 23:19:46 +01:00
*/
if (strftime(buf, sizeof(buf),
#ifndef WIN32
"%Y-%m-%d %H:%M:%S %Z",
#else
"%Y-%m-%d %H:%M:%S",
#endif
localtime(&tim)) != 0)
Revert addition of poorly-thought-out DUMP TIMESTAMP archive entry, which induced bug #1597 in addition to having several other misbehaviors (like labeling the dump with a completion time having nothing to do with reality). Instead just print out the desired strings where RestoreArchive was already emitting the 'PostgreSQL database dump' and 'PostgreSQL database dump complete' strings.
2005-04-15 18:40:36 +02:00
ahprintf(AH, "-- %s %s\n\n", msg, buf);
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Main engine for parallel restore.
*
* Work is done in three phases.
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
* First we process all SECTION_PRE_DATA tocEntries, in a single connection,
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* just as for a standard restore. Second we process the remaining non-ACL
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
* steps in parallel worker children (threads on Windows, processes on Unix),
* each of which connects separately to the database. Finally we process all
* the ACL entries in a single connection (that happens back in
* RestoreArchive).
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
static void
restore_toc_entries_parallel(ArchiveHandle *AH)
{
RestoreOptions *ropt = AH->ropt;
Rename pg_restore -m to -j, and add documentation about what good numbers are. per discussion on hackers
2009-03-20 10:21:09 +01:00
int n_slots = ropt->number_of_jobs;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ParallelSlot *slots;
int work_status;
int next_slot;
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
bool skipped_some;
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
TocEntry pending_list;
TocEntry ready_list;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
TocEntry *next_work_item;
thandle ret_child;
TocEntry *te;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ahlog(AH, 2, "entering restore_toc_entries_parallel\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
slots = (ParallelSlot *) pg_calloc(sizeof(ParallelSlot), n_slots);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* Adjust dependency information */
fix_dependencies(AH);
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Do all the early stuff in a single connection in the parent. There's no
* great point in running it in parallel, in fact it will actually run
* faster in a single connection because we avoid all the connection and
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* setup overhead. Also, pg_dump is not currently very good about showing
* all the dependencies of SECTION_PRE_DATA items, so we do not risk
* trying to process them out-of-order.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
skipped_some = false;
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
for (next_work_item = AH->toc->next; next_work_item != AH->toc; next_work_item = next_work_item->next)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
/* NB: process-or-continue logic must be the inverse of loop below */
if (next_work_item->section != SECTION_PRE_DATA)
{
/* DATA and POST_DATA items are just ignored for now */
if (next_work_item->section == SECTION_DATA ||
next_work_item->section == SECTION_POST_DATA)
{
skipped_some = true;
continue;
}
else
{
/*
* SECTION_NONE items, such as comments, can be processed now
* if we are still in the PRE_DATA part of the archive. Once
* we've skipped any items, we have to consider whether the
* comment's dependencies are satisfied, so skip it for now.
*/
if (skipped_some)
continue;
}
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ahlog(AH, 1, "processing item %d %s %s\n",
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
(void) restore_toc_entry(AH, next_work_item, ropt, false);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
/* there should be no touch of ready_list here, so pass NULL */
reduce_dependencies(AH, next_work_item, NULL);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Now close parent connection in prep for parallel steps. We do this
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
* mainly to ensure that we don't exceed the specified number of parallel
* connections.
*/
Invent on_exit_nicely for pg_dump. Per recent discussions on pgsql-hackers regarding parallel pg_dump.
2012-02-16 17:49:20 +01:00
DisconnectDatabase(&AH->public);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* blow away any transient state from the old connection */
if (AH->currUser)
free(AH->currUser);
AH->currUser = NULL;
if (AH->currSchema)
free(AH->currSchema);
AH->currSchema = NULL;
if (AH->currTablespace)
free(AH->currTablespace);
AH->currTablespace = NULL;
AH->currWithOids = -1;
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
/*
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* Initialize the lists of pending and ready items. After this setup, the
* pending list is everything that needs to be done but is blocked by one
* or more dependencies, while the ready list contains items that have no
* remaining dependencies. Note: we don't yet filter out entries that
* aren't going to be restored. They might participate in dependency
* chains connecting entries that should be restored, so we treat them as
* live until we actually process them.
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
*/
par_list_header_init(&pending_list);
par_list_header_init(&ready_list);
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
skipped_some = false;
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
for (next_work_item = AH->toc->next; next_work_item != AH->toc; next_work_item = next_work_item->next)
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
{
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
/* NB: process-or-continue logic must be the inverse of loop above */
if (next_work_item->section == SECTION_PRE_DATA)
{
/* All PRE_DATA items were dealt with above */
continue;
}
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
if (next_work_item->section == SECTION_DATA ||
next_work_item->section == SECTION_POST_DATA)
{
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
/* set this flag at same point that previous loop did */
skipped_some = true;
Improve parallel restore's ability to cope with selective restore (-L option). The original coding tended to break down in the face of modified restore orders, as shown in bug #5626 from Albert Ullrich, because it would flip over into parallel-restore operation too soon. That causes problems because we don't have sufficient dependency information in dump archives to allow safe parallel processing of SECTION_PRE_DATA items. Even if we did, it's probably undesirable to allow that to override the commanded restore order. To fix the problem of omitted items causing unexpected changes in restore order, tweak SortTocFromFile so that omitted items end up at the head of the list not the tail. This ensures that they'll be examined and their dependencies will be marked satisfied before we get to any interesting items. In HEAD and 9.0, we can easily change restore_toc_entries_parallel so that all SECTION_PRE_DATA items are guaranteed to be processed in the initial serial-restore loop, and hence in commanded order. Only DATA and POST_DATA items are candidates for parallel processing. For them there might be variations from the commanded order because of parallelism, but we should do it in a safe order thanks to dependencies. In 8.4 it's much harder to make such a guarantee. I settled for not letting the initial loop break out into parallel processing mode if it sees a DATA/POST_DATA item that's not to be restored; this at least prevents a non-restorable item from causing premature exit from the loop. This means that 8.4 will be more likely to fail given a badly-ordered -L list than 9.x, but we don't really promise any such thing will work anyway.
2010-08-21 15:59:44 +02:00
}
Fix parallel pg_restore to handle comments on POST_DATA items correctly. The previous coding would try to process all SECTION_NONE items in the initial sequential-restore pass, which failed if they were dependencies of not-yet-restored items. Fix by postponing such items into the parallel processing pass once we have skipped any non-PRE_DATA item. Back-patch into 9.0; the original parallel-restore coding in 8.4 did not have this bug, so no need to change it. Report and diagnosis by Arnd Hannemann.
2011-02-18 19:11:45 +01:00
else
{
/* SECTION_NONE items must be processed if previous loop didn't */
if (!skipped_some)
continue;
}
if (next_work_item->depCount > 0)
par_list_append(&pending_list, next_work_item);
else
par_list_append(&ready_list, next_work_item);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* main parent loop
*
* Keep going until there is no worker still running AND there is no work
* left to be done.
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ahlog(AH, 1, "entering main parallel loop\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
while ((next_work_item = get_next_work_item(AH, &ready_list,
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
slots, n_slots)) != NULL ||
work_in_progress(slots, n_slots))
{
if (next_work_item != NULL)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
teReqs reqs;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* If not to be dumped, don't waste time launching a worker */
reqs = _tocEntryRequired(next_work_item, AH->ropt, false);
if ((reqs & (REQ_SCHEMA | REQ_DATA)) == 0)
{
ahlog(AH, 1, "skipping item %d %s %s\n",
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
par_list_remove(next_work_item);
reduce_dependencies(AH, next_work_item, &ready_list);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
continue;
}
if ((next_slot = get_next_slot(slots, n_slots)) != NO_SLOT)
{
/* There is work still to do and a worker slot available */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
thandle child;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
RestoreArgs *args;
ahlog(AH, 1, "launching item %d %s %s\n",
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
par_list_remove(next_work_item);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* this memory is dealloced in mark_work_done() */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
args = pg_malloc(sizeof(RestoreArgs));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
args->AH = CloneArchive(AH);
args->te = next_work_item;
/* run the step in a worker child */
child = spawn_restore(args);
slots[next_slot].child_id = child;
slots[next_slot].args = args;
continue;
}
}
/*
* If we get here there must be work being done. Either there is no
* work available to schedule (and work_in_progress returned true) or
* there are no slots available. So we wait for a worker to finish,
* and process the result.
*/
ret_child = reap_child(slots, n_slots, &work_status);
if (WIFEXITED(work_status))
{
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
mark_work_done(AH, &ready_list,
ret_child, WEXITSTATUS(work_status),
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
slots, n_slots);
}
else
{
die_horribly(AH, modulename, "worker process crashed: status %d\n",
work_status);
}
}
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ahlog(AH, 1, "finished main parallel loop\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Now reconnect the single parent connection.
*/
ConnectDatabase((Archive *) AH, ropt->dbname,
ropt->pghost, ropt->pgport, ropt->username,
Add a -w/--no-password option that prevents all password prompts to all programs that have a -W/--password option. In passing, remove the ancient PSQL_ALWAYS_GET_PASSWORDS compile option.
2009-02-26 17:02:39 +01:00
ropt->promptPassword);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
_doSetFixedOutputState(AH);
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Make sure there is no non-ACL work left due to, say, circular
* dependencies, or some other pathological condition. If so, do it in the
* single parent connection.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
for (te = pending_list.par_next; te != &pending_list; te = te->par_next)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
ahlog(AH, 1, "processing missed item %d %s %s\n",
te->dumpId, te->desc, te->tag);
(void) restore_toc_entry(AH, te, ropt, false);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
/* The ACLs will be handled back in RestoreArchive. */
}
/*
* create a worker child to perform a restore step in parallel
*/
static thandle
spawn_restore(RestoreArgs *args)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
thandle child;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* Ensure stdio state is quiesced before forking */
fflush(NULL);
#ifndef WIN32
child = fork();
if (child == 0)
{
/* in child process */
parallel_restore(args);
die_horribly(args->AH, modulename,
"parallel_restore should not return\n");
}
else if (child < 0)
{
/* fork failed */
die_horribly(args->AH, modulename,
"could not create worker process: %s\n",
strerror(errno));
}
#else
child = (HANDLE) _beginthreadex(NULL, 0, (void *) parallel_restore,
args, 0, NULL);
if (child == 0)
die_horribly(args->AH, modulename,
"could not create worker thread: %s\n",
strerror(errno));
#endif
return child;
}
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* collect status from a completed worker child
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
static thandle
reap_child(ParallelSlot *slots, int n_slots, int *work_status)
{
#ifndef WIN32
/* Unix is so much easier ... */
return wait(work_status);
#else
static HANDLE *handles = NULL;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int hindex,
snum,
tnum;
thandle ret_child;
DWORD res;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* first time around only, make space for handles to listen on */
if (handles == NULL)
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
handles = (HANDLE *) pg_calloc(sizeof(HANDLE), n_slots);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* set up list of handles to listen to */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
for (snum = 0, tnum = 0; snum < n_slots; snum++)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (slots[snum].child_id != 0)
handles[tnum++] = slots[snum].child_id;
/* wait for one to finish */
hindex = WaitForMultipleObjects(tnum, handles, false, INFINITE);
/* get handle of finished thread */
ret_child = handles[hindex - WAIT_OBJECT_0];
/* get the result */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
GetExitCodeThread(ret_child, &res);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*work_status = res;
/* dispose of handle to stop leaks */
CloseHandle(ret_child);
return ret_child;
#endif
}
/*
* are we doing anything now?
*/
static bool
work_in_progress(ParallelSlot *slots, int n_slots)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (i = 0; i < n_slots; i++)
{
if (slots[i].child_id != 0)
return true;
}
return false;
}
/*
* find the first free parallel slot (if any).
*/
static int
get_next_slot(ParallelSlot *slots, int n_slots)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (i = 0; i < n_slots; i++)
{
if (slots[i].child_id == 0)
return i;
}
return NO_SLOT;
}
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
/*
* Check if te1 has an exclusive lock requirement for an item that te2 also
* requires, whether or not te2's requirement is for an exclusive lock.
*/
static bool
has_lock_conflicts(TocEntry *te1, TocEntry *te2)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int j,
k;
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
for (j = 0; j < te1->nLockDeps; j++)
{
for (k = 0; k < te2->nDeps; k++)
{
if (te1->lockDeps[j] == te2->dependencies[k])
return true;
}
}
return false;
}
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
/*
* Initialize the header of a parallel-processing list.
*
* These are circular lists with a dummy TocEntry as header, just like the
* main TOC list; but we use separate list links so that an entry can be in
* the main TOC list as well as in a parallel-processing list.
*/
static void
par_list_header_init(TocEntry *l)
{
l->par_prev = l->par_next = l;
}
/* Append te to the end of the parallel-processing list headed by l */
static void
par_list_append(TocEntry *l, TocEntry *te)
{
te->par_prev = l->par_prev;
l->par_prev->par_next = te;
l->par_prev = te;
te->par_next = l;
}
/* Remove te from whatever parallel-processing list it's in */
static void
par_list_remove(TocEntry *te)
{
te->par_prev->par_next = te->par_next;
te->par_next->par_prev = te->par_prev;
te->par_prev = NULL;
te->par_next = NULL;
}
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Find the next work item (if any) that is capable of being run now.
*
* To qualify, the item must have no remaining dependencies
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
* and no requirements for locks that are incompatible with
* items currently running. Items in the ready_list are known to have
* no remaining dependencies, but we have to check for lock conflicts.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
* Note that the returned item has *not* been removed from ready_list.
* The caller must do that after successfully dispatching the item.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
* pref_non_data is for an alternative selection algorithm that gives
* preference to non-data items if there is already a data load running.
* It is currently disabled.
*/
static TocEntry *
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
get_next_work_item(ArchiveHandle *AH, TocEntry *ready_list,
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ParallelSlot *slots, int n_slots)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
bool pref_non_data = false; /* or get from AH->ropt */
TocEntry *data_te = NULL;
TocEntry *te;
int i,
k;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Bogus heuristics for pref_non_data
*/
if (pref_non_data)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int count = 0;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
for (k = 0; k < n_slots; k++)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (slots[k].args->te != NULL &&
slots[k].args->te->section == SECTION_DATA)
count++;
if (n_slots == 0 || count * 4 < n_slots)
pref_non_data = false;
}
/*
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
* Search the ready_list until we find a suitable item.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
for (te = ready_list->par_next; te != ready_list; te = te->par_next)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
bool conflicts = false;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Check to see if the item would need exclusive lock on something
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* that a currently running item also needs lock on, or vice versa. If
* so, we don't want to schedule them together.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
for (i = 0; i < n_slots && !conflicts; i++)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
TocEntry *running_te;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (slots[i].args == NULL)
continue;
running_te = slots[i].args->te;
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
if (has_lock_conflicts(te, running_te) ||
has_lock_conflicts(running_te, te))
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
Fix logic to detect conflicts or blocks involving exclusive locks in parallel restore items. If a currently running item needs an exclusive lock on any item that the candidate items needs any sort of lock on, or vice versa, then the candidate item is not allowed to run now, and must wait till later.
2009-04-12 23:02:44 +02:00
conflicts = true;
break;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
}
if (conflicts)
continue;
if (pref_non_data && te->section == SECTION_DATA)
{
if (data_te == NULL)
data_te = te;
continue;
}
/* passed all tests, so this item can run */
return te;
}
if (data_te != NULL)
return data_te;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
ahlog(AH, 2, "no item ready\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
return NULL;
}
/*
* Restore a single TOC item in parallel with others
*
* this is the procedure run as a thread (Windows) or a
* separate process (everything else).
*/
static parallel_restore_result
parallel_restore(RestoreArgs *args)
{
ArchiveHandle *AH = args->AH;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
TocEntry *te = args->te;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
RestoreOptions *ropt = AH->ropt;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int retval;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Close and reopen the input file so we have a private file pointer that
* doesn't stomp on anyone else's file pointer, if we're actually going to
* need to read from the file. Otherwise, just close it except on Windows,
* where it will possibly be needed by other threads.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Note: on Windows, since we are using threads not processes, the reopen
* call *doesn't* close the original file pointer but just open a new one.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
if (te->section == SECTION_DATA)
Use thread-local storage for querybuffer in fmtId() on Windows, when needed (i.e. when running pg_restore, which might run in parallel). Only reopen archive file when we really need to read from it, in parallel code. Otherwise, close it immediately in a worker, if possible.
2009-03-11 04:33:29 +01:00
(AH->ReopenPtr) (AH);
#ifndef WIN32
else
(AH->ClosePtr) (AH);
#endif
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* We need our own database connection, too
*/
ConnectDatabase((Archive *) AH, ropt->dbname,
ropt->pghost, ropt->pgport, ropt->username,
Add a -w/--no-password option that prevents all password prompts to all programs that have a -W/--password option. In passing, remove the ancient PSQL_ALWAYS_GET_PASSWORDS compile option.
2009-02-26 17:02:39 +01:00
ropt->promptPassword);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
_doSetFixedOutputState(AH);
/* Restore the TOC item */
retval = restore_toc_entry(AH, te, ropt, true);
/* And clean up */
Invent on_exit_nicely for pg_dump. Per recent discussions on pgsql-hackers regarding parallel pg_dump.
2012-02-16 17:49:20 +01:00
DisconnectDatabase((Archive *) AH);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Use thread-local storage for querybuffer in fmtId() on Windows, when needed (i.e. when running pg_restore, which might run in parallel). Only reopen archive file when we really need to read from it, in parallel code. Otherwise, close it immediately in a worker, if possible.
2009-03-11 04:33:29 +01:00
/* If we reopened the file, we are done with it, so close it now */
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
if (te->section == SECTION_DATA)
Use thread-local storage for querybuffer in fmtId() on Windows, when needed (i.e. when running pg_restore, which might run in parallel). Only reopen archive file when we really need to read from it, in parallel code. Otherwise, close it immediately in a worker, if possible.
2009-03-11 04:33:29 +01:00
(AH->ClosePtr) (AH);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
if (retval == 0 && AH->public.n_errors)
retval = WORKER_IGNORED_ERRORS;
#ifndef WIN32
exit(retval);
#else
return retval;
#endif
}
/*
* Housekeeping to be done after a step has been parallel restored.
*
* Clear the appropriate slot, free all the extra memory we allocated,
* update status, and reduce the dependency count of any dependent items.
*/
static void
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
mark_work_done(ArchiveHandle *AH, TocEntry *ready_list,
thandle worker, int status,
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ParallelSlot *slots, int n_slots)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
TocEntry *te = NULL;
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (i = 0; i < n_slots; i++)
{
if (slots[i].child_id == worker)
{
slots[i].child_id = 0;
te = slots[i].args->te;
DeCloneArchive(slots[i].args->AH);
free(slots[i].args);
slots[i].args = NULL;
break;
}
}
if (te == NULL)
Message tuning
2009-03-05 15:51:10 +01:00
die_horribly(AH, modulename, "could not find slot of finished worker\n");
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
ahlog(AH, 1, "finished item %d %s %s\n",
te->dumpId, te->desc, te->tag);
if (status == WORKER_CREATE_DONE)
mark_create_done(AH, te);
else if (status == WORKER_INHIBIT_DATA)
{
inhibit_data_for_failed_table(AH, te);
AH->public.n_errors++;
}
else if (status == WORKER_IGNORED_ERRORS)
AH->public.n_errors++;
else if (status != 0)
die_horribly(AH, modulename, "worker process failed: exit code %d\n",
status);
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
reduce_dependencies(AH, te, ready_list);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
/*
* Process the dependency information into a form useful for parallel restore.
*
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
* This function takes care of fixing up some missing or badly designed
* dependencies, and then prepares subsidiary data structures that will be
* used in the main parallel-restore logic, including:
* 1. We build the tocsByDumpId[] index array.
* 2. We build the revDeps[] arrays of incoming dependency dumpIds.
* 3. We set up depCount fields that are the number of as-yet-unprocessed
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
* dependencies for each TOC entry.
*
* We also identify locking dependencies so that we can avoid trying to
* schedule conflicting items at the same time.
*/
static void
fix_dependencies(ArchiveHandle *AH)
{
TocEntry *te;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
* It is convenient to have an array that indexes the TOC entries by dump
* ID, rather than searching the TOC list repeatedly. Entries for dump
* IDs not present in the TOC will be NULL.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
* NOTE: because maxDumpId is just the highest dump ID defined in the
pgindent run for 9.0
2010-02-26 03:01:40 +01:00
* archive, there might be dependencies for IDs > maxDumpId. All uses of
* this array must guard against out-of-range dependency numbers.
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
*
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
* Also, initialize the depCount/revDeps/nRevDeps fields, and make sure
* the TOC items are marked as not being in any parallel-processing list.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
maxDumpId = AH->maxDumpId;
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
tocsByDumpId = (TocEntry **) pg_calloc(maxDumpId, sizeof(TocEntry *));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
tocsByDumpId[te->dumpId - 1] = te;
te->depCount = te->nDeps;
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
te->revDeps = NULL;
te->nRevDeps = 0;
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
te->par_prev = NULL;
te->par_next = NULL;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
/*
* POST_DATA items that are shown as depending on a table need to be
* re-pointed to depend on that table's data, instead. This ensures they
* won't get scheduled until the data has been loaded. We handle this by
* first finding TABLE/TABLE DATA pairs and then scanning all the
* dependencies.
*
* Note: currently, a TABLE DATA should always have exactly one
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* dependency, on its TABLE item. So we don't bother to search, but look
* just at the first dependency. We do trouble to make sure that it's a
* TABLE, if possible. However, if the dependency isn't in the archive
* then just assume it was a TABLE; this is to cover cases where the table
* was suppressed but we have the data and some dependent post-data items.
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
*
* XXX this is O(N^2) if there are a lot of tables. We ought to fix
* pg_dump to produce correctly-linked dependencies in the first place.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
if (strcmp(te->desc, "TABLE DATA") == 0 && te->nDeps > 0)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
DumpId tableId = te->dependencies[0];
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
if (tableId > maxDumpId ||
tocsByDumpId[tableId - 1] == NULL ||
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
strcmp(tocsByDumpId[tableId - 1]->desc, "TABLE") == 0)
{
repoint_table_dependencies(AH, tableId, te->dumpId);
}
}
}
/*
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* Pre-8.4 versions of pg_dump neglected to set up a dependency from BLOB
* COMMENTS to BLOBS. Cope. (We assume there's only one BLOBS and only
* one BLOB COMMENTS in such files.)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
if (AH->version < K_VERS_1_11)
{
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
if (strcmp(te->desc, "BLOB COMMENTS") == 0 && te->nDeps == 0)
{
TocEntry *te2;
for (te2 = AH->toc->next; te2 != AH->toc; te2 = te2->next)
{
if (strcmp(te2->desc, "BLOBS") == 0)
{
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
te->dependencies = (DumpId *) pg_malloc(sizeof(DumpId));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
te->dependencies[0] = te2->dumpId;
te->nDeps++;
te->depCount++;
break;
}
}
break;
}
}
}
/*
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
* At this point we start to build the revDeps reverse-dependency arrays,
* so all changes of dependencies must be complete.
*/
/*
* Count the incoming dependencies for each item. Also, it is possible
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* that the dependencies list items that are not in the archive at all.
* Subtract such items from the depCounts.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
for (i = 0; i < te->nDeps; i++)
{
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
DumpId depid = te->dependencies[i];
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
if (depid <= maxDumpId && tocsByDumpId[depid - 1] != NULL)
tocsByDumpId[depid - 1]->nRevDeps++;
else
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
te->depCount--;
}
}
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
/*
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* Allocate space for revDeps[] arrays, and reset nRevDeps so we can use
* it as a counter below.
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
*/
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
if (te->nRevDeps > 0)
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
te->revDeps = (DumpId *) pg_malloc(te->nRevDeps * sizeof(DumpId));
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
te->nRevDeps = 0;
}
/*
pgindent run before PG 9.1 beta 1.
2011-04-10 17:42:00 +02:00
* Build the revDeps[] arrays of incoming-dependency dumpIds. This had
* better agree with the loops above.
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
*/
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
for (i = 0; i < te->nDeps; i++)
{
DumpId depid = te->dependencies[i];
if (depid <= maxDumpId && tocsByDumpId[depid - 1] != NULL)
{
TocEntry *otherte = tocsByDumpId[depid - 1];
otherte->revDeps[otherte->nRevDeps++] = te->dumpId;
}
}
}
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/*
* Lastly, work out the locking dependencies.
*/
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
te->lockDeps = NULL;
te->nLockDeps = 0;
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
identify_locking_dependencies(te);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
}
/*
* Change dependencies on tableId to depend on tableDataId instead,
* but only in POST_DATA items.
*/
static void
repoint_table_dependencies(ArchiveHandle *AH,
DumpId tableId, DumpId tableDataId)
{
TocEntry *te;
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
if (te->section != SECTION_POST_DATA)
continue;
for (i = 0; i < te->nDeps; i++)
{
if (te->dependencies[i] == tableId)
{
te->dependencies[i] = tableDataId;
ahlog(AH, 2, "transferring dependency %d -> %d to %d\n",
te->dumpId, tableId, tableDataId);
}
}
}
}
/*
* Identify which objects we'll need exclusive lock on in order to restore
* the given TOC entry (*other* than the one identified by the TOC entry
* itself). Record their dump IDs in the entry's lockDeps[] array.
*/
static void
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
identify_locking_dependencies(TocEntry *te)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
DumpId *lockids;
int nlockids;
int i;
/* Quick exit if no dependencies at all */
if (te->nDeps == 0)
return;
/* Exit if this entry doesn't need exclusive lock on other objects */
if (!(strcmp(te->desc, "CONSTRAINT") == 0 ||
strcmp(te->desc, "CHECK CONSTRAINT") == 0 ||
strcmp(te->desc, "FK CONSTRAINT") == 0 ||
strcmp(te->desc, "RULE") == 0 ||
strcmp(te->desc, "TRIGGER") == 0))
return;
/*
Fix identify_locking_dependencies to reflect the fact that fix_dependencies previously repointed TABLE dependencies to TABLE DATA. Mea culpa.
2009-03-13 23:50:44 +01:00
* We assume the item requires exclusive lock on each TABLE DATA item
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
* listed among its dependencies. (This was originally a dependency on
* the TABLE, but fix_dependencies repointed it to the data item. Note
* that all the entry types we are interested in here are POST_DATA, so
* they will all have been changed this way.)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
lockids = (DumpId *) pg_malloc(te->nDeps * sizeof(DumpId));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
nlockids = 0;
for (i = 0; i < te->nDeps; i++)
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
DumpId depid = te->dependencies[i];
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
When doing a parallel restore, we must guard against out-of-range dependency dump IDs, because the array we're using is sized according to the highest dump ID actually defined in the archive file. In a partial dump there could be references to higher dump IDs that weren't dumped. Treat these the same as references to in-range IDs that weren't dumped. (The whole thing is a bit scary because the missing objects might have been part of dependency chains, which we won't know about. Not much we can do though --- throwing an error is probably overreaction.) Also, reject parallel restore with pre-1.8 archive version (made by pre-8.0 pg_dump). In these old versions the dependency entries are OIDs, not dump IDs, and we don't have enough information to interpret them. Per bug #5288 from Jon Erdman.
2010-01-19 19:39:19 +01:00
if (depid <= maxDumpId && tocsByDumpId[depid - 1] &&
Fix identify_locking_dependencies to reflect the fact that fix_dependencies previously repointed TABLE dependencies to TABLE DATA. Mea culpa.
2009-03-13 23:50:44 +01:00
strcmp(tocsByDumpId[depid - 1]->desc, "TABLE DATA") == 0)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
lockids[nlockids++] = depid;
}
if (nlockids == 0)
{
free(lockids);
return;
}
Clean up after recent pg_dump patches. Fix entirely broken handling of va_list printing routines, update some out-of-date comments, fix some bogus inclusion orders, fix NLS declarations, fix missed realloc calls.
2011-11-30 02:41:06 +01:00
te->lockDeps = pg_realloc(lockids, nlockids * sizeof(DumpId));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
te->nLockDeps = nlockids;
}
/*
* Remove the specified TOC entry from the depCounts of items that depend on
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
* it, thereby possibly making them ready-to-run. Any pending item that
* becomes ready should be moved to the ready list.
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
*/
static void
Modify parallel pg_restore to track pending and ready items by means of two new lists, rather than repeatedly rescanning the main TOC list. This avoids a potential O(N^2) slowdown, although you'd need a *lot* of tables to make that really significant; and it might simplify future improvements in the scheduling algorithm by making the set of ready items more easily inspectable. The original thought that it would in itself result in a more efficient job dispatch order doesn't seem to have been borne out in testing, but it seems worth doing anyway.
2009-08-08 00:48:34 +02:00
reduce_dependencies(ArchiveHandle *AH, TocEntry *te, TocEntry *ready_list)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
2009-06-11 16:49:15 +02:00
int i;
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
ahlog(AH, 2, "reducing dependencies for %d\n", te->dumpId);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
for (i = 0; i < te->nRevDeps; i++)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
TocEntry *otherte = tocsByDumpId[te->revDeps[i] - 1];
otherte->depCount--;
if (otherte->depCount == 0 && otherte->par_prev != NULL)
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
{
Eliminate O(N^2) behavior in parallel restore with many blobs. With hundreds of thousands of TOC entries, the repeated searches in reduce_dependencies() become the dominant cost. Get rid of that searching by constructing reverse-dependency lists, which we can do in O(N) time during the fix_dependencies() preprocessing. I chose to store the reverse dependencies as DumpId arrays for consistency with the forward-dependency representation, and keep the previously-transient tocsByDumpId[] array around to locate actual TOC entry structs quickly from dump IDs. While this fixes the slow case reported by Vlad Arkhipov, there is still a potential for O(N^2) behavior with sufficiently many tables: fix_dependencies itself, as well as mark_create_done and inhibit_data_for_failed_table, are doing repeated searches to deal with table-to-table-data dependencies. Possibly this work could be extended to deal with that, although the latter two functions are also used in non-parallel restore where we currently don't run fix_dependencies. Another TODO is that we fail to parallelize restore of multiple blobs at all. This appears to require changes in the archive format to fix. Back-patch to 9.0 where the problem was reported. 8.4 has potential issues as well; but since it doesn't create a separate TOC entry for each blob, it's at much less risk of having enough TOC entries to cause real problems.
2010-12-09 19:03:11 +01:00
/* It must be in the pending list, so remove it ... */
par_list_remove(otherte);
/* ... and add to ready_list */
par_list_append(ready_list, otherte);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
}
}
}
/*
* Set the created flag on the DATA member corresponding to the given
* TABLE member
*/
static void
mark_create_done(ArchiveHandle *AH, TocEntry *te)
{
TocEntry *tes;
for (tes = AH->toc->next; tes != AH->toc; tes = tes->next)
{
if (strcmp(tes->desc, "TABLE DATA") == 0 &&
strcmp(tes->tag, te->tag) == 0 &&
strcmp(tes->namespace ? tes->namespace : "",
te->namespace ? te->namespace : "") == 0)
{
tes->created = true;
break;
}
}
}
/*
* Mark the DATA member corresponding to the given TABLE member
* as not wanted
*/
static void
inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te)
{
RestoreOptions *ropt = AH->ropt;
TocEntry *tes;
ahlog(AH, 1, "table \"%s\" could not be created, will not restore its data\n",
te->tag);
for (tes = AH->toc->next; tes != AH->toc; tes = tes->next)
{
if (strcmp(tes->desc, "TABLE DATA") == 0 &&
strcmp(tes->tag, te->tag) == 0 &&
strcmp(tes->namespace ? tes->namespace : "",
te->namespace ? te->namespace : "") == 0)
{
/* mark it unwanted; we assume idWanted array already exists */
ropt->idWanted[tes->dumpId - 1] = false;
break;
}
}
}
/*
* Clone and de-clone routines used in parallel restoration.
*
* Enough of the structure is cloned to ensure that there is no
* conflict between different threads each with their own clone.
*
* These could be public, but no need at present.
*/
static ArchiveHandle *
CloneArchive(ArchiveHandle *AH)
{
ArchiveHandle *clone;
/* Make a "flat" copy */
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
clone = (ArchiveHandle *) pg_malloc(sizeof(ArchiveHandle));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
memcpy(clone, AH, sizeof(ArchiveHandle));
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
/* Handle format-independent fields */
memset(&(clone->sqlparse), 0, sizeof(clone->sqlparse));
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* The clone will have its own connection, so disregard connection state */
clone->connection = NULL;
clone->currUser = NULL;
clone->currSchema = NULL;
clone->currTablespace = NULL;
clone->currWithOids = -1;
/* savedPassword must be local in case we change it while connecting */
if (clone->savedPassword)
Modify pg_dump to use error-free memory allocation macros. This avoids ignoring errors and call-site error checking.
2011-11-25 21:40:51 +01:00
clone->savedPassword = pg_strdup(clone->savedPassword);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* clone has its own error count, too */
clone->public.n_errors = 0;
/* Let the format-specific code have a chance too */
(clone->ClonePtr) (clone);
return clone;
}
/*
* Release clone-local storage.
*
* Note: we assume any clone-local connection was already closed.
*/
static void
DeCloneArchive(ArchiveHandle *AH)
{
/* Clear format-specific state */
(AH->DeClonePtr) (AH);
Fix pg_restore's direct-to-database mode for INSERT-style table data. In commit 6545a901aaf84cb05212bb6a7674059908f527c3, I removed the mini SQL lexer that was in pg_backup_db.c, thinking that it had no real purpose beyond separating COPY data from SQL commands, which purpose had been obsoleted by long-ago fixes in pg_dump's archive file format. Unfortunately this was in error: that code was also used to identify command boundaries in INSERT-style table data, which is run together as a single string in the archive file for better compressibility. As a result, direct-to-database restores from archive files made with --inserts or --column-inserts fail in our latest releases, as reported by Dick Visser. To fix, restore the mini SQL lexer, but simplify it by adjusting the calling logic so that it's only required to cope with INSERT-style table data, not arbitrary SQL commands. This allows us to not have to deal with SQL comments, E'' strings, or dollar-quoted strings, none of which have ever been emitted by dumpTableData_insert. Also, fix the lexer to cope with standard-conforming strings, which was the actual bug that the previous patch was meant to solve. Back-patch to all supported branches. The previous patch went back to 8.2, which unfortunately means that the EOL release of 8.2 contains this bug, but I don't think we're doing another 8.2 release just because of that.
2012-01-06 19:04:09 +01:00
/* Clear state allocated by CloneArchive */
if (AH->sqlparse.curCmd)
destroyPQExpBuffer(AH->sqlparse.curCmd);
Provide for parallel restoration from a custom format archive. Each data and post-data step is run in a separate worker child (a thread on Windows, a child process elsewhere) up to the concurrent number specified by the new pg_restore command-line --multi-thread | -m switch. Andrew Dunstan, with some editing by Tom Lane.
2009-02-02 21:07:37 +01:00
/* Clear any connection-local state */
if (AH->currUser)
free(AH->currUser);
if (AH->currSchema)
free(AH->currSchema);
if (AH->currTablespace)
free(AH->currTablespace);
if (AH->savedPassword)
free(AH->savedPassword);
free(AH);
}