Commit Graph

3945 Commits

Author SHA1 Message Date
Robert Haas b89e151054 Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables.  The output format is controlled by a
so-called "output plugin"; an example is included.  To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.

Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.

Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve
Singer.
2014-03-03 16:32:18 -05:00
Peter Eisentraut de94b47c0a Fix whitespace 2014-03-03 14:05:33 -05:00
Alvaro Herrera 9067310cc5 pg_dump et al: Add --if-exists option
This option makes pg_dump, pg_dumpall and pg_restore inject an IF EXISTS
clause to each DROP command they emit.  (In pg_dumpall, the clause is
not added to individual objects drops, but rather to the CREATE DATABASE
commands, as well as CREATE ROLE and CREATE TABLESPACE.)

This allows for a better user dump experience when using --clean in case
some objects do not already exist.  Per bug #7873 by Dave Rolsky.

Author: Pavel Stěhule
Reviewed-by: Jeevan Chalke, Álvaro Herrera, Josh Kupershmidt
2014-03-03 15:02:18 -03:00
Stephen Frost b1aebbb6a8 Various Coverity-spotted fixes
A number of issues were identified by the Coverity scanner and are
addressed in this patch.  None of these appear to be security issues
and many are mostly cosmetic changes.

Short comments for each of the changes follows.

Correct the semi-colon placement in be-secure.c regarding SSL retries.
Remove a useless comparison-to-NULL in proc.c (value is dereferenced
  prior to this check and therefore can't be NULL).
Add checking of chmod() return values to initdb.
Fix a couple minor memory leaks in initdb.
Fix memory leak in pg_ctl- involves free'ing the config file contents.
Use an int to capture fgetc() return instead of an enum in pg_dump.
Fix minor memory leaks in pg_dump.
  (note minor change to convertOperatorReference()'s API)
Check fclose()/remove() return codes in psql.
Check fstat(), find_my_exec() return codes in psql.
Various ECPG memory leak fixes.
Check find_my_exec() return in ECPG.
Explicitly ignore pqFlush return in libpq error-path.
Change PQfnumber() to avoid doing an strdup() when no changes required.
Remove a few useless check-against-NULL's (value deref'd beforehand).
Check rmtree(), malloc() results in pg_regress.
Also check get_alternative_expectfile() return in pg_regress.
2014-03-01 22:14:14 -05:00
Alvaro Herrera ef5856fd9b Allow BASE_BACKUP to be throttled
A new MAX_RATE option allows imposing a limit to the network transfer
rate from the server side.  This is useful to limit the stress that
taking a base backup has on the server.

pg_basebackup is now able to specify a value to the server, too.

Author: Antonin Houska

Patch reviewed by Stefan Radomski, Andres Freund, Zoltán Böszörményi,
Fujii Masao, and Álvaro Herrera.
2014-02-27 18:55:57 -03:00
Bruce Momjian 4bad548d98 psql: add separate \d display for disabled system triggers
Previously if you disabled all triggers, only user triggers would
show as disabled

Per report from Andres Freund
2014-02-24 12:44:55 -05:00
Bruce Momjian d613861b95 pg_dump: fix subtle memory leak in func and arg signature processing 2014-02-24 12:32:41 -05:00
Peter Eisentraut fb05f3ce83 pg_basebackup: Add support for relocating tablespaces
Tablespaces can be relocated in plain backup mode by specifying one or
more -T olddir=newdir options.

Author: Steeve Lennmark <steevel@handeldsbanken.se>
Reviewed-by: Peter Eisentraut <peter_e@gmx.net>
2014-02-22 13:38:06 -05:00
Tom Lane ae5266f259 Remove inappropriate EXPORTS line.
Looks like this gets added later ...
2014-02-19 21:08:50 -05:00
Tom Lane 4f5f485d10 Avoid using dllwrap to build pgevent in Mingw builds.
If this works, we can get rid of configure's support for locating dllwrap
... but let's see what the buildfarm says, first.

Hiroshi Inoue
2014-02-19 19:34:50 -05:00
Tom Lane 01824385ae Prevent potential overruns of fixed-size buffers.
Coverity identified a number of places in which it couldn't prove that a
string being copied into a fixed-size buffer would fit.  We believe that
most, perhaps all of these are in fact safe, or are copying data that is
coming from a trusted source so that any overrun is not really a security
issue.  Nonetheless it seems prudent to forestall any risk by using
strlcpy() and similar functions.

Fixes by Peter Eisentraut and Jozef Mlich based on Coverity reports.

In addition, fix a potential null-pointer-dereference crash in
contrib/chkpass.  The crypt(3) function is defined to return NULL on
failure, but chkpass.c didn't check for that before using the result.
The main practical case in which this could be an issue is if libc is
configured to refuse to execute unapproved hashing algorithms (e.g.,
"FIPS mode").  This ideally should've been a separate commit, but
since it touches code adjacent to one of the buffer overrun changes,
I included it in this commit to avoid last-minute merge issues.
This issue was reported by Honza Horak.

Security: CVE-2014-0065 for buffer overruns, CVE-2014-0066 for crypt()
2014-02-17 11:20:21 -05:00
Tom Lane 60ff2fdd99 Centralize getopt-related declarations in a new header file pg_getopt.h.
We used to have externs for getopt() and its API variables scattered
all over the place.  Now that we find we're going to need to tweak the
variable declarations for Cygwin, it seems like a good idea to have
just one place to tweak.

In this commit, the variables are declared "#ifndef HAVE_GETOPT_H".
That may or may not work everywhere, but we'll soon find out.

Andres Freund
2014-02-15 14:31:30 -05:00
Bruce Momjian 32be1c8e90 Remove use of sscanf in pg_upgrade, and add C comment to pg_dump
Per report from Jackie Chang
2014-02-15 11:50:56 -05:00
Bruce Momjian a0d8947acb psql: Add C comment about gset_prefix being freed later 2014-02-15 00:09:40 -05:00
Tom Lane b8f00a46bc Clean up error cases in psql's COPY TO STDOUT/FROM STDIN code.
Adjust handleCopyOut() to stop trying to write data once it's failed
one time.  For typical cases such as out-of-disk-space or broken-pipe,
additional attempts aren't going to do anything but waste time, and
in any case clean truncation of the output seems like a better behavior
than randomly dropping blocks in the middle.

Also remove dubious (and misleadingly documented) attempt to force our way
out of COPY_OUT state if libpq didn't do that.  If we did have a situation
like that, it'd be a bug in libpq and would be better fixed there, IMO.
We can hope that commit fa4440f516 took care
of any such problems, anyway.

Also fix longstanding bug in handleCopyIn(): PQputCopyEnd() only supports
a non-null errormsg parameter in protocol version 3, and will actively
fail if one is passed in version 2.  This would've made our attempts
to get out of COPY_IN state after a failure into infinite loops when
talking to pre-7.4 servers.

Back-patch the COPY_OUT state change business back to 9.2 where it was
introduced, and the other two fixes into all supported branches.
2014-02-13 18:45:58 -05:00
Magnus Hagander 48870dd9f7 Add missing include, required on some platforms
Noted by the buildfarm and Andres Freund
2014-02-12 20:04:13 +01:00
Magnus Hagander 63ab2befe0 Kill pg_basebackup background process when exiting
If an error occurs in the foreground (backup) process of pg_basebackup,
and we exit in a controlled way, the background process (streaming
xlog process) would stay around and keep streaming.
2014-02-12 18:45:18 +01:00
Peter Eisentraut d3c4c47155 scripts: Remove newlines from end of generated SQL
This results in spurious empty lines in the server log.  Instead, add
the newlines only when printing out the --echo output.  In some cases,
this was already done, leading to two newlines being printed.  Clean
that up as well.

From: Fabrízio de Royes Mello <fabriziomello@gmail.com>
2014-02-10 21:47:19 -05:00
Stephen Frost dfb1e9bdc0 Further pg_dump / ftello improvements
Make ftello error-checking consistent to all calls and remove a
bit of ftello-related code which has been #if 0'd out since 2001.

Note that we are not concerned with the ftello() call under
snprintf() failing as it is just building a string to call
exit_horribly() with; printing -1 in such a case is fine.
2014-02-09 18:28:14 -05:00
Stephen Frost 5e8e794e3b Focus on ftello result < 0 instead of errno
Rather than reset errno (or just hope that its cleared already),
check just the result of the ftello for < 0 to determine if there
was an issue.

Oversight by me, pointed out by Tom.
2014-02-09 13:29:36 -05:00
Magnus Hagander 8198a321c9 Limit pg_basebackup progress output to 1/second
This prevents pg_basebackup from generating excessive output when
dumping large clusters. The status is now updated once / second,
still making it possible to see that there is progress happening,
but limiting the total bandwidth.

Mika Eloranta, reviewed by Sawada Masahiko and Oskari Saarenmaa
2014-02-09 12:51:42 +01:00
Magnus Hagander 01025d80a1 Avoid printing uninitialized filename variable in verbose mode
When using verbose mode for pg_basebackup, in tar format sent to
stdout, we'd print an unitialized buffer as the filename.

Reported by Pontus Lundkvist
2014-02-09 12:05:14 +01:00
Stephen Frost cfa1b4a711 Minor pg_dump improvements
Improve pg_dump by checking results on various fgetc() calls which
previously were unchecked, ditto for ftello.  Also clean up a couple
of very minor memory leaks by waiting to allocate structures until
after the initial check(s).

Issues spotted by Coverity.
2014-02-08 21:25:47 -05:00
Peter Eisentraut 66c04c981d Mark some more variables as static or include the appropriate header
Detected by clang's -Wmissing-variable-declarations.

From: Andres Freund <andres@anarazel.de>
2014-02-08 21:21:46 -05:00
Tom Lane 0c2338abbb Fix lexing of U& sequences just before EOF.
Commit a5ff502fce was a brick shy of a load
in the backend lexer too, not just psql.  Per further testing of bug #9068.

In passing, improve related comments.
2014-02-03 19:47:57 -05:00
Tom Lane 47aaebaac9 Switch in psql_scan() must cover all lexer states (except backslash cases).
Oversight in commit f7559c0101, which changed
UESCAPE lexing in psql.  Per bug #9068 from Manuel Gómez.
2014-02-02 18:59:34 -05:00
Fujii Masao 63be3b78f6 Fix typos in docs and comments.
Thom Brown
2014-02-02 10:28:18 +09:00
Robert Haas 858ec11858 Introduce replication slots.
Replication slots are a crash-safe data structure which can be created
on either a master or a standby to prevent premature removal of
write-ahead log segments needed by a standby, as well as (with
hot_standby_feedback=on) pruning of tuples whose removal would cause
replication conflicts.  Slots have some advantages over existing
techniques, as explained in the documentation.

In a few places, we refer to the type of replication slots introduced
by this patch as "physical" slots, because forthcoming patches for
logical decoding will also have slots, but with somewhat different
properties.

Andres Freund and Robert Haas
2014-01-31 22:45:36 -05:00
Bruce Momjian 5168c76964 pg_restore: make help output plural for multi-enabled options
per report from Josh Kupershmidt
2014-01-31 22:29:01 -05:00
Fujii Masao a87ae38be8 Add tab completion for ALTER TABLESPACE MOVE in psql. 2014-02-01 01:45:48 +09:00
Tom Lane 571addd729 Fix unsafe references to errno within error messaging logic.
Various places were supposing that errno could be expected to hold still
within an ereport() nest or similar contexts.  This isn't true necessarily,
though in some cases it accidentally failed to fail depending on how the
compiler chanced to order the subexpressions.  This class of thinko
explains recent reports of odd failures on clang-built versions, typically
missing or inappropriate HINT fields in messages.

Problem identified by Christian Kruse, who also submitted the patch this
commit is based on.  (I fixed a few issues in his patch and found a couple
of additional places with the same disease.)

Back-patch as appropriate to all supported branches.
2014-01-29 20:04:43 -05:00
Fujii Masao 7c619be623 Fix typos in comments for ALTER SYSTEM.
Michael Paquier
2014-01-27 12:23:20 +09:00
Stephen Frost 152d24f5dd Fix minor leak in pg_dump
Move allocation to after we check the remote server version, to avoid
a possible, very minor, memory leak.  This makes us more consistent
throughout as most places in pg_dump are done in the same way (due, in
part, to previous fixes like this).

Spotted by the Coverity scanner.
2014-01-26 17:58:48 -05:00
Magnus Hagander f2795f8b53 Move the options column of \db+ before the description
The convention is to have the description field at the end.

Noted by Tom Lane
2014-01-26 21:16:02 +01:00
Magnus Hagander cae10ca27e Include tablespace options in verbose output of \db 2014-01-26 18:42:08 +01:00
Andrew Dunstan cec8394b5c Enable building with Visual Studion 2013.
Backpatch to 9.3.

Brar Piening.
2014-01-26 09:49:10 -05:00
Noah Misch 3a5313265d psql: Mention SSL protocol version in \conninfo.
Marko Kreen, reviewed by Wim Lewis.
2014-01-24 19:23:56 -05:00
Stephen Frost 6794a9f9a1 Avoid minor leak in parallel pg_dump
During parallel pg_dump, a worker process closing the connection caused
a minor memory leak (particularly minor as we are likely about to exit
anyway).  Instead, free the memory in this case prior to returning NULL
to indicate connection closed.

Spotting by the Coverity scanner.

Back patch to 9.3 where this was introduced.
2014-01-24 15:10:08 -05:00
Fujii Masao 9f80f4835a Add libpq function PQhostaddr().
There was a bug in the psql's meta command \conninfo. When the
IP address was specified in the hostaddr and psql used it to create
a connection (i.e., psql -d "hostaddr=xxx"), \conninfo could not
display that address. This is because \conninfo got the connection
information only from PQhost() which could not return hostaddr.

This patch adds PQhostaddr(), and changes \conninfo so that it
can display not only the host name that PQhost() returns but also
the IP address which PQhostaddr() returns.

The bug has existed since 9.1 where \conninfo was introduced.
But it's too late to add new libpq function into the released versions,
so no backpatch.
2014-01-24 02:32:39 +09:00
Magnus Hagander 98de86e422 Remove support for native krb5 authentication
krb5 has been deprecated since 8.3, and the recommended way to do
Kerberos authentication is using the GSSAPI authentication method
(which is still fully supported).

libpq retains the ability to identify krb5 authentication, but only
gives an error message about it being unsupported. Since all authentication
is initiated from the backend, there is no need to keep it at all
in the backend.
2014-01-19 17:05:01 +01:00
Peter Eisentraut ad6bf0291a Fix client-only installation
The psql Makefile was not creating $(datadir) before installing
psqlrc.sample there.

In most cases, the directory would be created in some other way, but for
the documented from-source client-only installation procedure, it could
fail.

Reported-by: Mike Blackwell <mike.blackwell@rrd.com>
2014-01-17 23:08:22 -05:00
Tom Lane 515d2c596c Add display of oprcode (the underlying function's name) to psql's \do+.
The + modifier of \do didn't use to do anything, but now it adds an oprcode
column.  This is useful both as an additional form of documentation of what
the operator does, and to save a step when finding out properties of the
underlying function.

Marko Tiikkaja, reviewed by Rushabh Lathia, adjusted a bit by me
2014-01-16 15:29:33 -05:00
Tom Lane 5df99f6481 Improve FILES section of psql reference page.
Primarily, explain where to find the system-wide psqlrc file, per recent
gripe from John Sutton.  Do some general wordsmithing and improve the
markup, too.

Also adjust psqlrc.sample so its comments about file location are somewhat
trustworthy.  (Not sure why we bother with this file when it's empty,
but whatever.)

Back-patch to 9.2 where the startup file naming scheme was last changed.
2014-01-14 19:27:57 -05:00
Bruce Momjian bb953ad164 Fix pg_dumpall on pre-8.1 servers
rolname did not exist in pg_shadow.

Backpatch to 9.3

Report by Andrew Gierth via IRC
2014-01-12 22:25:36 -05:00
Bruce Momjian 111022eac6 Move username lookup functions from /port to /common
Per suggestion from Peter E and Alvaro
2014-01-10 18:03:28 -05:00
Bruce Momjian 7e04792a1c Update copyright for 2014
Update all files in head, and files COPYRIGHT and legal.sgml in all back
branches.
2014-01-07 16:05:30 -05:00
Heikki Linnakangas 10a82cda67 Remove bogus -K option from pg_dump.
I added it to the getopt call by accident in commit
691e595dd9.

Amit Kapila
2014-01-06 12:30:19 +02:00
Tom Lane 92459e7a7f Fix translatability markings in psql, and add defenses against future bugs.
Several previous commits have added columns to various \d queries without
updating their translate_columns[] arrays, leading to potentially incorrect
translations in NLS-enabled builds.  Offenders include commit 893686762
(added prosecdef to \df+), c9ac00e6e (added description to \dc+) and
3b17efdfd (added description to \dC+).  Fix those cases back to 9.3 or
9.2 as appropriate.

Since this is evidently more easily missed than one would like, in HEAD
also add an Assert that the supplied array is long enough.  This requires
an API change for printQuery(), so it seems inappropriate for back
branches, but presumably all future changes will be tested in HEAD anyway.

In HEAD and 9.3, also clean up a whole lot of sloppiness in the emitted
SQL for \dy (event triggers): lack of translatability due to failing to
pass words-to-be-translated through gettext_noop(), inadequate schema
qualification, and sloppy formatting resulting in unnecessarily ugly
-E output.

Peter Eisentraut and Tom Lane, per bug #8702 from Sergey Burladyan
2014-01-04 16:05:16 -05:00
Tom Lane c01bc51f8d Fix broken support for event triggers as extension members.
CREATE EVENT TRIGGER forgot to mark the event trigger as a member of its
extension, and pg_dump didn't pay any attention anyway when deciding
whether to dump the event trigger.  Per report from Moshe Jacobson.

Given the obvious lack of testing here, it's rather astonishing that
ALTER EXTENSION ADD/DROP EVENT TRIGGER work, but they seem to.
2013-12-30 14:00:02 -05:00
Kevin Grittner 47f50262e7 Don't attempt to limit target database for pg_restore.
There was an apparent attempt to limit the target database for
pg_restore to version 7.1.0 or later.  Due to a leading zero this
was interpreted as an octal number, which allowed targets with
version numbers down to 2.87.36.  The lowest actual release above
that was 6.0.0, so that was effectively the limit.

Since the success of the restore attempt will depend primarily on
on what statements were generated by the dump run, we don't want
pg_restore trying to guess whether a given target should be allowed
based on version number.  Allow a connection to any version.  Since
it is very unlikely that anyone would be using a recent version of
pg_restore to restore to a pre-6.0 database, this has little to no
practical impact, but it makes the code less confusing to read.

Issue reported and initial patch suggestion from Joel Jacobson
based on an article by Andrey Karpov reporting on issues found by
PVS-Studio static code analyzer.  Final patch based on analysis by
Tom Lane.  Back-patch to all supported branches.
2013-12-29 15:17:52 -06:00
Tom Lane 8d65da1f01 Support ordered-set (WITHIN GROUP) aggregates.
This patch introduces generic support for ordered-set and hypothetical-set
aggregate functions, as well as implementations of the instances defined in
SQL:2008 (percentile_cont(), percentile_disc(), rank(), dense_rank(),
percent_rank(), cume_dist()).  We also added mode() though it is not in the
spec, as well as versions of percentile_cont() and percentile_disc() that
can compute multiple percentile values in one pass over the data.

Unlike the original submission, this patch puts full control of the sorting
process in the hands of the aggregate's support functions.  To allow the
support functions to find out how they're supposed to sort, a new API
function AggGetAggref() is added to nodeAgg.c.  This allows retrieval of
the aggregate call's Aggref node, which may have other uses beyond the
immediate need.  There is also support for ordered-set aggregates to
install cleanup callback functions, so that they can be sure that
infrastructure such as tuplesort objects gets cleaned up.

In passing, make some fixes in the recently-added support for variadic
aggregates, and make some editorial adjustments in the recent FILTER
additions for aggregates.  Also, simplify use of IsBinaryCoercible() by
allowing it to succeed whenever the target type is ANY or ANYELEMENT.
It was inconsistent that it dealt with other polymorphic target types
but not these.

Atri Sharma and Andrew Gierth; reviewed by Pavel Stehule and Vik Fearing,
and rather heavily editorialized upon by Tom Lane
2013-12-23 16:11:35 -05:00
Fujii Masao 961bf59fb7 Rename wal_log_hintbits to wal_log_hints, per discussion on pgsql-hackers.
Sawada Masahiko
2013-12-21 03:33:16 +09:00
Fujii Masao 084e385a2f Add tab completion for ALTER SYSTEM SET in psql. 2013-12-20 02:33:27 +09:00
Bruce Momjian 613c6d26bd Fix incorrect error message reported for non-existent users
Previously, lookups of non-existent user names could return "Success";
it will now return "User does not exist" by resetting errno.  This also
centralizes the user name lookup code in libpgport.

Report and analysis by Nicolas Marchildon;  patch by me
2013-12-18 12:16:21 -05:00
Tatsuo Ishii 65d6e4cb5c Add ALTER SYSTEM command to edit the server configuration file.
Patch contributed by Amit Kapila. Reviewed by Hari Babu, Masao Fujii,
Boszormenyi Zoltan, Andres Freund, Greg Smith and others.
2013-12-18 23:42:44 +09:00
Heikki Linnakangas 30b96549ab Mark variables 'static' where possible. Move GinFuzzySearchLimit to ginget.c
Per "clang -Wmissing-variable-declarations" output, posted by Andres Freund.
I didn't silence all those warnings, though, only the most obvious cases.
2013-12-16 11:41:17 +02:00
Heikki Linnakangas 50e547096c Add GUC to enable WAL-logging of hint bits, even with checksums disabled.
WAL records of hint bit updates is useful to tools that want to examine
which pages have been modified. In particular, this is required to make
the pg_rewind tool safe (without checksums).

This can also be used to test how much extra WAL-logging would occur if
you enabled checksums, without actually enabling them (which you can't
currently do without re-initdb'ing).

Sawada Masahiko, docs by Samrat Revagade. Reviewed by Dilip Kumar, with
further changes by me.
2013-12-13 16:26:14 +02:00
Heikki Linnakangas 108e3992cd Display old and new values in pg_resetxlog -n output.
For extra clarity.

Rajeev Rastogi, reviewed by Amit Kapila
2013-12-12 11:57:18 +02:00
Robert Haas 6bea96dd49 Add a new option, -g, to createuser, to add membership in a role.
Chistopher Browne, reviewed by Sameer Thakur, Amit Kapila, and
Peter Eisentraut.
2013-12-11 07:50:36 -05:00
Robert Haas e55704d8b2 Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
07cacba983.  This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all.  Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.

Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.

Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.
2013-12-10 19:01:40 -05:00
Peter Eisentraut 3e3520cf7a Translation updates 2013-12-02 00:17:07 -05:00
Kevin Grittner 4bd371f6f8 Fix pg_dumpall to work for databases flagged as read-only.
pg_dumpall's charter is to be able to recreate a database cluster's
contents in a virgin installation, but it was failing to honor that
contract if the cluster had any ALTER DATABASE SET
default_transaction_read_only settings.  By including a SET command
for the connection for each connection opened by pg_dumpall output,
errors are avoided and the source cluster is successfully
recreated.

There was discussion of whether to also set this for the connection
applying pg_dump output, but it was felt that it was both less
appropriate in that context, and far easier to work around.

Backpatch to all supported branches.
2013-11-30 11:24:56 -06:00
Tom Lane 79193c75f8 Fix assorted issues in pg_ctl's pgwin32_CommandLine().
Ensure that the invocation command for postgres or pg_ctl runservice
double-quotes the executable's pathname; failure to do this leads to
trouble when the path contains spaces.

Also, ensure that the path ends in ".exe" in both cases and uses
backslashes rather than slashes as directory separators.  The latter issue
is reported to confuse some third-party tools such as Symantec Backup Exec.

Also, rewrite the function to avoid buffer overrun issues by using a
PQExpBuffer instead of a fixed-size static buffer.  Combinations of
very long executable pathnames and very long data directory pathnames
could have caused trouble before, for example.

Back-patch to all active branches, since this code has been like this
for a long while.

Naoya Anzai and Tom Lane, reviewed by Rajeev Rastogi
2013-11-29 18:34:07 -05:00
Fujii Masao d1b88f6b36 Add --xlogdir option to pg_basebackup, for specifying the pg_xlog directory.
Haribabu kommi, slightly modified by me.
2013-11-27 14:00:16 +09:00
Peter Eisentraut f4482a542c Tweak streamutil.c further to avoid scan-build warning
The previous change added a new scan-build warning about need_password
assigned but not read.
2013-11-21 21:46:43 -05:00
Fujii Masao 38f4328981 Fix pg_isready to handle -d option properly.
Previously, -d option for pg_isready was broken. When the name of the
database was specified by -d option, pg_isready failed with an error.
When the conninfo specified by -d option contained the setting of the
host name but not Numeric IP address (i.e., hostaddr), pg_isready
displayed wrong connection message. -d option could not handle a valid
URI prefix at all. This commit fixes these bugs of pg_isready.

Backpatch to 9.3, where pg_isready was introduced.

Per report from Josh Berkus and Robert Haas.
Original patch by Fabrízio de Royes Mello, heavily modified by me.
2013-11-21 21:52:03 +09:00
Fujii Masao b1543cc8a8 Add tab completion for \pset in psql.
Pavel Stehule, reviewed by Ian Lawrence Barwick
2013-11-19 23:44:14 +09:00
Heikki Linnakangas fea437681d Spell SQL keywords in uppercase in pg_dump's query.
The server won't care, but let's be consistent.

David Rowley.
2013-11-18 18:34:51 +02:00
Heikki Linnakangas 32ceba3ea7 Replace appendPQExpBuffer(..., <constant>) with appendPQExpBufferStr
Arguably makes the code a bit more readable, and might give a small
performance gain.

David Rowley
2013-11-18 18:34:51 +02:00
Tom Lane 6cb86143e8 Allow aggregates to provide estimates of their transition state data size.
Formerly the planner had a hard-wired rule of thumb for guessing the amount
of space consumed by an aggregate function's transition state data.  This
estimate is critical to deciding whether it's OK to use hash aggregation,
and in many situations the built-in estimate isn't very good.  This patch
adds a column to pg_aggregate wherein a per-aggregate estimate can be
provided, overriding the planner's default, and infrastructure for setting
the column via CREATE AGGREGATE.

It may be that additional smarts will be required in future, perhaps even
a per-aggregate estimation function.  But this is already a step forward.

This is extracted from a larger patch to improve the performance of numeric
and int8 aggregates.  I (tgl) thought it was worth reviewing and committing
this infrastructure separately.  In this commit, all built-in aggregates
are given aggtransspace = 0, so no behavior should change.

Hadi Moshayedi, reviewed by Pavel Stehule and Tomas Vondra
2013-11-16 16:03:40 -05:00
Tom Lane 97e1ec4670 Speed up printing of INSERT statements in pg_dump.
In --inserts and especially --column-inserts mode, we can get a useful
speedup by generating the common prefix of all a table's INSERT commands
just once, and then printing the prebuilt string for each row.  This avoids
multiple invocations of fmtId() and other minor fooling around.

David Rowley
2013-11-15 18:02:06 -05:00
Tom Lane 3172eea062 Clean up password prompting logic in streamutil.c.
The previous coding was fairly unreadable and drew double-free warnings
from clang.  I believe the double free was actually not reachable, because
PQconnectionNeedsPassword is coded to not return true if a password was
provided, so that the loop can't iterate more than twice.  Nonetheless
it seems worth rewriting.  No back-patch since this is just cosmetic.
2013-11-15 17:27:41 -05:00
Tom Lane e694cf25d7 Fix incorrect column name in psql \d code.
pg_index.indisreplident had at one time in its development been called
indisidentity.  describe.c got missed when it was renamed.
Bug introduced in commit 07cacba983.

Andres Freund
2013-11-14 10:27:24 -05:00
Magnus Hagander 705556a631 Don't abort pg_basebackup when receiving empty WAL block
This is a similar fix as c6ec8793aa
9.2. This should never happen in 9.3 and newer since the special case
cannot happen there, but this patch synchronizes up the code so there
is no confusion on why they're different. An empty block is as harmless
in 9.3 as it was in 9.2, and can safely be ignored.
2013-11-11 14:59:55 +01:00
Peter Eisentraut 001e114b8d Fix whitespace issues found by git diff --check, add gitattributes
Set per file type attributes in .gitattributes to fine-tune whitespace
checks.  With the associated cleanups, the tree is now clean for git
2013-11-10 14:48:29 -05:00
Robert Haas 07cacba983 Add the notion of REPLICA IDENTITY for a table.
Pending patches for logical replication will use this to determine
which columns of a tuple ought to be considered as its candidate key.

Andres Freund, with minor, mostly cosmetic adjustments by me
2013-11-08 12:30:43 -05:00
Tom Lane d4e6133c68 Revert commit 0725065b37.
The previous commit was intended to make psql show the full path name when
doing a \s (history save), but it was very badly implemented and would show
confusing if not outright wrong information in many situations; for
instance if the path name given to \s is absolute, or if \cd commands
involving relative paths have been issued.  Consensus seems to be that
we don't especially need this functionality in \s, and certainly not in \s
alone.  So revert rather than trying to fix it up.  Per gripe from
Ian Barwick.

Although the bogus behavior exists in all supported versions, I'm not
back-patching, because the work created for translators (by change of
a translatable message) would probably outweigh the value of what is
after all a mostly-cosmetic change.
2013-11-05 17:52:09 -05:00
Heikki Linnakangas 2103430179 Fix parsing of xlog file name in pg_receivexlog.
The parsing of WAL filenames of segments larger than > 255 was broken,
making pg_receivexlog unable to restart streaming after stopping it.

The bug was introduced by the changes in 9.3 to represent WAL segment number
as a 64-bit integer instead of two ints, log and seg. To fix, replace the
plain sscanf call with XLogFromFileName macro, which does the conversion
from log+seg to a 64-bit integer correcly.

Reported by Mika Eloranta.
2013-11-04 10:57:58 +02:00
Tom Lane 5e1e47c7c0 Ignore SIGSYS during initdb.
This prevents the recently-added probe for shm_open() from crashing
on platforms that are impolite enough to deliver a signal rather than
returning ENOSYS for an unimplemented kernel call.  At least on the
one known example (HPUX 10.20), ignoring SIGSYS does result in the
desired behavior of getting an ENOSYS error return instead.

Per discussion, we might later wish to do this in the backend as well,
but for now it seems sufficient to do it in initdb.
2013-10-24 21:51:30 -04:00
Tom Lane 3147acd63e Use improved vsnprintf calling logic in more places.
When we are using a C99-compliant vsnprintf implementation (which should be
most places, these days) it is worth the trouble to make use of its report
of how large the buffer needs to be to succeed.  This patch adjusts
stringinfo.c and some miscellaneous usages in pg_dump to do that, relying
on the logic recently added in libpgcommon's psprintf.c.  Since these
places want to know the number of bytes written once we succeed, modify the
API of pvsnprintf() to report that.

There remains near-duplicate logic in pqexpbuffer.c, but since that code
is in libpq, psprintf.c's approach of exit()-on-error isn't appropriate
for use there.  Also note that I didn't bother touching the multitude
of places that call (v)snprintf without any attempt to provide a resizable
buffer.

Release-note-worthy incompatibility: the API of appendStringInfoVA()
changed.  If there's any third-party code that's calling that directly,
it will need tweaking along the same lines as in this patch.

David Rowley and Tom Lane
2013-10-24 21:43:57 -04:00
Robert Haas 5c4dd2cd9a Simplify tab completion rules for views and foreign tables.
Since an increasing number of views and foreign tables are now able
to be updated, complete with any table, view, or foreign table in
the relevant contexts.  This avoids the need to use a complex
query that may be both confusing to end-users and nonperformant
to construct the list of possible completions.

Dean Rasheed, persuant to a complaint from Bernd Helme and a
suggestion from Peter Eisentraut
2013-10-23 13:16:25 -04:00
Tom Lane 2c66f9924c Replace pg_asprintf() with psprintf().
This eliminates an awkward coding pattern that's also unnecessarily
inconsistent with backend coding.  psprintf() is now the thing to
use everywhere.
2013-10-22 19:40:26 -04:00
Tom Lane 09a89cb5fc Get rid of use of asprintf() in favor of a more portable implementation.
asprintf(), aside from not being particularly portable, has a fundamentally
badly-designed API; the psprintf() function that was added in passing in
the previous patch has a much better API choice.  Moreover, the NetBSD
implementation that was borrowed for the previous patch doesn't work with
non-C99-compliant vsnprintf, which is something we still have to cope with
on some platforms; and it depends on va_copy which isn't all that portable
either.  Get rid of that code in favor of an implementation similar to what
we've used for many years in stringinfo.c.  Also, move it into libpgcommon
since it's not really libpgport material.

I think this patch will be enough to turn the buildfarm green again, but
there's still cosmetic work left to do, namely get rid of pg_asprintf()
in favor of using psprintf().  That will come in a followon patch.
2013-10-22 18:42:13 -04:00
Peter Eisentraut 2e6bc4b806 Move rmtree() from libpgport to libpgcommon
It requires pgfnames() from libpgcommon.
2013-10-19 10:20:51 -04:00
Peter Eisentraut ba7c5975ad Move pgfnames() from libpgport to libpgcommon
It requires pstrdup() from libpgcommon.
2013-10-18 21:28:15 -04:00
Peter Eisentraut f39418e9b3 Switch dependency order of libpgcommon and libpgport
Continuing 63f32f3416, libpgcommon should
depend on libpgport, but not vice versa.  But wait_result_to_str() in
wait_error.c depends on pstrdup() in libpgcommon.  So move exec.c and
wait_error.c from libpgport to libpgcommon.  Also switch the link order
in the place that's actually used by the failing ecpg builds.

The function declarations have been left in port.h for now.  That should
perhaps be separated sometime.
2013-10-17 22:02:35 -04:00
Robert Haas d41ab71712 initdb: Suppress dynamic shared memory when probing for max_connections.
This might not be the right long-term solution here, but it will
hopefully turn the buildfarm green again.

Oversight noted by Andres Freund
2013-10-16 09:41:03 -04:00
Peter Eisentraut 382b694175 Translation updates to fix build failures
Now that msgfmt is run with -c by default, older versions of gettext are
complaining about the PO headers Last-Translator and Language-Team
still having their default values.  Newer gettext versions fail to catch
this because of a bug (https://savannah.gnu.org/bugs/?40261), which is
why this hasn't been noticed before.

Copy updated versions of affected translation files from the
pgtranslations repository, were those files have been fixed.
2013-10-13 22:14:26 -04:00
Peter Eisentraut 5b6d08cd29 Add use of asprintf()
Add asprintf(), pg_asprintf(), and psprintf() to simplify string
allocation and composition.  Replacement implementations taken from
NetBSD.

Reviewed-by: Álvaro Herrera <alvherre@2ndquadrant.com>
Reviewed-by: Asif Naeem <anaeem.it@gmail.com>
2013-10-13 00:09:18 -04:00
Robert Haas 7195fb3cec initdb: Select working dynamic shared memory implementation.
If POSIX shared memory is supported and works, we prefer it.
Otherwise, we prefer System V, except on Windows, where we use
the implementation specific to that platform.
2013-10-10 19:46:31 -04:00
Robert Haas 0ac5e5a7e1 Allow dynamic allocation of shared memory segments.
Patch by myself and Amit Kapila.  Design help from Noah Misch.  Review
by Andres Freund.
2013-10-09 21:05:02 -04:00
Peter Eisentraut 0b109c822b Translation updates 2013-10-07 16:51:52 -04:00
Robert Haas d90ced8bb2 Add DISCARD SEQUENCES command.
DISCARD ALL will now discard cached sequence information, as well.

Fabrízio de Royes Mello, reviewed by Zoltán Böszörményi, with some
further tweaks by me.
2013-10-03 16:23:31 -04:00
Robert Haas c64e68fd9f psql: Make \pset without arguments show all settings.
Gilles Darold, reviewed by Pavel Stehule
2013-10-03 15:18:02 -04:00
Peter Eisentraut 9099e4afe0 psql: Set up cancel handler later
The cancel handler was uselessly set up even before the first connection
was opened.  By setting it up afterwards, the user can use Ctrl+C to
abort psql if the initial connection attempt hangs.

Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
Reviewed-by: Ryan Kelly <rpkelly22@gmail.com>
2013-10-02 21:05:08 -04:00
Heikki Linnakangas b882246e3a Fix two timeline handling bugs in pg_receivexlog.
When a timeline history file is fetched from server, it is initially created
with a temporary file name, and renamed to place. However, the temporary
file name was constructed using an uninitialized buffer. Usually that meant
that the file was created in current directory instead of the target, which
usually goes unnoticed, but if the target is on a different filesystem than
the current dir, the rename() would fail. Fix that.

The second issue is that pg_receivexlog would not take .partial files into
account when determining when scanning the target directory for existing
WAL files. If the timeline has switched in the server several times in the
last WAL segment, and pg_receivexlog is restarted, it would choose a too
old starting point. That's not a problem as long as the old WAL segment
exists in the server and can be streamed over, but will cause a failure if
it's not.

Backpatch to 9.3, where this timeline handling code was written.

Analysed by Andrew Gierth, bug #8453, based on a bug report on IRC.
2013-09-23 10:40:11 +03:00
Fujii Masao 6e5f6e5913 Fix typo in comment.
Ian Lawrence Barwick
2013-09-19 01:58:32 +09:00
Bruce Momjian d8a5608d4a psql: fix \copy stdin trailing space requirement
Previously a trailing space was required for \copy ... stdin:

	copy foo from stdin ;

Etsuro Fujita
2013-09-10 19:36:10 -04:00
Bruce Momjian 601f48076f psql: honor 'footer' option for expanded output
"No rows" previously only honored the tuples-only option.

Per report from Eli Mesika
2013-09-10 19:07:06 -04:00
Tom Lane 0d3f4406df Allow aggregate functions to be VARIADIC.
There's no inherent reason why an aggregate function can't be variadic
(even VARIADIC ANY) if its transition function can handle the case.
Indeed, this patch to add the feature touches none of the planner or
executor, and little of the parser; the main missing stuff was DDL and
pg_dump support.

It is true that variadic aggregates can create the same sort of ambiguity
about parameters versus ORDER BY keys that was complained of when we
(briefly) had both one- and two-argument forms of string_agg().  However,
the policy formed in response to that discussion only said that we'd not
create any built-in aggregates with varying numbers of arguments, not that
we shouldn't allow users to do it.  So the logical extension of that is
we can allow users to make variadic aggregates as long as we're wary about
shipping any such in core.

In passing, this patch allows aggregate function arguments to be named, to
the extent of remembering the names in pg_proc and dumping them in pg_dump.
You can't yet call an aggregate using named-parameter notation.  That seems
like a likely future extension, but it'll take some work, and it's not what
this patch is really about.  Likewise, there's still some work needed to
make window functions handle VARIADIC fully, but I left that for another
day.

initdb forced because of new aggvariadic field in Aggref parse nodes.
2013-09-03 17:08:46 -04:00