Fix all the standard PLs to be able to return tuples from FOO_RETURNING
statements as well as utility statements that return tuples. Also,
fix oversight that SPI_processed wasn't set for a utility statement
returning tuples. Per recent discussion.
loaded libraries: call functions _PG_init() and _PG_fini() if the library
defines such symbols. Hence we no longer need to specify an initialization
function in preload_libraries: we can assume that the library used the
_PG_init() convention, instead. This removes one source of pilot error
in use of preloaded libraries. Original patch by Ralf Engelschall,
preload_libraries changes by me.
pg_regress: there's no other way to cope with testing a relocated
installation. Seems better to call it --psqldir though, since the
only thing we need to find in that case is psql. It'd be better if
we could use find_other_exec, but that's not happening unless we are
willing to install pg_regress alongside psql, which seems unlikely
to happen.
This allows it to be used on Windows without installing mingw
(though you do still need 'diff'), and opens the door to future
improvements such as message localization.
Magnus Hagander and Tom Lane.
by creating a reference-count mechanism, similar to what we did a long time
ago for catcache entries. The back branches have an ugly solution involving
lots of extra copies, but this way is more efficient. Reference counting is
only applied to tupdescs that are actually in caches --- there seems no need
to use it for tupdescs that are generated in the executor, since they'll go
away during plan shutdown by virtue of being in the per-query memory context.
Neil Conway and Tom Lane
kept but now deprecated. Patch from Adam Sjøgren. Add regression test to
show plperl trigger data (Andrew).
TBD: apply similar changes to plpgsql, plpython and pltcl.
functions are not strict, they will be called (passing a NULL first parameter)
during any attempt to input a NULL value of their datatype. Currently, all
our input functions are strict and so this commit does not change any
behavior. However, this will make it possible to build domain input functions
that centralize checking of domain constraints, thereby closing numerous holes
in our domain support, as per previous discussion.
While at it, I took the opportunity to introduce convenience functions
InputFunctionCall, OutputFunctionCall, etc to use in code that calls I/O
functions. This eliminates a lot of grotty-looking casts, but the main
motivation is to make it easier to grep for these places if we ever need
to touch them again.
during parse analysis, not only errors detected in the flex/bison stages.
This is per my earlier proposal. This commit includes all the basic
infrastructure, but locations are only tracked and reported for errors
involving column references, function calls, and operators. More could
be done later but this seems like a good set to start with. I've also
moved the ReportSyntaxErrorPosition logic out of psql and into libpq,
which should make it available to more people --- even within psql this
is an improvement because warnings weren't handled by ReportSyntaxErrorPosition.
memory in the executor's per-query memory context. It also inefficient:
it invokes get_call_result_type() and TupleDescGetAttInMetadata() for
every call to return_next, rather than invoking them once (per PL/Perl
function call) and memoizing the result.
This patch makes the following changes:
- refactor the code to include all the "per PL/Perl function call" data
inside a single struct, "current_call_data". This means we don't need to
save and restore N pointers for every recursive call into PL/Perl, we
can just save and restore one.
- lookup the return type metadata needed by plperl_return_next() once,
and then stash it in "current_call_data", so as to avoid doing the
lookup for every call to return_next.
- create a temporary memory context in which to evaluate the return
type's input functions. This memory context is reset for each call to
return_next.
The patch appears to fix the memory leak, and substantially reduces
the overhead imposed by return_next.
comment line where output as too long, and update typedefs for /lib
directory. Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).
Backpatch to 8.1.X.
return arays nicely without having to make the plperl programmer aware
of anything. The attached patch allows plperl to return an arrayref
where the function returns an array type. It silently calls a perl
function to stringify the array before passing it to the pg array
parser. Non-array returns are handled as before (i.e. passed through
this process) so it is backwards compatible. I will presently submit
regression tests and docs.
example:
andrew=# create or replace function blah() returns text[][] language
plperl as $$ return [['a"b','c,d'],['e\\f','g']]; $$;
CREATE FUNCTION
andrew=# select blah();
blah
-----------------------------
{{"a\"b","c,d"},{"e\\f",g}}
This would complete half of the TODO item:
. Pass arrays natively instead of as text between plperl and postgres
(The other half is translating pg array arguments to perl arrays - that
will have to wait for 8.1).
Some of this patch is adapted from a previously submitted patch from
Sergej Sergeev. Both he and Abhijit Menon-Sen have looked it over
briefly and tentatively said it looks ok.
Andrew Dunstan
for PL/Perl, to avoid loading the entire result set into memory as the
existing spi_exec_query() function does.
Here's how one might use the new functions:
$x = spi_query("select ...");
while (defined ($y = spi_fetchrow($x))) {
...
return_next(...);
}
The changes do not affect the spi_exec_query() interface in any way.
Abhijit Menon-Sen
plperl - the attached small patch remedies that omission, and adds a
small regression test for error and warning output - the new regression
input and expected output are in separate attached files.
Andrew Dunstan
> against rc1. It simply checks with GetDatabaseEncoding() if the current
> database is in UTF-8, and if so, sets the UTF-8 flag on the arguments
> that are passed to perl. This means that it isn't necessary to
> utf8::upgrade() every string, as perl has no way of knowing offhand
> that a string is UTF-8 -- but postgres does, because the database
> encoding is specified, so it makes sense to turn the flag on. You
> should also be able to properly manipulate UTF-8 strings now from
> plperl as opposed to plperlu, because otherwise you'd have to use
> encoding 'utf8' which was not allowed. It could also eliminate some
> unexpected bugs if you assume that perl knows the string is unicode.
It
> is enabled only for perl 5.6 and higher, so earlier versions will not
> be affected.
>
> I have been assured by crab that the patch is quite harmless and will
> not break anything. It would be great to see it in 8 final! :-)
David Kamholz
1. Rename spi_return_next to return_next.
2. Add a new test for return_next.
3. Update the expected output.
4. Update the documentation.
Abhijit Menon-Sen
>
> > The second issue is where plperl returns a large result set.
I have attached the following seven patches to address this problem:
1. Trivial. Replaces some errant spaces with tabs.
2. Trivial. Fixes the spelling of Jan's name, and gets rid of many
inane, useless, annoying, and often misleading comments. Here's
a sample: "plperl_init_all() - Initialize all".
(I have tried to add some useful comments here and there, and will
continue to do so now and again.)
3. Trivial. Splits up some long lines.
4. Converts SRFs in PL/Perl to use a Tuplestore and SFRM_Materialize
to return the result set, based on the PL/PgSQL model.
There are two major consequences: result sets will spill to disk when
they can no longer fit in work_mem; and "select foo_srf()" no longer
works. (I didn't lose sleep over the latter, since that form is not
valid in PL/PgSQL, and it's not documented in PL/Perl.)
5. Trivial, but important. Fixes use of "undef" instead of undef. This
would cause empty functions to fail in bizarre ways. I suspect that
there's still another (old) bug here. I'll investigate further.
6. Moves the majority of (4) out into a new plperl_return_next()
function, to make it possible to expose the functionality to
Perl; cleans up some of the code besides.
7. Add an spi_return_next function for use in Perl code.
If you want to apply the patches and try them out, 8-composite.diff is
what you should use. (Note: my patches depend upon Andrew's use-strict
and %_SHARED patches being applied.)
Here's something to try:
create or replace function foo() returns setof record as $$
$i = 0;
for ("World", "PostgreSQL", "PL/Perl") {
spi_return_next({f1=>++$i, f2=>'Hello', f3=>$_});
}
return;
$$ language plperl;
select * from foo() as (f1 integer, f2 text, f3 text);
(Many thanks to Andrews Dunstan and Supernews for their help.)
Abhijit Menon-Sen
for testing PLs and contrib_regression for testing contrib, instead of
overwriting the core system's regression database as formerly done.
Andrew Dunstan
which is neither needed by nor related to that header. Remove the bogus
inclusion and instead include the header in those C files that actually
need it. Also fix unnecessary inclusions and bad inclusion order in
tsearch2 files.
only one argument. (Per recent discussion, the option to accept multiple
arguments is pretty useless for user-defined types, and would be a likely
source of security holes if it was used.) Simplify call sites of
output/send functions to not bother passing more than one argument.
change saves a great deal of space in pg_proc and its primary index,
and it eliminates the former requirement that INDEX_MAX_KEYS and
FUNC_MAX_ARGS have the same value. INDEX_MAX_KEYS is still embedded
in the on-disk representation (because it affects index tuple header
size), but FUNC_MAX_ARGS is not. I believe it would now be possible
to increase FUNC_MAX_ARGS at little cost, but haven't experimented yet.
There are still a lot of vestigial references to FUNC_MAX_ARGS, which
I will clean up in a separate pass. However, getting rid of it
altogether would require changing the FunctionCallInfoData struct,
and I'm not sure I want to buy into that.
data returned from Perl. Consolidate multiple bits of code to convert
a Perl hash to a tuple, and drive the conversion off the keys present
in the hash rather than the tuple column names, so we detect error if
the hash contains keys it shouldn't. (This means keys not in the hash
will silently default to NULL, which seems ok to me.) Fix a bunch of
reference-count leaks too.
operations are now run as subtransactions, so that errors in them
can be reported as ordinary Perl or Tcl errors and caught by the
normal error handling convention of those languages. Also do some
minor code cleanup in pltcl.c: extract a large chunk of duplicated
code in pltcl_SPI_execute and pltcl_SPI_execute_plan into a shared
subroutine.
rather than longjmp'ing clear out of Perl and thereby leaving Perl in
a broken state. Also some minor prettification of error messages.
Still need to do something with spi_exec_query() error handling.
for the languages even when not installed in a standard directory.
pltcl may need this treatment as well, but we don't have the right path
conveniently available, so I'll leave it alone as long as there aren't
actual reports of trouble.
may expand the Perl stack, therefore we must SPAGAIN to reload the local
stack pointer after calling it. Also a couple other marginal readability
improvements.
some of the bugs exposed thereby. The remaining 'might be used uninitialized'
warnings look like live bugs, but I am not familiar enough with Perl/C hacking
to tell how to fix them.
1. Two minor cleanups:
- We don't need to call hv_exists+hv_fetch; we should just check the
return value of hv_fetch.
- newSVpv("undef",0) is the string "undef", not a real undef.
2. This should fix the bug Andrew Dunstan described in a recent -hackers
post. It replaces three bogus "eval_pv(key, 0)" calls with newSVpv,
and eliminates another redundant hv_exists+hv_fetch pair.
3. plperl_build_tuple_argument builds up a string of Perl code to create
a hash representing the tuple. This patch creates the hash directly.
4. Another minor cleanup: replace a couple of av_store()s with av_push.
5. Analogous to #3 for plperl_trigger_build_args. This patch removes the
static sv_add_tuple_value function, which does much the same as two
other utility functions defined later, and merges the functionality
into plperl_hash_from_tuple.
I have tested the patches to the best of my limited ability, but I would
appreciate it very much if someone else could review and test them too.
(Thanks to Andrew and David Fetter for their help with some testing.)
Abhijit Menon-Sen
mode see a fresh snapshot for each command in the function, rather than
using the latest interactive command's snapshot. Also, suppress fresh
snapshots as well as CommandCounterIncrement inside STABLE and IMMUTABLE
functions, instead using the snapshot taken for the most closely nested
regular query. (This behavior is only sane for read-only functions, so
the patch also enforces that such functions contain only SELECT commands.)
As per my proposal of 6-Sep-2004; I note that I floated essentially the
same proposal on 19-Jun-2002, but that discussion tailed off without any
action. Since 8.0 seems like the right place to be taking possibly
nontrivial backwards compatibility hits, let's get it done now.
Create a shared function to convert a SPI error code into a string
(replacing near-duplicate code in several PLs), and use it anywhere
that a SPI function call error is reported.
possible to trap an error inside a function rather than letting it
propagate out to PostgresMain. You still have to use AbortCurrentTransaction
to clean up, but at least the error handling itself will cooperate.
* perl_useshrplib gets set to "yes" and not to "true". I assume it's set
to "true" on unix, so I left both.
* Need to translate backslashes into slashes
* The linker config coming out of perl was for MSVC and not for mingw
Magnus Hagander
currently unapplied regarding spi_internal.c, makes some additional
fixes relating to return types, and also contains the fix for
preventing the use of insecure versions of Safe.pm.
There is one remaing return case that does not appear to work, namely
return of a composite directly in a select, i.e. if foo returns some
composite type, 'select * from foo()' works but 'select foo()' doesn't.
We will either fix that or document it as a limitation.
The function plperl_func_handler is a mess - I will try to get it
cleaned up (and split up) in a subsequent patch, time permitting.
Also, reiterating previous advice - this changes slightly the API for
spi_exec_query - the returned object has either 2 or 3 members: 'status'
(string) and 'proceesed' (int,- number of rows) and, if rows are
returned, 'rows' (array of tuple hashes).
Andrew Dunstan
plperlNG. Review and minor cleanup/improvements by Joe Conway.
Summary of new functionality:
- Shared data space and namespace. There is a new global variable %_SHARED
that functions can use to store and save data between invocations of a
function, or between different functions. Also, all trusted plperl function
now share a common Safe container (this is an optimization, also), which
they can use for storing non-lexical variables, functions, etc.
- Triggers are now supported
- Records can now be returned (as a hash reference)
- Sets of records can now be returned (as a reference to an array of hash
references).
- New function spi_exec_query() provided for performing db functions or
getting data from db.
- Optimization for counting hash keys (Abhijit Menon-Sen)
- Allow return of 'record' and 'setof record'
of a composite type to get that type's OID as their second parameter,
in place of typelem which is useless. The actual changes are mostly
centralized in getTypeInputInfo and siblings, but I had to fix a few
places that were fetching pg_type.typelem for themselves instead of
using the lsyscache.c routines. Also, I renamed all the related variables
from 'typelem' to 'typioparam' to discourage people from assuming that
they necessarily contain array element types.
results with tuples as ordinary varlena Datums. This commit does not
in itself do much for us, except eliminate the horrid memory leak
associated with evaluation of whole-row variables. However, it lays the
groundwork for allowing composite types as table columns, and perhaps
some other useful features as well. Per my proposal of a few days ago.
parameters to be declared with names. pg_proc has a column to store
names, and CREATE FUNCTION can insert data into it, but that's all as
yet. I need to do more work on the pg_dump and plpgsql portions of the
patch before committing those, but I thought I'd get the bulky changes
in before the tree drifts under me.
initdb forced due to pg_proc change.
with OPAQUE. CREATE LANGUAGE, CREATE TRIGGER, and CREATE TYPE will all
accept references to functions declared with OPAQUE --- but they will
issue a NOTICE, and will modify the function entries in pg_proc to have
the preferred type-safe argument or result types instead of OPAQUE.
Per recent pghackers discussions.
with OPAQUE, as per recent pghackers discussion. I still want to do some
more work on the 'cstring' pseudo-type, but I'm going to commit the bulk
of the changes now before the tree starts shifting under me ...
HeapTupleHeaderData in setter and getter macros called
HeapTupleHeaderGetXmin, HeapTupleHeaderSetXmin etc.
It also introduces a "virtual" field xvac by defining
HeapTupleHeaderGetXvac and HeapTupleHeaderSetXvac. Xvac is used by
VACUUM, in fact it is stored in t_cmin.
Manfred Koizar
As proof of concept, provide an alternate implementation based on POSIX
semaphores. Also push the SysV shared-memory implementation into a
separate file so that it can be replaced conveniently.
now just below FATAL in server_min_messages. Added more text to
highlight ordering difference between it and client_min_messages.
---------------------------------------------------------------------------
REALLYFATAL => PANIC
STOP => PANIC
New INFO level the prints to client by default
New LOG level the prints to server log by default
Cause VACUUM information to print only to the client
NOTICE => INFO where purely information messages are sent
DEBUG => LOG for purely server status messages
DEBUG removed, kept as backward compatible
DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1 added
DebugLvl removed in favor of new DEBUG[1-5] symbols
New server_min_messages GUC parameter with values:
DEBUG[5-1], INFO, NOTICE, ERROR, LOG, FATAL, PANIC
New client_min_messages GUC parameter with values:
DEBUG[5-1], LOG, INFO, NOTICE, ERROR, FATAL, PANIC
Server startup now logged with LOG instead of DEBUG
Remove debug_level GUC parameter
elog() numbers now start at 10
Add test to print error message if older elog() values are passed to elog()
Bootstrap mode now has a -d that requires an argument, like postmaster
The attached patch enables plperl to build under Cygwin. It is
basically yet another BE_DLLLIBS patch with a perl MakeMaker twist. I
tried the patch under Red Hat 7.1 Linux too and I did not observe any
ill effects.
Jason Tishler
lookup info in the relcache for index access method support functions.
This makes a huge difference for dynamically loaded support functions,
and should save a few cycles even for built-in ones. Also tweak dfmgr.c
so that load_external_function is called only once, not twice, when
doing fmgr_info for a dynamically loaded function. All per performance
gripe from Teodor Sigaev, 5-Oct-01.
under libdir, for a cleaner separation in the installation layout
and compatibility with binary packaging standards. Point backend's
default search location there. The contrib modules are also
installed in the said location, giving them the benefit of the
default search path as well. No changes in user interface
nevertheless.
in interfaces/perl5 a brief while ago.
Also, since building PL/Perl without a shared libperl actually works on
some platforms we can enable it there to get some development happening.
I've only checked off linux right now, but others should be added in the
future.
modifiable repositories, I have a clean untrusted plperl patch to offer
you :)
Highlights:
* There's one perl interpreter used for both trusted and untrusted
procedures. I do think its unnecessary to keep two perl
interpreters around. If someone can break out from trusted "Safe" perl
mode, well, they can do what they want already. If someone disagrees, I
can change this.
* Opcode is not statically loaded anymore. Instead, we load Dynaloader,
which then can grab Opcode (and anything else you can 'use') on its own.
* Checked to work on FreeBSD 4.3 + perl 5.5.3 , OpenBSD 2.8 + perl5.6.1,
RedHat 6.2 + perl 5.5.3
* Uses ExtUtils::Embed to find what options are necessary to link with
perl shared libraries
* createlang is also updated, it can create untrusted perl using 'plperlu'
* Example script (assuming you have Mail::Sendmail installed):
create function foo() returns text as '
use Mail::Sendmail;
%mail = ( To => q(you@yourname.com),
From => q(me@here.com),
Message => "This is a very short message"
);
sendmail(%mail) or die $Mail::Sendmail::error;
return "OK. Log says:\n", $Mail::Sendmail::log;
' language 'plperlu';
Alex Pilosov
in plpgsql: they fail for datatypes that have old-style I/O functions
due to caching FmgrInfo structs with wrong fn_mcxt lifetime.
Although the plpython fix seems straightforward, I can't check it here
since I don't have Python installed --- would someone check it?
> > enable the :bash_math opcodes. Currently plperl.c only
> > enables the :default opcodes. This leave out about five of six
> > math functions including sqrt().
Travis Bauer
in pghackers list. Support for oldstyle internal functions is gone
(no longer needed, since conversion is complete) and pg_language entry
'internal' now implies newstyle call convention. pg_language entry
'newC' is gone; both old and newstyle dynamically loaded C functions
are now called language 'C'. A newstyle function must be identified
by an associated info routine. See src/backend/utils/fmgr/README.
maintained for each cache entry. A cache entry will not be freed until
the matching ReleaseSysCache call has been executed. This eliminates
worries about cache entries getting dropped while still in use. See
my posting to pg-hackers of even date for more info.
DESTDIR=/else/where' and prepends the value of DESTDIR to the full
installation paths (e.g., /else/where/usr/local/pgsql/bin). This allows
users to install the package into a location different from the one that
was configured and hard-coded into various scripts, e.g., for creating
binary packages.
DESTDIR is in many cases preferrable over `make install
prefix=/else/where' because
a) `prefix' affects the path that is hard-coded into the files, which can
lead to a `make install prefix=xxx' (as done by the regression test
driver) corrupting the files in the source tree with wrong paths.
b) it doesn't work at all if a directory was overridden to not depend on
`prefix', e.g., --sysconfdir=/etc.
(Updating the regression test driver to use DESTDIR is a separate
undertaking.)
See also autoconf@gnu.org, From: Akim Demaille <akim@epita.fr>, Date: 08
Sep 2000 12:48:59 +0200, Message-ID:
<mv4em2vb1lw.fsf@nostromo.lrde.epita.fr>, Subject: Re: HTML format
documentation.
7.0.2 release. Sorry, if that's fixed ages ago - I don't track
development versions of PostgreSQL.
Patch is just a little bit tested (some valid functions created and
successfully run as well as some erroneous ones created and emitted proper
error messages when used).
My platform is FreeBSD 5.0-CURRENT (with perl 5.6.0 provided in the
base system).
Alex Kapranoff
standard targets and behaviour. Replaced Makefile.in's with
Makefile's and declared the respective variables in Makefile.global.
maintainer-clean target now available at top level, although it does
not work in the backend tree yet.
Cleanup pass over Makefile.shlib, renamed some targets and variables.
The shared library symlink tests are now done by make, not the shell.
ecpg: Remove one warning in sloppy flex output.
PL/Perl and Perl interface: the MakeMaker documentation is confusing,
the realclean target *does* "delete derived files", but it also
uninstalls them. Don't use that.
The submake targets in the various bin directories that update libpq
should `make all', not `make libpq.a'. That is a) unportable, and
b) doesn't build the shared library.
that now functions as a wrapper around the MakeMaker stuff. It might
even behave sensically when we have separate build dirs. Same for plperl,
which of course still doesn't work very well. Made sure that plperl
respects the choice of --libdir.
Added --with-python to automatically build and install the Python interface.
Works similarly to the Perl5 stuff.
Moved the burden of the distclean targets lower down into the source tree.
Eventually, each make file should have its own.
Added automatic remaking of makefiles and configure. Currently only for the
top-level because of a bug(?) in Autoconf. Use GNU `missing' to work around
missing autoconf and aclocal. Start factoring out macros into their own
config/*.m4 files to increase readability and organization.
inputs have been converted to newstyle. This should go a long way towards
fixing our portability problems with platforms where char and short
parameters are passed differently from int-width parameters. Still
more to do for the Alpha port however.
key call sites are changed, but most called functions are still oldstyle.
An exception is that the PL managers are updated (so, for example, NULL
handling now behaves as expected in plperl and plpgsql functions).
NOTE initdb is forced due to added column in pg_proc.
--with-includes) to makefiles for pltcl and plperl, so that these
switches will be used even though we do not want other top-level
CFLAGS. Ain't it fun trying to support multiple-compiler platforms?
this is an old patch which I have already submitted and never seen
in the sources. It corrects the datatype oids used in some iterator
functions. This bug has been reported to me by many other people.
contrib-datetime.patch
some code contributed by Reiner Dassing <dassing@wettzell.ifag.de>
contrib-makefiles.patch
fixes all my contrib makefiles which don't work with some compilers,
as reported to me by another user.
contrib-miscutil.patch
an old patch for one of my old contribs.
contrib-string.patch
a small change to the c-like text output functions. Now the '{'
is escaped only at the beginning of the string to distinguish it
from arrays, and the '}' is no more escaped.
elog-lineno.patch
adds the current lineno of CopyFrom to elog messages. This is very
useful when you load a 1 million tuples table from an external file
and there is a bad value somehere. Currently you get an error message
but you can't know where is the bad data. The patch uses a variable
which was declared static in copy.c. The variable is now exported
and initialized to 0. It is always cleared at the end of the copy
or at the first elog message or when the copy is canceled.
I know this is very ugly but I can't find any better way of knowing
where the copy fails and I have this problem quite often.
plperl-makefile.patch
fixes a typo in a makefile, but the error must be elsewhere because
it is a file generated automatically. Please have a look.
tprintf-timestamp.patch
restores the original 2-digit year format, assuming that the two
century digits don't carry much information and that '000202' is
easier to read than 20000202. Being only a log file it shouldn't
break anything.
Please apply the patches before the next scheduled code freeze.
I also noticed that some of the contribs don't compile correcly. Should we
ask people to fix their code or rename their makefiles so that they are
ignored by the top makefile?
--
Massimo Dal Zotto
3 new files and two patches for the plperl subdir.
These changes add the ability for plperl functions
to call 'elog'. It also sets up the frame work to
allow me to add access to the SPI functions.
--
Mark Hollomon
that kept me from making perl secure.
Attached is uuencoded tarball to add PL/perl
to postgresql.
Things I know don't work.
-- triggers
-- SPI
The README file has a _VERY_ short tutorial.
Mark Hollomon
Tom Lane
Bruce Momjian
Andrew Dunstan
Neil Conway
Peter Eisentraut
Joe Conway
PostgreSQL Daemon