rewritten and the protocol is changed, but most elog calls are still
elog calls. Also, we need to contemplate mechanisms for controlling
all this functionality --- eg, how much stuff should appear in the
postmaster log? And what API should libpq expose for it?
have length words. COPY OUT reimplemented per new protocol: it doesn't
need \. anymore, thank goodness. COPY BINARY to/from frontend works,
at least as far as the backend is concerned --- libpq's PQgetline API
is not up to snuff, and will have to be replaced with something that is
null-safe. libpq uses message length words for performance improvement
(no cycles wasted rescanning long messages), but not yet for error
recovery.
Output \r\n termination on Win32.
Disallow literal carriage return as a data value,
backslash-carriage-return and \r still allowed.
Doc changes already committed.
(materialization into a tuple store) discussed on pgsql-hackers earlier.
I've updated the documentation and the regression tests.
Notes on the implementation:
- I needed to change the tuple store API slightly -- it assumes that it
won't be used to hold data across transaction boundaries, so the temp
files that it uses for on-disk storage are automatically reclaimed at
end-of-transaction. I added a flag to tuplestore_begin_heap() to control
this behavior. Is changing the tuple store API in this fashion OK?
- in order to store executor results in a tuple store, I added a new
CommandDest. This works well for the most part, with one exception: the
current DestFunction API doesn't provide enough information to allow the
Executor to store results into an arbitrary tuple store (where the
particular tuple store to use is chosen by the call site of
ExecutorRun). To workaround this, I've temporarily hacked up a solution
that works, but is not ideal: since the receiveTuple DestFunction is
passed the portal name, we can use that to lookup the Portal data
structure for the cursor and then use that to get at the tuple store the
Portal is using. This unnecessarily ties the Portal code with the
tupleReceiver code, but it works...
The proper fix for this is probably to change the DestFunction API --
Tom suggested passing the full QueryDesc to the receiveTuple function.
In that case, callers of ExecutorRun could "subclass" QueryDesc to add
any additional fields that their particular CommandDest needed to get
access to. This approach would work, but I'd like to think about it for
a little bit longer before deciding which route to go. In the mean time,
the code works fine, so I don't think a fix is urgent.
- (semi-related) I added a NO SCROLL keyword to DECLARE CURSOR, and
adjusted the behavior of SCROLL in accordance with the discussion on
-hackers.
- (unrelated) Cleaned up some SGML markup in sql.sgml, copy.sgml
Neil Conway
startup, not in the parser; this allows ALTER DOMAIN to work correctly
with domain constraint operations stored in rules. Rod Taylor;
code review by Tom Lane.
a per-query memory context created by CreateExecutorState --- and destroyed
by FreeExecutorState. This provides a final solution to the longstanding
problem of memory leaked by various ExecEndNode calls.
execution state trees, and ExecEvalExpr takes an expression state tree
not an expression plan tree. The plan tree is now read-only as far as
the executor is concerned. Next step is to begin actually exploiting
this property.
so that all executable expression nodes inherit from a common supertype
Expr. This is somewhat of an exercise in code purity rather than any
real functional advance, but getting rid of the extra Oper or Func node
formerly used in each operator or function call should provide at least
a little space and speed improvement.
initdb forced by changes in stored-rules representation.
and eliminate its manual pfree() calls. This solves the encoding-conversion
bug recently reported, and should be faster and more robust than the
original coding anyway. For example, we are no longer at risk if
datatype output routines leak memory or choose to return a constant string.
sublink results and COPY's domain constraint checking. A Const that
isn't really constant is just a Bad Idea(tm). Remove hacks in
parse_coerce and other places that were needed because of the former
klugery.
-hackers a couple days ago.
Notes/caveats:
- added regression tests for the new functionality, all
regression tests pass on my machine
- added pg_dump support
- updated PL/PgSQL to support per-statement triggers; didn't
look at the other procedural languages.
- there's (even) more code duplication in trigger.c than there
was previously. Any suggestions on how to refactor the
ExecXXXTriggers() functions to reuse more code would be
welcome -- I took a brief look at it, but couldn't see an
easy way to do it (there are several subtly-different
versions of the code in question)
- updated the documentation. I also took the liberty of
removing a big chunk of duplicated syntax documentation in
the Programmer's Guide on triggers, and moving that
information to the CREATE TRIGGER reference page.
- I also included some spelling fixes and similar small
cleanups I noticed while making the changes. If you'd like
me to split those into a separate patch, let me know.
Neil Conway
query that uses it. This ensures that triggers will be applied consistently
throughout a query even if someone commits changes to the relation's
pg_class.reltriggers field meanwhile. Per crash report from Laurette Cisneros.
While at it, simplify memory management in relcache.c, which no longer
needs the old hack to try to keep trigger info in the same place over
a relcache entry rebuild. (Should try to fix rd_att and rewrite-rule
access similarly, someday.) And make RelationBuildTriggers simpler and
more robust by making it build the trigdesc in working memory and then
CopyTriggerDesc() into cache memory.
(overlaying low byte of page size) and add HEAP_HASOID bit to t_infomask,
per earlier discussion. Simplify scheme for overlaying fields in tuple
header (no need for cmax to live in more than one place). Don't try to
clear infomask status bits in tqual.c --- not safe to do it there. Don't
try to force output table of a SELECT INTO to have OIDs, either. Get rid
of unnecessarily complex three-state scheme for TupleDesc.tdhasoids, which
has already caused one recent failure. Improve documentation.
value '-2' is used to indicate a variable-width type whose width is
computed as strlen(datum)+1. Everything that looks at typlen is updated
except for array support, which Joe Conway is working on; at the moment
it wouldn't work to try to create an array of cstring.
with OPAQUE, as per recent pghackers discussion. I still want to do some
more work on the 'cstring' pseudo-type, but I'm going to commit the bulk
of the changes now before the tree starts shifting under me ...
latent wrong-struct-type bugs and makes the coding style more uniform,
since the majority of places working with lists of column names were
already using Strings not Idents. While at it, remove vestigial
support for Stream node type, and otherwise-unreferenced nodes.h entries
for T_TupleCount and T_BaseNode.
NB: full recompile is recommended due to changes of Node type numbers.
This shouldn't force an initdb though.
> There's no longer a separate call to heap_storage_create in that routine
> --- the right place to make the test is now in the storage_create
> boolean parameter being passed to heap_create. A simple change, but
> it passeth patch's understanding ...
Thanks.
Attached is a patch against cvs tip as of 8:30 PM PST or so. Turned out
that even after fixing the failed hunks, there was a new spot in
bufmgr.c which needed to be fixed (related to temp relations;
RelationUpdateNumberOfBlocks). But thankfully the regression test code
caught it :-)
Joe Conway
code review by Tom Lane. Remaining issues: functions that take or
return tuple types are likely to break if one drops (or adds!)
a column in the table defining the type. Need to think about what
to do here.
Along the way: some code review for recent COPY changes; mark system
columns attnotnull = true where appropriate, per discussion a month ago.
The attached patch completes the following TODO item:
* Generate failure on short COPY lines rather than pad NULLs
I also restructed a lot of the existing COPY code, did some code
review on the column list patch sent in by Brent Verner a little
while ago, and added some regression tests. I also added an
explicit check (and resultant error) for extra data before
the end-of-line.
Neil Conway
bitmap, if present).
Per Tom Lane's suggestion the information whether a tuple has an oid
or not is carried in the tuple descriptor. For debugging reasons
tdhasoid is of type char, not bool. There are predefined values for
WITHOID, WITHOUTOID and UNDEFOID.
This patch has been generated against a cvs snapshot from last week
and I don't expect it to apply cleanly to current sources. While I
post it here for public review, I'm working on a new version against a
current snapshot. (There's been heavy activity recently; hope to
catch up some day ...)
This is a long patch; if it is too hard to swallow, I can provide it
in smaller pieces:
Part 1: Accessor macros
Part 2: tdhasoid in TupDesc
Part 3: Regression test
Part 4: Parameter withoid to heap_addheader
Part 5: Eliminate t_oid from HeapTupleHeader
Part 2 is the most hairy part because of changes in the executor and
even in the parser; the other parts are straightforward.
Up to part 4 the patched postmaster stays binary compatible to
databases created with an unpatched version. Part 5 is small (100
lines) and finally breaks compatibility.
Manfred Koizar
COPY x (a,d,c,b) from stdin;
COPY x (a,c) to stdout;
as well as the corresponding changes to pg_dump to use the new
functionality. This functionality is not available when using
the BINARY option. If a column is not specified in the COPY FROM
statement, its default values will be used.
In addition to this functionality, I tweaked a couple of the
error messages emitted by the new COPY <options> checks.
Brent Verner
in snapshots, per my proposal of a few days ago. Also, tweak heapam.c
routines (heap_insert, heap_update, heap_delete, heap_mark4update) to
be passed the command ID to use, instead of doing GetCurrentCommandID.
For catalog updates they'll still get passed current command ID, but
for updates generated from the main executor they'll get passed the
command ID saved in the snapshot the query is using. This should fix
some corner cases associated with functions and triggers that advance
current command ID while an outer query is still in progress.
yesterday's proposal to pghackers. Also remove unnecessary parameters
to heap_beginscan, heap_rescan. I modified pg_proc.h to reflect the
new numbers of parameters for the AM interface routines, but did not
force an initdb because nothing actually looks at those fields.
messages more uniform and internationalizable: the global array
aclcheck_error_strings[] is gone in favor of a subroutine
aclcheck_error(). Partial implementation of namespace-related
permission checks --- not all done yet.
objects to be privilege-checked. Some change in their APIs would be
necessary no matter what in the schema environment, and simply getting
rid of the name-based interface entirely seems like the best way.
o Change all current CVS messages of NOTICE to WARNING. We were going
to do this just before 7.3 beta but it has to be done now, as you will
see below.
o Change current INFO messages that should be controlled by
client_min_messages to NOTICE.
o Force remaining INFO messages, like from EXPLAIN, VACUUM VERBOSE, etc.
to always go to the client.
o Remove INFO from the client_min_messages options and add NOTICE.
Seems we do need three non-ERROR elog levels to handle the various
behaviors we need for these messages.
Regression passed.
(backslash-r, backslash-n) for protection against newline-conversion
munging. In future we will also tweak COPY FROM, but this part of the
change should be backwards-compatible. Per pghackers discussion.
Also, update COPY reference page to describe the backslash conversions
more completely and accurately.
to use and significantly faster. This tweak saves 25% (!) of the runtime
of COPY IN in a test with 8000-character lines. I wouldn't normally
commit a performance improvement this late in the cycle, but 25% got
my attention...
default, but OIDS are removed from many system catalogs that don't need them.
Some interesting side effects: TOAST pointers are 20 bytes not 32 now;
pg_description has a three-column key instead of one.
Bugs fixed in passing: BINARY cursors work again; pg_class.relhaspkey
has some usefulness; pg_dump dumps comments on indexes, rules, and
triggers in a valid order.
initdb forced.
report on old-style functions invoked by RI triggers. We had a number of
other places that were being sloppy about which memory context FmgrInfo
subsidiary data will be allocated in. Turns out none of them actually
cause a problem in 7.1, but this is for arcane reasons such as the fact
that old-style triggers aren't supported anyway. To avoid getting burnt
later, I've restructured the trigger support so that we don't keep trigger
FmgrInfo structs in relcache memory. Some other related cleanups too:
it's not really necessary to call fmgr_info at all while setting up
the index support info in relcache entries, because those ScanKeyEntry
structs are never used to invoke the functions. This should speed up
relcache initialization a tiny bit.
I'm not sure that it's really necessary to save insert events when there
are only after update or delete triggers, but certainly it's wrong for
COPY to behave differently from an INSERT query.
allocated by plan nodes are not leaked at end of query. This doesn't
really matter for normal queries, but it sure does for queries invoked
repetitively inside SQL functions. Clean up some other grotty code
associated with tupdescs, and fix a few other memory leaks exposed by
tests with simple SQL functions.
are treated more like 'cancel' interrupts: the signal handler sets a
flag that is examined at well-defined spots, rather than trying to cope
with an interrupt that might happen anywhere. See pghackers discussion
of 1/12/01.
sequences. This is done by disabling multi-byte awareness when it's
not necessary. This is kind of a workaround, not a perfect solution.
However, there is no ideal way to parse broken multi-byte character
sequences. So I guess this is the best way what we could do right
now...
1. Distinguish cases where a Datum representing a tuple datatype is an OID
from cases where it is a pointer to TupleTableSlot, and make sure we use
the right typlen in each case.
2. Make fetchatt() and related code support 8-byte by-value datatypes on
machines where Datum is 8 bytes. Centralize knowledge of the available
by-value datatype sizes in two macros in tupmacs.h, so that this will be
easier if we ever have to do it again.
maintained for each cache entry. A cache entry will not be freed until
the matching ReleaseSysCache call has been executed. This eliminates
worries about cache entries getting dropped while still in use. See
my posting to pg-hackers of even date for more info.
joins, and clean things up a good deal at the same time. Append plan node
no longer hacks on rangetable at runtime --- instead, all child tables are
given their own RT entries during planning. Concept of multiple target
tables pushed up into execMain, replacing bug-prone implementation within
nodeAppend. Planner now supports generating Append plans for inheritance
sets either at the top of the plan (the old way) or at the bottom. Expanding
at the bottom is appropriate for tables used as sources, since they may
appear inside an outer join; but we must still expand at the top when the
target of an UPDATE or DELETE is an inheritance set, because we actually need
a different targetlist and junkfilter for each target table in that case.
Fortunately a target table can't be inside an outer join... Bizarre mutual
recursion between union_planner and prepunion.c is gone --- in fact,
union_planner doesn't really have much to do with union queries anymore,
so I renamed it grouping_planner.
user is now defined in terms of the user id, the user name is only computed
upon request (for display purposes). This is kind of the opposite of the
previous state, which would maintain the user name and compute the user id
for permission checks.
Besides perhaps saving a few cycles (integer vs string), this now creates a
single point of attack for changing the user id during a connection, for
purposes of "setuid" functions, etc.
right circumstances a hash join executed as a DECLARE CURSOR/FETCH
query would crash the backend. Problem as seen in current sources was
that the hash tables were stored in a context that was a child of
TransactionCommandContext, which got zapped at completion of the FETCH
command --- but cursor cleanup executed at COMMIT expected the tables
to still be valid. I haven't chased down the details as seen in 7.0.*
but I'm sure it's the same general problem.
thing when there are multiple result relations. Formerly, during
something like 'UPDATE foo*', foo's constraints and *only* foo's
constraints would be applied to all foo's children. Wrong-o ...
pass-by-ref data types --- eg, an index on lower(textfield) --- no longer
leak memory during index creation or update. Clean up a lot of redundant
code ... did you know that copy, vacuum, truncate, reindex, extend index,
and bootstrap each basically duplicated the main executor's logic for
extracting information about an index and preparing index entries?
Functional indexes should be a little faster now too, due to removal
of repeated function lookups.
CREATE INDEX 'opt_type' clause is deimplemented by these changes,
but I haven't removed it from the parser yet (need to merge with
Thomas' latest change set first).
memory contexts. Currently, only leaks in expressions executed as
quals or projections are handled. Clean up some old dead cruft in
executor while at it --- unused fields in state nodes, that sort of thing.
discussion of 5/19/00). pg_index is now searched for indexes of a
relation using an indexscan. Moreover, this is done once and cached
in the relcache entry for the relation, in the form of a list of OIDs
for the indexes. This list is used by the parser and executor to drive
lookups in the pg_index syscache when they want to know the properties
of the indexes. Net result: index information will be fully cached
for repetitive operations such as inserts.
we'll get there one day.
Use `cat' to create aclocal.m4, not `aclocal'. Some people don't
have automake installed.
Only run the autoconf rule in the top-level GNUmakefile if the
invoker specified `make configure', don't run it automatically
because of CVS timestamp skew.
inputs have been converted to newstyle. This should go a long way towards
fixing our portability problems with platforms where char and short
parameters are passed differently from int-width parameters. Still
more to do for the Alpha port however.
key call sites are changed, but most called functions are still oldstyle.
An exception is that the PL managers are updated (so, for example, NULL
handling now behaves as expected in plperl and plpgsql functions).
NOTE initdb is forced due to added column in pg_proc.
RowExclusive (my fault). Also, install a check to prevent people
from trying COPY BINARY to stdout/from stdin. No way that will
work unless we redesign the frontend COPY protocol ... which is
not worth the trouble in the near future ...
this is an old patch which I have already submitted and never seen
in the sources. It corrects the datatype oids used in some iterator
functions. This bug has been reported to me by many other people.
contrib-datetime.patch
some code contributed by Reiner Dassing <dassing@wettzell.ifag.de>
contrib-makefiles.patch
fixes all my contrib makefiles which don't work with some compilers,
as reported to me by another user.
contrib-miscutil.patch
an old patch for one of my old contribs.
contrib-string.patch
a small change to the c-like text output functions. Now the '{'
is escaped only at the beginning of the string to distinguish it
from arrays, and the '}' is no more escaped.
elog-lineno.patch
adds the current lineno of CopyFrom to elog messages. This is very
useful when you load a 1 million tuples table from an external file
and there is a bad value somehere. Currently you get an error message
but you can't know where is the bad data. The patch uses a variable
which was declared static in copy.c. The variable is now exported
and initialized to 0. It is always cleared at the end of the copy
or at the first elog message or when the copy is canceled.
I know this is very ugly but I can't find any better way of knowing
where the copy fails and I have this problem quite often.
plperl-makefile.patch
fixes a typo in a makefile, but the error must be elsewhere because
it is a file generated automatically. Please have a look.
tprintf-timestamp.patch
restores the original 2-digit year format, assuming that the two
century digits don't carry much information and that '000202' is
easier to read than 20000202. Being only a log file it shouldn't
break anything.
Please apply the patches before the next scheduled code freeze.
I also noticed that some of the contribs don't compile correcly. Should we
ask people to fix their code or rename their makefiles so that they are
ignored by the top makefile?
--
Massimo Dal Zotto
from a constraint condition does not violate the constraint (cf. discussion
on pghackers 12/9/99). Implemented by adding a parameter to ExecQual,
specifying whether to return TRUE or FALSE when the qual result is
really NULL in three-valued boolean logic. Currently, ExecRelCheck is
the only caller that asks for TRUE, but if we find any other places that
have the wrong response to NULL, it'll be easy to fix them.
Tom Lane
Bruce Momjian
Peter Eisentraut
Tatsuo Ishii
Hiroshi Inoue