when we need to move to a new page; as long as we can insert the new
tuple on the same page as before, we only need LockBuffer and not the
expensive stuff. Also, twiddle bufmgr interfaces to avoid redundant
lseeks in RelationGetBufferForTuple and BufferAlloc. Successive inserts
now require one lseek per page added, rather than one per tuple with
several additional ones at each page boundary as happened before.
Lock contention when multiple backends are inserting in same table
is also greatly reduced.
a separate statement (though it can still be invoked as part of VACUUM, too).
pg_statistic redesigned to be more flexible about what statistics are
stored. ANALYZE now collects a list of several of the most common values,
not just one, plus a histogram (not just the min and max values). Random
sampling is used to make the process reasonably fast even on very large
tables. The number of values and histogram bins collected is now
user-settable via an ALTER TABLE command.
There is more still to do; the new stats are not being used everywhere
they could be in the planner. But the remaining changes for this project
should be localized, and the behavior is already better than before.
A not-very-related change is that sorting now makes use of btree comparison
routines if it can find one, rather than invoking '<' twice.
give consistent results for all datatypes. Types float4, float8, and
numeric were broken for NaN values; abstime, timestamp, and interval
were broken for INVALID values; timetz was just plain broken (some
possible pairs of values were neither < nor = nor >). Also clean up
text, bpchar, varchar, and bit/varbit to eliminate duplicate code and
thereby reduce the probability of similar inconsistencies arising in
the future.
O_SYNC, or O_DSYNC (as available on a given platform). Add GUC parameter
to control sync method.
Also, add defense to XLogWrite to prevent it from going nuts if passed
a target write position that's past the end of the buffers so far filled
by XLogInsert.
detect case that next page in log came from an older run than the prior
page. This avoids the necessity to re-zero the log after recovery from
a crash, which is good because we need not risk destroying valuable log
information.
This forces another initdb since yesterday :-(. Need to get that log
reset utility done...
* Store two past checkpoint locations, not just one, in pg_control.
On startup, we fall back to the older checkpoint if the newer one
is unreadable. Also, a physical copy of the newest checkpoint record
is kept in pg_control for possible use in disaster recovery (ie,
complete loss of pg_xlog). Also add a version number for pg_control
itself. Remove archdir from pg_control; it ought to be a GUC
parameter, not a special case (not that it's implemented yet anyway).
* Suppress successive checkpoint records when nothing has been entered
in the WAL log since the last one. This is not so much to avoid I/O
as to make it actually useful to keep track of the last two
checkpoints. If the things are right next to each other then there's
not a lot of redundancy gained...
* Change CRC scheme to a true 64-bit CRC, not a pair of 32-bit CRCs
on alternate bytes. Polynomial borrowed from ECMA DLT1 standard.
* Fix XLOG record length handling so that it will work at BLCKSZ = 32k.
* Change XID allocation to work more like OID allocation. (This is of
dubious necessity, but I think it's a good idea anyway.)
* Fix a number of minor bugs, such as off-by-one logic for XLOG file
wraparound at the 4 gig mark.
* Add documentation and clean up some coding infelicities; move file
format declarations out to include files where planned contrib
utilities can get at them.
* Checkpoint will now occur every CHECKPOINT_SEGMENTS log segments or
every CHECKPOINT_TIMEOUT seconds, whichever comes first. It is also
possible to force a checkpoint by sending SIGUSR1 to the postmaster
(undocumented feature...)
* Defend against kill -9 postmaster by storing shmem block's key and ID
in postmaster.pid lockfile, and checking at startup to ensure that no
processes are still connected to old shmem block (if it still exists).
* Switch backends to accept SIGQUIT rather than SIGUSR1 for emergency
stop, for symmetry with postmaster and xlog utilities. Clean up signal
handling in bootstrap.c so that xlog utilities launched by postmaster
will react to signals better.
* Standalone bootstrap now grabs lockfile in target directory, as added
insurance against running it in parallel with live postmaster.
succeeds or not. Revise rtree page split algorithm to take care about
making a feasible split --- ie, will the incoming tuple actually fit?
Failure to make a feasible split, combined with failure to notice the
failure, account for Jim Stone's recent bug report. I suspect that
hash and gist indices may have the same type of bug, but at least now
we'll get error messages rather than silent failures if so. Also clean
up rtree code to use Datum rather than char* where appropriate.
only if at least N other backends currently have open transactions. This
is not a great deal of intelligence about whether a delay might be
profitable ... but it beats no intelligence at all. Note that the default
COMMIT_DELAY is still zero --- this new code does nothing unless that
setting is changed.
Also, mark ENABLEFSYNC as a system-wide setting. It's no longer safe to
allow that to be set per-backend, since we may be relying on some other
backend's fsync to have synced the WAL log.
waste of cycles on single-CPU machines, and of dubious utility on multi-CPU
machines too.
Tweak s_lock_stuck so that caller can specify timeout interval, and
increase interval before declaring stuck spinlock for buffer locks and XLOG
locks.
On systems that have fdatasync(), use that rather than fsync() to sync WAL
log writes. Ensure that WAL file is entirely allocated during XLogFileInit.
compressed storage works perfectly well. Might as well have a coherent
strategy for applying it, rather than the haphazard store-what-you-get
approach that was in the code before. The strategy I've set up here is
to attempt compression of any compressible index value exceeding
BLCKSZ/16, or about 500 bytes by default.
allocated by plan nodes are not leaked at end of query. This doesn't
really matter for normal queries, but it sure does for queries invoked
repetitively inside SQL functions. Clean up some other grotty code
associated with tupdescs, and fix a few other memory leaks exposed by
tests with simple SQL functions.
and new root page if old root one was splitted but new root page
wasn't created.
New code is protected by FixBTree bool flag setted to FALSE, so
nothing should be affected by this untested approach.
bothering to check the return value --- which meant that in case the
update or delete failed because of a concurrent update, you'd not find
out about it, except by observing later that the transaction produced
the wrong outcome. There are now subroutines simple_heap_update and
simple_heap_delete that should be used anyplace that you're not prepared
to do the full nine yards of coping with concurrent updates. In
practice, that seems to mean absolutely everywhere but the executor,
because *noplace* else was checking.
are treated more like 'cancel' interrupts: the signal handler sets a
flag that is examined at well-defined spots, rather than trying to cope
with an interrupt that might happen anywhere. See pghackers discussion
of 1/12/01.
are now critical sections, so as to ensure die() won't interrupt us while
we are munging shared-memory data structures. Avoid insecure intermediate
states in some code that proc_exit will call, like palloc/pfree. Rename
START/END_CRIT_CODE to START/END_CRIT_SECTION, since that seems to be
what people tend to call them anyway, and make them be called with () like
a function call, in hopes of not confusing pg_indent.
I doubt that this is sufficient to make SIGTERM safe anywhere; there's
just too much code that could get invoked during proc_exit().
1. Support of variable size keys - new algorithm of insertion to tree
(GLI - gist layrered insertion). Previous algorithm was implemented
as described in paper by Joseph M. Hellerstein et.al
"Generalized Search Trees for Database Systems". This (old)
algorithm was not suitable for variable size keys and could be
not effective ( walking up-down ) in case of multiple levels split
Bug fixed:
1. fixed bug in gistPageAddItem - key values were written to disk
uncompressed. This caused failure if decompression function
does real job.
2. NULLs handling - we keep NULLs in tree. Right way is to remove them,
but we don't know how to inform vacuum about index statistics. This is
just cosmetic warning message (like in case with R-Tree),
but I'm not sure how to recognize real problem if we remove NULLs
and suppress this warning as Tom suggested.
3. various memory leaks
This work was done by Teodor Sigaev (teodor@stack.net) and
Oleg Bartunov (oleg@sai.msu.su).
- no more elog(STOP) in StartupXLOG();
- both checkpoint' undo & redo are used to define
oldest on-line log file.
2. Ability to pre-allocate a few log files at checkpoint time
(wal_files option). Off by default.
before calling RelationInvalidateHeapTuple(), which is bad because the
latter needs to look at the tuple data, which is in the shared disk
buffer. If another backend manages to recycle the buffer while this
is going on, we will compute the wrong hashindex for the tuple or
maybe even crash outright. Must hold buffer refcount until afterwards.
(This bug is not in 7.0.*; seems to be have introduced during WAL changes.)
starting a new hashtable search no longer clobbers any other search
active anywhere in the system. Fix RelationCacheInvalidate() so that
it will not crash or go into an infinite loop if invoked recursively,
as for example by a second SI Reset message arriving while we are still
processing a prior one.
assume that TAS() will always succeed the first time, even if the lock
is known to be free. Also, make sure that code will eventually time out
and report a stuck spinlock, rather than looping forever. Small cleanups
in s_lock.h, too.
1. Distinguish cases where a Datum representing a tuple datatype is an OID
from cases where it is a pointer to TupleTableSlot, and make sure we use
the right typlen in each case.
2. Make fetchatt() and related code support 8-byte by-value datatypes on
machines where Datum is 8 bytes. Centralize knowledge of the available
by-value datatype sizes in two macros in tupmacs.h, so that this will be
easier if we ever have to do it again.
level" locks. A session lock is not released at transaction commit (but it
is released on transaction abort, to ensure recovery after an elog(ERROR)).
In VACUUM, use a session lock to protect the master table while vacuuming a
TOAST table, so that the TOAST table can be done in an independent
transaction.
I also took this opportunity to do some cleanup and renaming in the lock
code. The previously noted bug in ProcLockWakeup, that it couldn't wake up
any waiters beyond the first non-wakeable waiter, is now fixed. Also found
a previously unknown bug of the same kind (failure to scan all members of
a lock queue in some cases) in DeadLockCheck. This might have led to failure
to detect a deadlock condition, resulting in indefinite waits, but it's
difficult to characterize the conditions required to trigger a failure.
>> xlog.c : special case for beos to avoid 'link' which does not work yet
>> beos/sem.c : implementation of new sem_ctl call (GETPID) and a new
>sem_op
>> flag (IPCNOWAIT)
>> dynloader/beos.c : add a verification of symbol validity (seem that
the
>> loader sometime return OK with an invalid symbol)
>> postmaster.c : add beos forking support for the new checkpoint
process
>> postgres.c : remove beos special case for getrusage
>> beos.h : Correction of a bas definition of AF_UNIX, misc defnitions
>>
>>
>> thanks
>>
>>
>> cyril
Cyril VELTER
to ensure that we have released buffer refcounts and so forth, rather than
putting ad-hoc operations before (some of the calls to) proc_exit. Add
commentary to discourage future hackers from repeating that mistake.
varlena type. (I did not force initdb, but you won't see the fix
unless you do one.) Also, make sure all index support operators and
functions are careful not to leak memory for toasted inputs; I had
missed some hash and rtree support ops on this point before.
both MULTIBYTE and TOAST prevent char(n) from being truly fixed-size.
Simplify and speed up fastgetattr() and index_getattr() macros by
eliminating special cases for attnum=1. It's just as fast to handle
the first attribute by presetting its attcacheoff to zero; so do that
instead when loading the tupledesc in relcache.c.
included by everything that includes bufmgr.h --- it's supposed to be
internals, after all, not part of the API! This fixes the conflict
against FreeBSD headers reported by Rosenman, by making it unnecessary
for s_lock.h to be included by plperl.c.
IPC key assignment will now work correctly even when multiple postmasters
are using same logical port number (which is possible given -k switch).
There is only one shared-mem segment per postmaster now, not 3.
Rip out broken code for non-TAS case in bufmgr and xlog, substitute a
complete S_LOCK emulation using semaphores in spin.c. TAS and non-TAS
logic is now exactly the same.
When deadlock is detected, "Deadlock detected" is now the elog(ERROR)
message, rather than a NOTICE that comes out before an unhelpful ERROR.
re-adopt these settings at every postmaster or standalone-backend startup.
This should fix problems with indexes becoming corrupt due to failure to
provide consistent locale environment for postmaster at all times. Also,
refuse to start up a non-locale-enabled compilation in a database originally
initdb'd with a non-C locale. Suppress LIKE index optimization if locale
is not "C" or "POSIX" (are there any other locales where it's safe?).
Issue NOTICE during initdb if selected locale disables LIKE optimization.
maintained for each cache entry. A cache entry will not be freed until
the matching ReleaseSysCache call has been executed. This eliminates
worries about cache entries getting dropped while still in use. See
my posting to pg-hackers of even date for more info.
Context diff this time.
Remove -m486 compile args for FreeBSD-i386, compile -O2 on i386.
Compile with only -O on alpha for codegen safety.
Make the port use the TEST_AND_SET for alpha and i386 on FreeBSD.
Fix a lot of bogus string formats for outputting pointers (cast to int
and %u/%x replaced with no cast and %p), and 'Size'(size_t) are now
cast to 'unsigned long' and output with %lu/
Remove an unused variable.
Alfred Perlstein
message about recursive use of a syscache. Also remove most of the
specialized indexscan routines in indexing.c --- it turns out that
catcache.c is perfectly able to perform the indexscan for itself,
in fact has already looked up all the information needed to do so!
This should be faster as well as needing far less boilerplate code.
(WAL logging for this is not done yet, however.) Clean up a number of really
crufty things that are no longer needed now that DROP behaves nicely. Make
temp table mapper do the right things when drop or rename affecting a temp
table is rolled back. Also, remove "relation modified while in use" error
check, in favor of locking tables at first reference and holding that lock
throughout the statement.
'AbortTransaction and not in in-progress state' when client disconnects
just after an error. Notice seems pretty harmless, so I'm not going
to worry about back-patching this into 7.0.* ...
source, due to addition of header overhead), store it as plain data
rather than pseudo-compressed data. This saves a few microseconds when
reading it out, but much more importantly guarantees that the toaster
won't actually expand tuples that contain incompressible data. That's
essential to avoid 'Tuple too big' failures with large objects.
from bufmgr - it would be nice to have separate hash in smgr
for node <--> fd mappings, but for the moment it's easy to
add new hash to relcache.
Fixed small bug in xlog.c:ReadRecord.
* Makefile: Add more standard targets. Improve shell redirection in GNU
make detection.
* src/backend/access/transam/rmgr.c: Fix incorrect(?) C.
* src/backend/libpq/pqcomm.c (StreamConnection): Work around accept() bug.
* src/include/port/unixware.h: ...with help from here.
* src/backend/nodes/print.c (plannode_type): Remove some "break"s after
"return"s.
* src/backend/tcop/dest.c (DestToFunction): ditto.
* src/backend/nodes/readfuncs.c: Add proper prototypes.
* src/backend/utils/adt/numutils.c (pg_atoi): Cope specially with strtol()
setting EINVAL. This saves us from creating an extra set of regression test
output for the affected systems.
* src/include/storage/s_lock.h (tas): Correct prototype.
* src/interfaces/libpq/fe-connect.c (parseServiceInfo): Don't use variable
as dimension in array definition.
* src/makefiles/Makefile.unixware: Add support for GCC.
* src/template/unixware: same here
* src/test/regress/expected/abstime-solaris-1947.out: Adjust whitespace.
* src/test/regress/expected/horology-solaris-1947.out: Part of this file
was evidently missing.
* src/test/regress/pg_regress.sh: Fix shell. mkdir -p returns non-zero if
the directory exists.
* src/test/regress/resultmap: Add entries for Unixware.
trying to toast tuples inserted into toast tables! Fix is two-pronged:
first, ensure all columns of a toast table are marked attstorage='p',
and second, alter the target chunk size so that it's less than the
threshold for trying to toast a tuple. (Code tried to do that but the
expression was wrong.) A few cosmetic cleanups in tuptoaster too.
NOTE: initdb forced due to change in toaster chunk-size.
These two routines will now ALWAYS elog() on failure, whether you ask for
a lock or not. If you really want to get a NULL return on failure, call
the new routines heap_open_nofail()/heap_openr_nofail(). By my count there
are only about three places that actually want that behavior. There were
rather more than three places that were missing the check they needed to
make under the old convention :-(.
allows fixing problems with operators that expected to be able to
return a NULL, such as the '#' line-segment-intersection operator
that tried to return NULL when the two segments don't intersect.
(See, eg, bug report from 1-Nov-99 on pghackers.) Fix some other
bugs in passing, such as backwards comparison in path_distance().
actually, but who could understand it with no comments? Fix bug
while at it: _bt_orderkeys would try to invoke comparisons on
NULL inputs, given the right sort of redundant quals.
left keys during bottom-up index build, and leave some free space
instead of packing the pages to the brim (so as to avoid vast numbers
of page splits during the first interactive insertions).
duplicate keys by letting search go to the left rather than right when an
equal key is seen at an upper tree level. Fix poor choice of page split
point (leading to insertion failures) that was forced by chaining logic.
Don't store leftmost key in non-leaf pages, since it's not necessary.
Don't create root page until something is first stored in the index, so an
unused index is now 8K not 16K. (Doesn't seem to be as easy to get rid of
the metadata page, unfortunately.) Massive cleanup of unreadable code,
fix poor, obsolete, and just plain wrong documentation and comments.
See src/backend/access/nbtree/README for the gory details.
pass-by-ref data types --- eg, an index on lower(textfield) --- no longer
leak memory during index creation or update. Clean up a lot of redundant
code ... did you know that copy, vacuum, truncate, reindex, extend index,
and bootstrap each basically duplicated the main executor's logic for
extracting information about an index and preparing index entries?
Functional indexes should be a little faster now too, due to removal
of repeated function lookups.
CREATE INDEX 'opt_type' clause is deimplemented by these changes,
but I haven't removed it from the parser yet (need to merge with
Thomas' latest change set first).
memory contexts. Currently, only leaks in expressions executed as
quals or projections are handled. Clean up some old dead cruft in
executor while at it --- unused fields in state nodes, that sort of thing.
Don't use DISABLE_COMPLEX_MACRO on Solaris. Don't define the
replacement function in the header file. Use -KPIC, not -K PIC.
Use CC to link C++ libraries, not ld/ar.
Eliminate file not found warnings in tcl build code.
for details). It doesn't really do that much yet, since there are no
short-term memory contexts in the executor, but the infrastructure is
in place and long-term contexts are handled reasonably. A few long-
standing bugs have been fixed, such as 'VACUUM; anything' in a single
query string crashing. Also, out-of-memory is now considered a
recoverable ERROR, not FATAL.
Eliminate a large amount of crufty, now-dead code in and around
memory management.
Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and
backend startup.
entries now for int8 and network hash indexes. int24_ops and int42_ops
are gone. pg_opclass no longer contains multiple entries claiming to be
the default opclass for the same datatype. opr_sanity regress test
extended to catch errors like these in the future.
materialized tupleset is small enough) instead of a temporary relation.
This was something I was thinking of doing anyway for performance, and Jan
says he needs it for TOAST because he doesn't want to cope with toasting
noname relations. With this change, the 'noname table' support in heap.c
is dead code, and I have accordingly removed it. Also clean up 'noname'
plan handling in planner --- nonames are either sort or materialize plans,
and it seems less confusing to handle them separately under those names.
passing the index-is-unique flag to index build routines (duh! ...
why wasn't it done this way to begin with?). Aside from eliminating
an eyesore, this should save a few milliseconds in btree index creation
because a full scan of pg_index is not needed any more.
Bruce Momjian
Tom Lane
Vadim B. Mikheev
Hiroshi Inoue
Marc G. Fournier
Peter Eisentraut
Jan Wieck