*before* acquiring shlock on buffer context. This way we should be
protected against conflicts with FlushRelationBuffers.
(Seems we never do excl lock and then StartBufferIO for the same
buffer, so there should be no deadlock here, - but we'd better
check this very soon).
* Store two past checkpoint locations, not just one, in pg_control.
On startup, we fall back to the older checkpoint if the newer one
is unreadable. Also, a physical copy of the newest checkpoint record
is kept in pg_control for possible use in disaster recovery (ie,
complete loss of pg_xlog). Also add a version number for pg_control
itself. Remove archdir from pg_control; it ought to be a GUC
parameter, not a special case (not that it's implemented yet anyway).
* Suppress successive checkpoint records when nothing has been entered
in the WAL log since the last one. This is not so much to avoid I/O
as to make it actually useful to keep track of the last two
checkpoints. If the things are right next to each other then there's
not a lot of redundancy gained...
* Change CRC scheme to a true 64-bit CRC, not a pair of 32-bit CRCs
on alternate bytes. Polynomial borrowed from ECMA DLT1 standard.
* Fix XLOG record length handling so that it will work at BLCKSZ = 32k.
* Change XID allocation to work more like OID allocation. (This is of
dubious necessity, but I think it's a good idea anyway.)
* Fix a number of minor bugs, such as off-by-one logic for XLOG file
wraparound at the 4 gig mark.
* Add documentation and clean up some coding infelicities; move file
format declarations out to include files where planned contrib
utilities can get at them.
* Checkpoint will now occur every CHECKPOINT_SEGMENTS log segments or
every CHECKPOINT_TIMEOUT seconds, whichever comes first. It is also
possible to force a checkpoint by sending SIGUSR1 to the postmaster
(undocumented feature...)
* Defend against kill -9 postmaster by storing shmem block's key and ID
in postmaster.pid lockfile, and checking at startup to ensure that no
processes are still connected to old shmem block (if it still exists).
* Switch backends to accept SIGQUIT rather than SIGUSR1 for emergency
stop, for symmetry with postmaster and xlog utilities. Clean up signal
handling in bootstrap.c so that xlog utilities launched by postmaster
will react to signals better.
* Standalone bootstrap now grabs lockfile in target directory, as added
insurance against running it in parallel with live postmaster.
only if at least N other backends currently have open transactions. This
is not a great deal of intelligence about whether a delay might be
profitable ... but it beats no intelligence at all. Note that the default
COMMIT_DELAY is still zero --- this new code does nothing unless that
setting is changed.
Also, mark ENABLEFSYNC as a system-wide setting. It's no longer safe to
allow that to be set per-backend, since we may be relying on some other
backend's fsync to have synced the WAL log.
does not lead to a one-second delay, but to an immediate EINVAL failure.
This causes CHECKPOINT to crash with s_lock_stuck much too quickly :-(.
Fix by breaking down the requested wait div/mod 1e6.
> Is there one LOCKMETHODCTL for every backend? I thought there was only
> one of them.
>>
>> You're right, that line is erroneous; it should read
>>
>> size += MAX_LOCK_METHODS * MAXALIGN(sizeof(LOCKMETHODCTL));
>>
>> Not a significant error but it should be changed for clarity ...
waste of cycles on single-CPU machines, and of dubious utility on multi-CPU
machines too.
Tweak s_lock_stuck so that caller can specify timeout interval, and
increase interval before declaring stuck spinlock for buffer locks and XLOG
locks.
On systems that have fdatasync(), use that rather than fsync() to sync WAL
log writes. Ensure that WAL file is entirely allocated during XLogFileInit.
are now separate files "postgres.h" and "postgres_fe.h", which are meant
to be the primary include files for backend .c files and frontend .c files
respectively. By default, only include files meant for frontend use are
installed into the installation include directory. There is a new make
target 'make install-all-headers' that adds the whole content of the
src/include tree to the installed fileset, for use by people who want to
develop server-side code without keeping the complete source tree on hand.
Cleaned up a whole lot of crufty and inconsistent header inclusions.
bothering to check the return value --- which meant that in case the
update or delete failed because of a concurrent update, you'd not find
out about it, except by observing later that the transaction produced
the wrong outcome. There are now subroutines simple_heap_update and
simple_heap_delete that should be used anyplace that you're not prepared
to do the full nine yards of coping with concurrent updates. In
practice, that seems to mean absolutely everywhere but the executor,
because *noplace* else was checking.
rewrite of deadlock checking. Lock holder objects are now reachable from
the associated LOCK as well as from the owning PROC. This makes it
practical to find all the processes holding a lock, as well as all those
waiting on the lock. Also, clean up some of the grottier aspects of the
SHMQueue API, and cause the waitProcs list to be stored in the intuitive
direction instead of the nonintuitive one. (Bet you didn't know that
the code followed the 'prev' link to get to the next waiting process,
instead of the 'next' link. It doesn't do that anymore.)
here is the patch attached which do check in each BLOB operation, if we are
in transaction, and raise an error otherwise. This will prevent such mistakes.
--
Sincerely Yours,
Denis Perchine
are treated more like 'cancel' interrupts: the signal handler sets a
flag that is examined at well-defined spots, rather than trying to cope
with an interrupt that might happen anywhere. See pghackers discussion
of 1/12/01.
are now critical sections, so as to ensure die() won't interrupt us while
we are munging shared-memory data structures. Avoid insecure intermediate
states in some code that proc_exit will call, like palloc/pfree. Rename
START/END_CRIT_CODE to START/END_CRIT_SECTION, since that seems to be
what people tend to call them anyway, and make them be called with () like
a function call, in hopes of not confusing pg_indent.
I doubt that this is sufficient to make SIGTERM safe anywhere; there's
just too much code that could get invoked during proc_exit().
starting a new hashtable search no longer clobbers any other search
active anywhere in the system. Fix RelationCacheInvalidate() so that
it will not crash or go into an infinite loop if invoked recursively,
as for example by a second SI Reset message arriving while we are still
processing a prior one.
In theory we should always get EEXIST if there's a key collision, but
if the kernel code tests error conditions in a weird order, perhaps
EACCES or EIDRM could occur too.
assume that TAS() will always succeed the first time, even if the lock
is known to be free. Also, make sure that code will eventually time out
and report a stuck spinlock, rather than looping forever. Small cleanups
in s_lock.h, too.
level" locks. A session lock is not released at transaction commit (but it
is released on transaction abort, to ensure recovery after an elog(ERROR)).
In VACUUM, use a session lock to protect the master table while vacuuming a
TOAST table, so that the TOAST table can be done in an independent
transaction.
I also took this opportunity to do some cleanup and renaming in the lock
code. The previously noted bug in ProcLockWakeup, that it couldn't wake up
any waiters beyond the first non-wakeable waiter, is now fixed. Also found
a previously unknown bug of the same kind (failure to scan all members of
a lock queue in some cases) in DeadLockCheck. This might have led to failure
to detect a deadlock condition, resulting in indefinite waits, but it's
difficult to characterize the conditions required to trigger a failure.
might change it. Experimentation shows that the signal handler call
mechanism does not save/restore errno for you, at least not on Linux
or HPUX, so this is definitely a real risk.
to ensure that we have released buffer refcounts and so forth, rather than
putting ad-hoc operations before (some of the calls to) proc_exit. Add
commentary to discourage future hackers from repeating that mistake.
included by everything that includes bufmgr.h --- it's supposed to be
internals, after all, not part of the API! This fixes the conflict
against FreeBSD headers reported by Rosenman, by making it unnecessary
for s_lock.h to be included by plperl.c.
IPC key assignment will now work correctly even when multiple postmasters
are using same logical port number (which is possible given -k switch).
There is only one shared-mem segment per postmaster now, not 3.
Rip out broken code for non-TAS case in bufmgr and xlog, substitute a
complete S_LOCK emulation using semaphores in spin.c. TAS and non-TAS
logic is now exactly the same.
When deadlock is detected, "Deadlock detected" is now the elog(ERROR)
message, rather than a NOTICE that comes out before an unhelpful ERROR.
Context diff this time.
Remove -m486 compile args for FreeBSD-i386, compile -O2 on i386.
Compile with only -O on alpha for codegen safety.
Make the port use the TEST_AND_SET for alpha and i386 on FreeBSD.
Fix a lot of bogus string formats for outputting pointers (cast to int
and %u/%x replaced with no cast and %p), and 'Size'(size_t) are now
cast to 'unsigned long' and output with %lu/
Remove an unused variable.
Alfred Perlstein
that search loops only have to scan that far and not through all maxBackends
entries. This eliminates a performance penalty for setting maxBackends
much higher than the average number of active backends. Also, eliminate
no-longer-used 'backend tag' concept. Remove setting of environment
variables at backend start (except for CYR_RECODE), since none of them
are being examined by the backend any longer.
(WAL logging for this is not done yet, however.) Clean up a number of really
crufty things that are no longer needed now that DROP behaves nicely. Make
temp table mapper do the right things when drop or rename affecting a temp
table is rolled back. Also, remove "relation modified while in use" error
check, in favor of locking tables at first reference and holding that lock
throughout the statement.
kibitzing from Tom Lane. Large objects are now all stored in a single
system relation "pg_largeobject" --- no more xinv or xinx files, no more
relkind 'l'. This should offer substantial performance improvement for
large numbers of LOs, since there won't be directory bloat anymore.
It'll also fix problems like running out of locktable space when you
access thousands of LOs in one transaction.
Also clean up cruft in read/write routines. LOs with "holes" in them
(never-written byte ranges) now work just like Unix files with holes do:
a hole reads as zeroes but doesn't occupy storage space.
INITDB forced!
from bufmgr - it would be nice to have separate hash in smgr
for node <--> fd mappings, but for the moment it's easy to
add new hash to relcache.
Fixed small bug in xlog.c:ReadRecord.
and DropBuffers. Formerly we cleared the flag for each buffer currently
belonging to the target rel or database, but that's completely wrong!
Must look at BufferTagLastDirtied to see whether the BufferDirtiedByMe
flag is relevant to target rel or not; this is *independent* of the
current contents of the buffer. Vadim spotted this problem, but his
fix was only partially correct...
> Regression tests opr_sanity and sanity_check are now failing.
Um, Bruce, I've said several times that I didn't think Perchine's large
object changes should be applied until someone had actually reviewed
them.
I tested it restoring my database with > 100000 BLOBS, and dumping it out.
But unfortunatly I can not restore it back due to problems in pg_dump.
--
Sincerely Yours,
Denis Perchine
source directory. This involves mostly makefiles using $(srcdir) when they
might have used ".". (Regression tests don't work with this, yet.)
Sort out usage of CPPFLAGS, CFLAGS (and CXXFLAGS). Add "override" keyword
in most places, to preserve necessary flags even when the user overrode the
flags.
> this is patch v 0.4 to support transactions with BLOBs.
> All BLOBs are in one table. You need to make initdb.
>
> --
> Sincerely Yours,
> Denis Perchine
after that dynamic loading isn't working and shared memory handling is
broken.
Attached with this message, there is a Zip file which contain :
* beos.diff = patch file generated with difforig
* beos = folder with beos support files which need to be moved in /
src/backend/port
* expected = foler with three file for message and precision
difference in regression test
* regression.diff = rule problem (need to kill the backend manualy)
* dynloader = dynloader files (they are also in the pacth files,
but there is so much modification that I have join full files)
Everything works except a problem in 'rules' Is there some problems
with rules in the current tree ? It used to works with last week tree.
Cyril VELTER
working on the VERY latest version of BeOS. I'm sure there will be
alot of comments, but then if there weren't I'd be disappointed!
Thanks for your continuing efforts to get this into your tree.
Haven't bothered with the new files as they haven't changed.
BTW Peter, the compiler is "broken" about the bool define and so on.
I'm filing a bug report to try and get it addressed. Hopefully then we
can tidy up the code a bit.
I await the replies with interest :)
David Reid
virtual FDs, we just return the ENFILE/EMFILE error to the caller,
rather than immediate elog(). This allows more robust behavior in
the postmaster, which uses AllocateFile() but does not want elog().
duplicate keys by letting search go to the left rather than right when an
equal key is seen at an upper tree level. Fix poor choice of page split
point (leading to insertion failures) that was forced by chaining logic.
Don't store leftmost key in non-leaf pages, since it's not necessary.
Don't create root page until something is first stored in the index, so an
unused index is now 8K not 16K. (Doesn't seem to be as easy to get rid of
the metadata page, unfortunately.) Massive cleanup of unreadable code,
fix poor, obsolete, and just plain wrong documentation and comments.
See src/backend/access/nbtree/README for the gory details.
There's now only one transition value and transition function.
NULL handling in aggregates is a lot cleaner. Also, use Numeric
accumulators instead of integer accumulators for sum/avg on integer
datatypes --- this avoids overflow at the cost of being a little slower.
Implement VARIANCE() and STDDEV() aggregates in the standard backend.
Also, enable new LIKE selectivity estimators by default. Unrelated
change, but as long as I had to force initdb anyway...
pass-by-ref data types --- eg, an index on lower(textfield) --- no longer
leak memory during index creation or update. Clean up a lot of redundant
code ... did you know that copy, vacuum, truncate, reindex, extend index,
and bootstrap each basically duplicated the main executor's logic for
extracting information about an index and preparing index entries?
Functional indexes should be a little faster now too, due to removal
of repeated function lookups.
CREATE INDEX 'opt_type' clause is deimplemented by these changes,
but I haven't removed it from the parser yet (need to merge with
Thomas' latest change set first).
Don't go through pg_exec_query_dest(), but directly to the execution
routines. Also, extend parameter lists so that there's no need to
change the global setting of allowSystemTableMods, a hack that was
certain to cause trouble in the event of any error.
for details). It doesn't really do that much yet, since there are no
short-term memory contexts in the executor, but the infrastructure is
in place and long-term contexts are handled reasonably. A few long-
standing bugs have been fixed, such as 'VACUUM; anything' in a single
query string crashing. Also, out-of-memory is now considered a
recoverable ERROR, not FATAL.
Eliminate a large amount of crufty, now-dead code in and around
memory management.
Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and
backend startup.
passing the index-is-unique flag to index build routines (duh! ...
why wasn't it done this way to begin with?). Aside from eliminating
an eyesore, this should save a few milliseconds in btree index creation
because a full scan of pg_index is not needed any more.
was inappropriately relying on rel->rd_nblocks to tell if the LO is
empty (apparently a hack to get around a long-dead index bug), causing
misbehavior on a written-but-never-vacuumed LO. Also, inv_read failed
to cope gracefully with 'holes' (unwritten regions) in the object.
> situation is already tracked in File routines, but a little bit
> incorrectly.
> After small survey in Linux kernel code, I am not sure about
> it. New patch set pos to unknown in the case of read/write
> fails. And do lseek again.
> Here is the full patch for this. This patch reduce amount of
> lseek call ten ti mes for update statement and twenty times for
> select statement. I tested joined up date and count(*) select
> for table with rows > 170000 and 10 indices. I think this is
> worse of trying. Before lseek calls account for more than 5% o
> f time. Now they are 0.89 and 0.15 respectevly.
>
> Due to only one file modification patch should be applied in
> src/backedn/stora ge/file/ dir.
-- Sincerely Yours,
Denis Perchine
inputs have been converted to newstyle. This should go a long way towards
fixing our portability problems with platforms where char and short
parameters are passed differently from int-width parameters. Still
more to do for the Alpha port however.
it will close VFDs if necessary to surmount ENFILE or EMFILE failures.
Make use of this in md.c, xlog.c, and user.c routines that were
formerly vulnerable to these failures. In particular, this should
handle failures of mdblindwrt() that have been observed under heavy
load conditions. (By golly, every other process on the system may
crash after Postgres eats up all the kernel FDs, but Postgres will
keep going!)
That means you can now set your options in either or all of $PGDATA/configuration,
some postmaster option (--enable-fsync=off), or set a SET command. The list of
options is in backend/utils/misc/guc.c, documentation will be written post haste.
pg_options is gone, so is that pq_geqo config file. Also removed were backend -K,
-Q, and -T options (no longer applicable, although -d0 does the same as -Q).
Added to configure an --enable-syslog option.
changed all callers from TPRINTF to elog(DEBUG)
key call sites are changed, but most called functions are still oldstyle.
An exception is that the PL managers are updated (so, for example, NULL
handling now behaves as expected in plperl and plpgsql functions).
NOTE initdb is forced due to added column in pg_proc.
Hiroshi. ReleaseRelationBuffers now removes rel's buffers from pool,
instead of merely marking them nondirty. The old code would leave valid
buffers for a deleted relation, which didn't cause any known problems
but can't possibly be a good idea. There were several places which called
ReleaseRelationBuffers *and* FlushRelationBuffers, which is now
unnecessary; but there were others that did not. FlushRelationBuffers
no longer emits a warning notice if it finds dirty buffers to flush,
because with the current bufmgr behavior that's not an unexpected
condition. Also, FlushRelationBuffers will flush out all dirty buffers
for the relation regardless of block number. This ensures that
pg_upgrade's expectations are met about tuple on-row status bits being
up-to-date on disk. Lastly, tweak BufTableDelete() to clear the
buffer's tag so that no one can mistake it for being a still-valid
buffer for the page it once held. Formerly, the buffer would not be
found by buffer hashtable searches after BufTableDelete(), but it would
still be thought to belong to its old relation by the routines that
sequentially scan the shared-buffer array. Again I know of no bugs
caused by that, but it still can't be a good idea.
IRIX systems using the native compilers. A summary is:
- Various files use "//" as a comment delimiter in c files.
- Problems caused by assuming "char" is signed.
cash.in: building -signed the rules regression test fails as described
in FAQ_QNX4. If CHAR_MAX is "255U" then ((signed char)CHAR_MAX) is -1.
postmaster.c: random number regression test failed without this change.
- Some generic build issues and warning message cleanup.
David Kaelbling
whether to do fsync or not, and if so (which should be seldom) just
do the fsync immediately. This way we need not build data structures
in md.c/fd.c for blind writes.
as a shared dirtybit for each shared buffer. The shared dirtybit still
controls writing the buffer, but the local bit controls whether we need
to fsync the buffer's file. This arrangement fixes a bug that allowed
some required fsyncs to be missed, and should improve performance as well.
For more info see my post of same date on pghackers.
In the event of an elog() while the mode was set to immediate write,
there was no way for it to be set back to the normal delayed write.
The mechanism was a waste of space and cycles anyway, since the only user
was varsup.c, which could perfectly well call FlushBuffer directly.
Now it does just that, and the notion of a write mode is gone.
running gcc and HP's cc with warnings cranked way up. Signed vs unsigned
comparisons, routines declared static and then defined not-static,
that kind of thing. Tedious, but perhaps useful...
it's a good idea to choose the directory size based on the expected
number of entries. But ShmemInitHash was using a hard-wired constant.
Boo hiss. This accounts for recent report of postmaster failure when
asking for 64K or more buffers.
integers) to be strings instead of 'double'. We convert from string form
to internal representation only after type resolution has determined the
correct type for the constant. This eliminates loss-of-precision worries
and gets rid of the change in behavior seen at 17 digits with the
previous kluge.
Tom Lane
Bruce Momjian
Vadim B. Mikheev
Hiroshi Inoue
Peter Eisentraut