Replace the former method of determining snapshot xmax --- to wit, calling

ReadNewTransactionId from GetSnapshotData --- with a "latestCompletedXid"
variable that is updated during transaction commit or abort.  Since
latestCompletedXid is written only in places that had to lock ProcArrayLock
exclusively anyway, and is read only in places that had to lock ProcArrayLock
shared anyway, it adds no new locking requirements to the system despite being
cluster-wide.  Moreover, removing ReadNewTransactionId from snapshot
acquisition eliminates the need to take both XidGenLock and ProcArrayLock at
the same time.  Since XidGenLock is sometimes held across I/O this can be a
significant win.  Some preliminary benchmarking suggested that this patch has
no effect on average throughput but can significantly improve the worst-case
transaction times seen in pgbench.  Concept by Florian Pflug, implementation
by Tom Lane.

src/backend/access/transam/README

@@ -1,4 +1,4 @@
-$PostgreSQL: pgsql/src/backend/access/transam/README,v 1.8 2007/09/07 20:59:26 tgl Exp $
+$PostgreSQL: pgsql/src/backend/access/transam/README,v 1.9 2007/09/08 20:31:14 tgl Exp $
 
 The Transaction System
 ----------------------
@@ -238,8 +238,10 @@ reason why this would be bad is that C would see (in the row inserted by A)
 earlier changes by B, and it would be inconsistent for C not to see any
 of B's changes elsewhere in the database.
 
-Formally, the correctness requirement is "if A sees B as committed,
+Formally, the correctness requirement is "if a snapshot A considers
-and B sees C as committed, then A must see C as committed".
+transaction X as committed, and any of transaction X's snapshots considered
+transaction Y as committed, then snapshot A must consider transaction Y as
+committed".
 
 What we actually enforce is strict serialization of commits and rollbacks
 with snapshot-taking: we do not allow any transaction to exit the set of
@@ -248,42 +250,45 @@ stronger than necessary for consistency, but is relatively simple to
 enforce, and it assists with some other issues as explained below.)  The
 implementation of this is that GetSnapshotData takes the ProcArrayLock in
 shared mode (so that multiple backends can take snapshots in parallel),
-but xact.c must take the ProcArrayLock in exclusive mode while clearing
+but ProcArrayEndTransaction must take the ProcArrayLock in exclusive mode
-MyProc->xid at transaction end (either commit or abort).
+while clearing MyProc->xid at transaction end (either commit or abort).
 
-GetSnapshotData must in fact acquire ProcArrayLock before it calls
+ProcArrayEndTransaction also holds the lock while advancing the shared
-ReadNewTransactionId.  Otherwise it would be possible for a transaction A
+latestCompletedXid variable.  This allows GetSnapshotData to use
-postdating the xmax to commit, and then an existing transaction B that saw
+latestCompletedXid + 1 as xmax for its snapshot: there can be no
-A as committed to commit, before GetSnapshotData is able to acquire
+transaction >= this xid value that the snapshot needs to consider as
-ProcArrayLock and finish taking its snapshot.  This would violate the
+completed.
-consistency requirement, because A would be still running and B not
-according to this snapshot.
 
 In short, then, the rule is that no transaction may exit the set of
-currently-running transactions between the time we fetch xmax and the time
+currently-running transactions between the time we fetch latestCompletedXid
-we finish building our snapshot.  However, this restriction only applies
+and the time we finish building our snapshot.  However, this restriction
-to transactions that have an XID --- read-only transactions can end without
+only applies to transactions that have an XID --- read-only transactions
-acquiring ProcArrayLock, since they don't affect anyone else's snapshot.
+can end without acquiring ProcArrayLock, since they don't affect anyone
+else's snapshot nor latestCompletedXid.
 
 Transaction start, per se, doesn't have any interlocking with these
 considerations, since we no longer assign an XID immediately at transaction
-start.  But when we do decide to allocate an XID, we must require
+start.  But when we do decide to allocate an XID, GetNewTransactionId must
-GetNewTransactionId to store the new XID into the shared ProcArray before
+store the new XID into the shared ProcArray before releasing XidGenLock.
-releasing XidGenLock.  This ensures that when GetSnapshotData calls
+This ensures that all top-level XIDs <= latestCompletedXid are either
-ReadNewTransactionId (which also takes XidGenLock), all active XIDs before
+present in the ProcArray, or not running anymore.  (This guarantee doesn't
-the returned value of nextXid are already present in the ProcArray and
+apply to subtransaction XIDs, because of the possibility that there's not
-can't be missed by GetSnapshotData.  Unfortunately, we can't have
+room for them in the subxid array; instead we guarantee that they are
-GetNewTransactionId take ProcArrayLock to do this, else it could deadlock
+present or the overflow flag is set.)  If a backend released XidGenLock
-against GetSnapshotData.  Therefore, we simply let GetNewTransactionId
+before storing its XID into MyProc, then it would be possible for another
-store into MyProc->xid without any lock.  We are thereby relying on
+backend to allocate and commit a later XID, causing latestCompletedXid to
-fetch/store of an XID to be atomic, else other backends might see a
+pass the first backend's XID, before that value became visible in the
-partially-set XID.  (NOTE: for multiprocessors that need explicit memory
+ProcArray.  That would break GetOldestXmin, as discussed below.
-access fence instructions, this means that acquiring/releasing XidGenLock
+
-is just as necessary as acquiring/releasing ProcArrayLock for
+We allow GetNewTransactionId to store the XID into MyProc->xid (or the
-GetSnapshotData to ensure it sees up-to-date xid fields.)  This also means
+subxid array) without taking ProcArrayLock.  This was once necessary to
-that readers of the ProcArray xid fields must be careful to fetch a value
+avoid deadlock; while that is no longer the case, it's still beneficial for
-only once, rather than assume they can read it multiple times and get the
+performance.  We are thereby relying on fetch/store of an XID to be atomic,
-same answer each time.
+else other backends might see a partially-set XID.  This also means that
+readers of the ProcArray xid fields must be careful to fetch a value only
+once, rather than assume they can read it multiple times and get the same
+answer each time.  (Use volatile-qualified pointers when doing this, to
+ensure that the C compiler does exactly what you tell it to.)
 
 Another important activity that uses the shared ProcArray is GetOldestXmin,
 which must determine a lower bound for the oldest xmin of any active MVCC
@@ -303,12 +308,10 @@ currently-active XIDs: no xact, in particular not the oldest, can exit
 while we hold shared ProcArrayLock.  So GetOldestXmin's view of the minimum
 active XID will be the same as that of any concurrent GetSnapshotData, and
 so it can't produce an overestimate.  If there is no active transaction at
-all, GetOldestXmin returns the result of ReadNewTransactionId.  Note that
+all, GetOldestXmin returns latestCompletedXid + 1, which is a lower bound
-two concurrent executions of GetOldestXmin might not see the same result
+for the xmin that might be computed by concurrent or later GetSnapshotData
-from ReadNewTransactionId --- but if there is a difference, the intervening
+calls.  (We know that no XID less than this could be about to appear in
-execution(s) of GetNewTransactionId must have stored their XIDs into the
+the ProcArray, because of the XidGenLock interlock discussed above.)
-ProcArray, so the later execution of GetOldestXmin will see them and
-compute the same global xmin anyway.
 
 GetSnapshotData also performs an oldest-xmin calculation (which had better
 match GetOldestXmin's) and stores that into RecentGlobalXmin, which is used

src/backend/access/transam/transam.c

@@ -8,7 +8,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/access/transam/transam.c,v 1.70 2007/08/01 22:45:07 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/access/transam/transam.c,v 1.71 2007/09/08 20:31:14 tgl Exp $
  *
  * NOTES
  *	  This file contains the high level access-method interface to the
@@ -432,6 +432,33 @@ TransactionIdFollowsOrEquals(TransactionId id1, TransactionId id2)
 	return (diff >= 0);
 }
 
+
+/*
+ * TransactionIdLatest --- get latest XID among a main xact and its children
+ */
+TransactionId
+TransactionIdLatest(TransactionId mainxid,
+					int nxids, const TransactionId *xids)
+{
+	TransactionId	result;
+
+	/*
+	 * In practice it is highly likely that the xids[] array is sorted, and
+	 * so we could save some cycles by just taking the last child XID, but
+	 * this probably isn't so performance-critical that it's worth depending
+	 * on that assumption.  But just to show we're not totally stupid, scan
+	 * the array back-to-front to avoid useless assignments.
+	 */
+	result = mainxid;
+	while (--nxids >= 0)
+	{
+		if (TransactionIdPrecedes(result, xids[nxids]))
+			result = xids[nxids];
+	}
+	return result;
+}
+
+
 /*
  * TransactionIdGetCommitLSN
  *

src/backend/access/transam/twophase.c

@@ -7,7 +7,7 @@
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * IDENTIFICATION
- *		$PostgreSQL: pgsql/src/backend/access/transam/twophase.c,v 1.34 2007/09/05 20:53:17 tgl Exp $
+ *		$PostgreSQL: pgsql/src/backend/access/transam/twophase.c,v 1.35 2007/09/08 20:31:14 tgl Exp $
  *
  * NOTES
  *		Each global transaction is associated with a global transaction
@@ -1127,6 +1127,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
 	char	   *buf;
 	char	   *bufptr;
 	TwoPhaseFileHeader *hdr;
+	TransactionId latestXid;
 	TransactionId *children;
 	RelFileNode *commitrels;
 	RelFileNode *abortrels;
@@ -1162,6 +1163,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
 	abortrels = (RelFileNode *) bufptr;
 	bufptr += MAXALIGN(hdr->nabortrels * sizeof(RelFileNode));
 
+	/* compute latestXid among all children */
+	latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
+
 	/*
 	 * The order of operations here is critical: make the XLOG entry for
 	 * commit or abort, then mark the transaction committed or aborted in
@@ -1179,7 +1183,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
 									   hdr->nsubxacts, children,
 									   hdr->nabortrels, abortrels);
 
-	ProcArrayRemove(&gxact->proc);
+	ProcArrayRemove(&gxact->proc, latestXid);
 
 	/*
 	 * In case we fail while running the callbacks, mark the gxact invalid so

src/backend/access/transam/varsup.c

@@ -6,7 +6,7 @@
  * Copyright (c) 2000-2007, PostgreSQL Global Development Group
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/access/transam/varsup.c,v 1.78 2007/02/15 23:23:22 alvherre Exp $
+ *	  $PostgreSQL: pgsql/src/backend/access/transam/varsup.c,v 1.79 2007/09/08 20:31:14 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -31,7 +31,9 @@ VariableCache ShmemVariableCache = NULL;
 
 
 /*
- * Allocate the next XID for my new transaction.
+ * Allocate the next XID for my new transaction or subtransaction.
+ *
+ * The new XID is also stored into MyProc before returning.
  */
 TransactionId
 GetNewTransactionId(bool isSubXact)
@@ -43,7 +45,11 @@ GetNewTransactionId(bool isSubXact)
 	 * transaction id.
 	 */
 	if (IsBootstrapProcessingMode())
+	{
+		Assert(!isSubXact);
+		MyProc->xid = BootstrapTransactionId;
 		return BootstrapTransactionId;
+	}
 
 	LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
 
@@ -112,19 +118,19 @@ GetNewTransactionId(bool isSubXact)
 	TransactionIdAdvance(ShmemVariableCache->nextXid);
 
 	/*
-	 * We must store the new XID into the shared PGPROC array before releasing
+	 * We must store the new XID into the shared ProcArray before releasing
-	 * XidGenLock.	This ensures that when GetSnapshotData calls
+	 * XidGenLock.  This ensures that every active XID older than
-	 * ReadNewTransactionId, all active XIDs before the returned value of
+	 * latestCompletedXid is present in the ProcArray, which is essential
-	 * nextXid are already present in PGPROC.  Else we have a race condition.
+	 * for correct OldestXmin tracking; see src/backend/access/transam/README.
 	 *
 	 * XXX by storing xid into MyProc without acquiring ProcArrayLock, we are
 	 * relying on fetch/store of an xid to be atomic, else other backends
 	 * might see a partially-set xid here.	But holding both locks at once
-	 * would be a nasty concurrency hit (and in fact could cause a deadlock
+	 * would be a nasty concurrency hit.  So for now, assume atomicity.
-	 * against GetSnapshotData).  So for now, assume atomicity. Note that
+	 *
-	 * readers of PGPROC xid field should be careful to fetch the value only
+	 * Note that readers of PGPROC xid fields should be careful to fetch the
-	 * once, rather than assume they can read it multiple times and get the
+	 * value only once, rather than assume they can read a value multiple
-	 * same answer each time.
+	 * times and get the same answer each time.
 	 *
 	 * The same comments apply to the subxact xid count and overflow fields.
 	 *
@@ -138,11 +144,10 @@ GetNewTransactionId(bool isSubXact)
 	 * race-condition window, in that the new XID will not appear as running
 	 * until its parent link has been placed into pg_subtrans. However, that
 	 * will happen before anyone could possibly have a reason to inquire about
-	 * the status of the XID, so it seems OK. (Snapshots taken during this
+	 * the status of the XID, so it seems OK.  (Snapshots taken during this
 	 * window *will* include the parent XID, so they will deliver the correct
 	 * answer later on when someone does have a reason to inquire.)
 	 */
-	if (MyProc != NULL)
 	{
 		/*
 		 * Use volatile pointer to prevent code rearrangement; other backends

src/backend/access/transam/xact.c

@@ -10,7 +10,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/access/transam/xact.c,v 1.249 2007/09/07 20:59:26 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/access/transam/xact.c,v 1.250 2007/09/08 20:31:14 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -233,7 +233,7 @@ static void CallSubXactCallbacks(SubXactEvent event,
 					 SubTransactionId parentSubid);
 static void CleanupTransaction(void);
 static void CommitTransaction(void);
-static void RecordTransactionAbort(bool isSubXact);
+static TransactionId RecordTransactionAbort(bool isSubXact);
 static void StartTransaction(void);
 
 static void RecordSubTransactionCommit(void);
@@ -748,13 +748,17 @@ AtSubStart_ResourceOwner(void)
 /*
  *	RecordTransactionCommit
  *
+ * Returns latest XID among xact and its children, or InvalidTransactionId
+ * if the xact has no XID.  (We compute that here just because it's easier.)
+ *
  * This is exported only to support an ugly hack in VACUUM FULL.
  */
-void
+TransactionId
 RecordTransactionCommit(void)
 {
 	TransactionId xid = GetTopTransactionIdIfAny();
 	bool 		markXidCommitted = TransactionIdIsValid(xid);
+	TransactionId latestXid = InvalidTransactionId;
 	int			nrels;
 	RelFileNode *rels;
 	bool		haveNonTemp;
@@ -930,6 +934,9 @@ RecordTransactionCommit(void)
 		END_CRIT_SECTION();
 	}
 
+	/* Compute latestXid while we have the child XIDs handy */
+	latestXid = TransactionIdLatest(xid, nchildren, children);
+
 	/* Reset XactLastRecEnd until the next transaction writes something */
 	XactLastRecEnd.xrecoff = 0;
 
@@ -939,6 +946,8 @@ cleanup:
 		pfree(rels);
 	if (children)
 		pfree(children);
+
+	return latestXid;
 }
 
 
@@ -1084,11 +1093,15 @@ RecordSubTransactionCommit(void)
 
 /*
  *	RecordTransactionAbort
+ *
+ * Returns latest XID among xact and its children, or InvalidTransactionId
+ * if the xact has no XID.  (We compute that here just because it's easier.)
  */
-static void
+static TransactionId
 RecordTransactionAbort(bool isSubXact)
 {
 	TransactionId xid = GetCurrentTransactionIdIfAny();
+	TransactionId latestXid;
 	int			nrels;
 	RelFileNode *rels;
 	int			nchildren;
@@ -1108,7 +1121,7 @@ RecordTransactionAbort(bool isSubXact)
 		/* Reset XactLastRecEnd until the next transaction writes something */
 		if (!isSubXact)
 			XactLastRecEnd.xrecoff = 0;
-		return;
+		return InvalidTransactionId;
 	}
 
 	/*
@@ -1186,6 +1199,9 @@ RecordTransactionAbort(bool isSubXact)
 
 	END_CRIT_SECTION();
 
+	/* Compute latestXid while we have the child XIDs handy */
+	latestXid = TransactionIdLatest(xid, nchildren, children);
+
 	/*
 	 * If we're aborting a subtransaction, we can immediately remove failed
 	 * XIDs from PGPROC's cache of running child XIDs.  We do that here for
@@ -1193,7 +1209,7 @@ RecordTransactionAbort(bool isSubXact)
 	 * main xacts, the equivalent happens just after this function returns.
 	 */
 	if (isSubXact)
-		XidCacheRemoveRunningXids(xid, nchildren, children);
+		XidCacheRemoveRunningXids(xid, nchildren, children, latestXid);
 
 	/* Reset XactLastRecEnd until the next transaction writes something */
 	if (!isSubXact)
@@ -1204,6 +1220,8 @@ RecordTransactionAbort(bool isSubXact)
 		pfree(rels);
 	if (children)
 		pfree(children);
+
+	return latestXid;
 }
 
 /*
@@ -1481,6 +1499,7 @@ static void
 CommitTransaction(void)
 {
 	TransactionState s = CurrentTransactionState;
+	TransactionId latestXid;
 
 	ShowTransactionState("CommitTransaction");
 
@@ -1552,7 +1571,7 @@ CommitTransaction(void)
 	/*
 	 * Here is where we really truly commit.
 	 */
-	RecordTransactionCommit();
+	latestXid = RecordTransactionCommit();
 
 	PG_TRACE1(transaction__commit, MyProc->lxid);
 
@@ -1560,47 +1579,8 @@ CommitTransaction(void)
 	 * Let others know about no transaction in progress by me. Note that
 	 * this must be done _before_ releasing locks we hold and _after_
 	 * RecordTransactionCommit.
-	 *
-	 * Note: MyProc may be null during bootstrap.
 	 */
-	if (MyProc != NULL)
+	ProcArrayEndTransaction(MyProc, latestXid);
-	{
-		if (TransactionIdIsValid(MyProc->xid))
-		{
-			/*
-			 * We must lock ProcArrayLock while clearing MyProc->xid, so
-			 * that we do not exit the set of "running" transactions while
-			 * someone else is taking a snapshot.  See discussion in
-			 * src/backend/access/transam/README.
-			 */
-			LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
-
-			MyProc->xid = InvalidTransactionId;
-			MyProc->lxid = InvalidLocalTransactionId;
-			MyProc->xmin = InvalidTransactionId;
-			MyProc->inVacuum = false;	/* must be cleared with xid/xmin */
-
-			/* Clear the subtransaction-XID cache too while holding the lock */
-			MyProc->subxids.nxids = 0;
-			MyProc->subxids.overflowed = false;
-
-			LWLockRelease(ProcArrayLock);
-		}
-		else
-		{
-			/*
-			 * If we have no XID, we don't need to lock, since we won't
-			 * affect anyone else's calculation of a snapshot.  We might
-			 * change their estimate of global xmin, but that's OK.
-			 */
-			MyProc->lxid = InvalidLocalTransactionId;
-			MyProc->xmin = InvalidTransactionId;
-			MyProc->inVacuum = false;	/* must be cleared with xid/xmin */
-
-			Assert(MyProc->subxids.nxids == 0);
-			Assert(MyProc->subxids.overflowed == false);
-		}
-	}
 
 	/*
 	 * This is all post-commit cleanup.  Note that if an error is raised here,
@@ -1824,20 +1804,8 @@ PrepareTransaction(void)
 	 * Let others know about no transaction in progress by me.	This has to be
 	 * done *after* the prepared transaction has been marked valid, else
 	 * someone may think it is unlocked and recyclable.
-	 *
-	 * We can skip locking ProcArrayLock here, because this action does not
-	 * actually change anyone's view of the set of running XIDs: our entry
-	 * is duplicate with the gxact that has already been inserted into the
-	 * ProcArray.
 	 */
-	MyProc->xid = InvalidTransactionId;
+	ProcArrayClearTransaction(MyProc);
-	MyProc->lxid = InvalidLocalTransactionId;
-	MyProc->xmin = InvalidTransactionId;
-	MyProc->inVacuum = false;	/* must be cleared with xid/xmin */
-
-	/* Clear the subtransaction-XID cache too */
-	MyProc->subxids.nxids = 0;
-	MyProc->subxids.overflowed = false;
 
 	/*
 	 * This is all post-transaction cleanup.  Note that if an error is raised
@@ -1921,6 +1889,7 @@ static void
 AbortTransaction(void)
 {
 	TransactionState s = CurrentTransactionState;
+	TransactionId latestXid;
 
 	/* Prevent cancel/die interrupt while cleaning up */
 	HOLD_INTERRUPTS();
@@ -1987,7 +1956,7 @@ AbortTransaction(void)
 	 * Advertise the fact that we aborted in pg_clog (assuming that we got as
 	 * far as assigning an XID to advertise).
 	 */
-	RecordTransactionAbort(false);
+	latestXid = RecordTransactionAbort(false);
 
 	PG_TRACE1(transaction__abort, MyProc->lxid);
 
@@ -1995,49 +1964,8 @@ AbortTransaction(void)
 	 * Let others know about no transaction in progress by me. Note that this
 	 * must be done _before_ releasing locks we hold and _after_
 	 * RecordTransactionAbort.
-	 *
-	 * Note: MyProc may be null during bootstrap.
 	 */
-	if (MyProc != NULL)
+	ProcArrayEndTransaction(MyProc, latestXid);
-	{
-		if (TransactionIdIsValid(MyProc->xid))
-		{
-			/*
-			 * We must lock ProcArrayLock while clearing MyProc->xid, so
-			 * that we do not exit the set of "running" transactions while
-			 * someone else is taking a snapshot.  See discussion in
-			 * src/backend/access/transam/README.
-			 */
-			LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
-
-			MyProc->xid = InvalidTransactionId;
-			MyProc->lxid = InvalidLocalTransactionId;
-			MyProc->xmin = InvalidTransactionId;
-			MyProc->inVacuum = false;	/* must be cleared with xid/xmin */
-			MyProc->inCommit = false;	/* be sure this gets cleared */
-
-			/* Clear the subtransaction-XID cache too while holding the lock */
-			MyProc->subxids.nxids = 0;
-			MyProc->subxids.overflowed = false;
-
-			LWLockRelease(ProcArrayLock);
-		}
-		else
-		{
-			/*
-			 * If we have no XID, we don't need to lock, since we won't
-			 * affect anyone else's calculation of a snapshot.  We might
-			 * change their estimate of global xmin, but that's OK.
-			 */
-			MyProc->lxid = InvalidLocalTransactionId;
-			MyProc->xmin = InvalidTransactionId;
-			MyProc->inVacuum = false;	/* must be cleared with xid/xmin */
-			MyProc->inCommit = false;	/* be sure this gets cleared */
-
-			Assert(MyProc->subxids.nxids == 0);
-			Assert(MyProc->subxids.overflowed == false);
-		}
-	}
 
 	/*
 	 * Post-abort cleanup.	See notes in CommitTransaction() concerning
@@ -3863,7 +3791,7 @@ AbortSubTransaction(void)
 									s->parent->subTransactionId);
 
 		/* Advertise the fact that we aborted in pg_clog. */
-		RecordTransactionAbort(true);
+		(void) RecordTransactionAbort(true);
 
 		/* Post-abort cleanup */
 		if (TransactionIdIsValid(s->transactionId))

src/backend/access/transam/xlog.c

@@ -7,7 +7,7 @@
  * Portions Copyright (c) 1996-2007, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
- * $PostgreSQL: pgsql/src/backend/access/transam/xlog.c,v 1.280 2007/09/05 18:10:47 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/access/transam/xlog.c,v 1.281 2007/09/08 20:31:14 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -5196,6 +5196,10 @@ StartupXLOG(void)
 	XLogCtl->ckptXidEpoch = ControlFile->checkPointCopy.nextXidEpoch;
 	XLogCtl->ckptXid = ControlFile->checkPointCopy.nextXid;
 
+	/* also initialize latestCompletedXid, to nextXid - 1 */
+	ShmemVariableCache->latestCompletedXid = ShmemVariableCache->nextXid;
+	TransactionIdRetreat(ShmemVariableCache->latestCompletedXid);
+
 	/* Start up the commit log and related stuff, too */
 	StartupCLOG();
 	StartupSUBTRANS(oldestActiveXID);

src/backend/catalog/heap.c

@@ -8,7 +8,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/catalog/heap.c,v 1.322 2007/06/03 22:16:02 petere Exp $
+ *	  $PostgreSQL: pgsql/src/backend/catalog/heap.c,v 1.323 2007/09/08 20:31:14 tgl Exp $
  *
  *
  * INTERFACE ROUTINES
@@ -704,10 +704,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 		 * We know that no xacts older than RecentXmin are still running,
 		 * so that will do.
 		 */
-		if (!IsBootstrapProcessingMode())
+		new_rel_reltup->relfrozenxid = RecentXmin;
-			new_rel_reltup->relfrozenxid = RecentXmin;
-		else
-			new_rel_reltup->relfrozenxid = FirstNormalTransactionId;
 	}
 	else
 	{

src/backend/commands/vacuum.c

@@ -13,7 +13,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/commands/vacuum.c,v 1.356 2007/09/05 18:10:47 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/commands/vacuum.c,v 1.357 2007/09/08 20:31:14 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -2437,14 +2437,15 @@ repair_frag(VRelStats *vacrelstats, Relation onerel,
 		 * For now, a quick hack: record status of current transaction as
 		 * committed, and continue.  We force the commit to be synchronous
 		 * so that it's down to disk before we truncate.  (Note: tqual.c
-		 * knows that VACUUM FULL always uses sync commit, too.)
+		 * knows that VACUUM FULL always uses sync commit, too.)  The
+		 * transaction continues to be shown as running in the ProcArray.
 		 *
 		 * XXX This desperately needs to be revisited.  Any failure after
 		 * this point will result in a PANIC "cannot abort transaction nnn,
 		 * it was already committed"!
 		 */
 		ForceSyncCommit();
-		RecordTransactionCommit();
+		(void) RecordTransactionCommit();
 	}
 
 	/*

src/backend/storage/ipc/procarray.c

@@ -23,7 +23,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/storage/ipc/procarray.c,v 1.32 2007/09/07 20:59:26 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/storage/ipc/procarray.c,v 1.33 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -147,9 +147,16 @@ ProcArrayAdd(PGPROC *proc)
 
 /*
  * Remove the specified PGPROC from the shared array.
+ *
+ * When latestXid is a valid XID, we are removing a live 2PC gxact from the
+ * array, and thus causing it to appear as "not running" anymore.  In this
+ * case we must advance latestCompletedXid.  (This is essentially the same
+ * as ProcArrayEndTransaction followed by removal of the PGPROC, but we take
+ * the ProcArrayLock only once, and don't damage the content of the PGPROC;
+ * twophase.c depends on the latter.)
  */
 void
-ProcArrayRemove(PGPROC *proc)
+ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
 {
 	ProcArrayStruct *arrayP = procArray;
 	int			index;
@@ -162,6 +169,21 @@ ProcArrayRemove(PGPROC *proc)
 
 	LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
 
+	if (TransactionIdIsValid(latestXid))
+	{
+		Assert(TransactionIdIsValid(proc->xid));
+
+		/* Advance global latestCompletedXid while holding the lock */
+		if (TransactionIdPrecedes(ShmemVariableCache->latestCompletedXid,
+								  latestXid))
+			ShmemVariableCache->latestCompletedXid = latestXid;
+	}
+	else
+	{
+		/* Shouldn't be trying to remove a live transaction here */
+		Assert(!TransactionIdIsValid(proc->xid));
+	}
+
 	for (index = 0; index < arrayP->numProcs; index++)
 	{
 		if (arrayP->procs[index] == proc)
@@ -180,6 +202,100 @@ ProcArrayRemove(PGPROC *proc)
 }
 
 
+/*
+ * ProcArrayEndTransaction -- mark a transaction as no longer running
+ *
+ * This is used interchangeably for commit and abort cases.  The transaction
+ * commit/abort must already be reported to WAL and pg_clog.
+ *
+ * proc is currently always MyProc, but we pass it explicitly for flexibility.
+ * latestXid is the latest Xid among the transaction's main XID and
+ * subtransactions, or InvalidTransactionId if it has no XID.  (We must ask
+ * the caller to pass latestXid, instead of computing it from the PGPROC's
+ * contents, because the subxid information in the PGPROC might be
+ * incomplete.)
+ */
+void
+ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
+{
+	if (TransactionIdIsValid(latestXid))
+	{
+		/*
+		 * We must lock ProcArrayLock while clearing proc->xid, so
+		 * that we do not exit the set of "running" transactions while
+		 * someone else is taking a snapshot.  See discussion in
+		 * src/backend/access/transam/README.
+		 */
+		Assert(TransactionIdIsValid(proc->xid));
+
+		LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+
+		proc->xid = InvalidTransactionId;
+		proc->lxid = InvalidLocalTransactionId;
+		proc->xmin = InvalidTransactionId;
+		proc->inVacuum = false;			/* must be cleared with xid/xmin */
+		proc->inCommit = false;			/* be sure this is cleared in abort */
+
+		/* Clear the subtransaction-XID cache too while holding the lock */
+		proc->subxids.nxids = 0;
+		proc->subxids.overflowed = false;
+
+		/* Also advance global latestCompletedXid while holding the lock */
+		if (TransactionIdPrecedes(ShmemVariableCache->latestCompletedXid,
+								  latestXid))
+			ShmemVariableCache->latestCompletedXid = latestXid;
+
+		LWLockRelease(ProcArrayLock);
+	}
+	else
+	{
+		/*
+		 * If we have no XID, we don't need to lock, since we won't
+		 * affect anyone else's calculation of a snapshot.  We might
+		 * change their estimate of global xmin, but that's OK.
+		 */
+		Assert(!TransactionIdIsValid(proc->xid));
+
+		proc->lxid = InvalidLocalTransactionId;
+		proc->xmin = InvalidTransactionId;
+		proc->inVacuum = false;			/* must be cleared with xid/xmin */
+		proc->inCommit = false;			/* be sure this is cleared in abort */
+
+		Assert(proc->subxids.nxids == 0);
+		Assert(proc->subxids.overflowed == false);
+	}
+}
+
+
+/*
+ * ProcArrayClearTransaction -- clear the transaction fields
+ *
+ * This is used after successfully preparing a 2-phase transaction.  We are
+ * not actually reporting the transaction's XID as no longer running --- it
+ * will still appear as running because the 2PC's gxact is in the ProcArray
+ * too.  We just have to clear out our own PGPROC.
+ */
+void
+ProcArrayClearTransaction(PGPROC *proc)
+{
+	/*
+	 * We can skip locking ProcArrayLock here, because this action does not
+	 * actually change anyone's view of the set of running XIDs: our entry
+	 * is duplicate with the gxact that has already been inserted into the
+	 * ProcArray.
+	 */
+	proc->xid = InvalidTransactionId;
+	proc->lxid = InvalidLocalTransactionId;
+	proc->xmin = InvalidTransactionId;
+	proc->inVacuum = false;			/* redundant, but just in case */
+	proc->inCommit = false;			/* ditto */
+
+	/* Clear the subtransaction-XID cache too */
+	proc->subxids.nxids = 0;
+	proc->subxids.overflowed = false;
+}
+
+
 /*
  * TransactionIdIsInProgress -- is given transaction running in some backend
  *
@@ -416,23 +532,18 @@ GetOldestXmin(bool allDbs, bool ignoreVacuum)
 	TransactionId result;
 	int			index;
 
-	/*
-	 * We need to initialize the MIN() calculation with something.
-	 * ReadNewTransactionId() is guaranteed to work, but is relatively
-	 * expensive due to locking; so first we try a couple of shortcuts.
-	 * If we have a valid xmin in our own PGPROC entry, that will do;
-	 * or if we have assigned ourselves an XID, that will do.
-	 */
-	result = MyProc ? MyProc->xmin : InvalidTransactionId;
-	if (!TransactionIdIsValid(result))
-	{
-		result = GetTopTransactionIdIfAny();
-		if (!TransactionIdIsValid(result))
-			result = ReadNewTransactionId();
-	}
-
 	LWLockAcquire(ProcArrayLock, LW_SHARED);
 
+	/*
+	 * We initialize the MIN() calculation with latestCompletedXid + 1.
+	 * This is a lower bound for the XIDs that might appear in the ProcArray
+	 * later, and so protects us against overestimating the result due to
+	 * future additions.
+	 */
+	result = ShmemVariableCache->latestCompletedXid;
+	Assert(TransactionIdIsNormal(result));
+	TransactionIdAdvance(result);
+
 	for (index = 0; index < arrayP->numProcs; index++)
 	{
 		volatile PGPROC	   *proc = arrayP->procs[index];
@@ -473,7 +584,7 @@ GetOldestXmin(bool allDbs, bool ignoreVacuum)
  * GetSnapshotData -- returns information about running transactions.
  *
  * The returned snapshot includes xmin (lowest still-running xact ID),
- * xmax (next xact ID to be assigned), and a list of running xact IDs
+ * xmax (highest completed xact ID + 1), and a list of running xact IDs
  * in the range xmin <= xid < xmax.  It is used as follows:
  *		All xact IDs < xmin are considered finished.
  *		All xact IDs >= xmax are considered still running.
@@ -555,13 +666,10 @@ GetSnapshotData(Snapshot snapshot, bool serializable)
 	 */
 	LWLockAcquire(ProcArrayLock, LW_SHARED);
 
-	/*
+	/* xmax is always latestCompletedXid + 1 */
-	 * Unfortunately, we have to call ReadNewTransactionId() after acquiring
+	xmax = ShmemVariableCache->latestCompletedXid;
-	 * ProcArrayLock.  It's not good because ReadNewTransactionId() does
+	Assert(TransactionIdIsNormal(xmax));
-	 * LWLockAcquire(XidGenLock), but *necessary*.  See discussion in
+	TransactionIdAdvance(xmax);
-	 * src/backend/access/transam/README.
-	 */
-	xmax = ReadNewTransactionId();
 
 	/* initialize xmin calculation with xmax */
 	globalxmin = xmin = xmax;
@@ -592,9 +700,8 @@ GetSnapshotData(Snapshot snapshot, bool serializable)
 		/*
 		 * If the transaction has been assigned an xid < xmax we add it to the
 		 * snapshot, and update xmin if necessary.  There's no need to store
-		 * XIDs above what we got from ReadNewTransactionId, since we'll treat
+		 * XIDs >= xmax, since we'll treat them as running anyway.  We don't
-		 * them as running anyway.  We don't bother to examine their subxids
+		 * bother to examine their subxids either.
-		 * either.
 		 *
 		 * We don't include our own XID (if any) in the snapshot, but we must
 		 * include it into xmin.
@@ -612,7 +719,9 @@ GetSnapshotData(Snapshot snapshot, bool serializable)
 		/*
 		 * Save subtransaction XIDs if possible (if we've already overflowed,
 		 * there's no point).  Note that the subxact XIDs must be later than
-		 * their parent, so no need to check them against xmin.
+		 * their parent, so no need to check them against xmin.  We could
+		 * filter against xmax, but it seems better not to do that much work
+		 * while holding the ProcArrayLock.
 		 *
 		 * The other backend can add more subxids concurrently, but cannot
 		 * remove any.	Hence it's important to fetch nxids just once. Should
@@ -1096,9 +1205,12 @@ CheckOtherDBBackends(Oid databaseId)
  * Remove a bunch of TransactionIds from the list of known-running
  * subtransactions for my backend.	Both the specified xid and those in
  * the xids[] array (of length nxids) are removed from the subxids cache.
+ * latestXid must be the latest XID among the group.
  */
 void
-XidCacheRemoveRunningXids(TransactionId xid, int nxids, TransactionId *xids)
+XidCacheRemoveRunningXids(TransactionId xid,
+						  int nxids, const TransactionId *xids,
+						  TransactionId latestXid)
 {
 	int			i,
 				j;
@@ -1155,6 +1267,11 @@ XidCacheRemoveRunningXids(TransactionId xid, int nxids, TransactionId *xids)
 	if (j < 0 && !MyProc->subxids.overflowed)
 		elog(WARNING, "did not find subXID %u in MyProc", xid);
 
+	/* Also advance global latestCompletedXid while holding the lock */
+	if (TransactionIdPrecedes(ShmemVariableCache->latestCompletedXid,
+							  latestXid))
+		ShmemVariableCache->latestCompletedXid = latestXid;
+
 	LWLockRelease(ProcArrayLock);
 }
 

src/backend/storage/lmgr/proc.c

@@ -8,7 +8,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/storage/lmgr/proc.c,v 1.193 2007/09/05 18:10:47 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/storage/lmgr/proc.c,v 1.194 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -581,7 +581,7 @@ static void
 RemoveProcFromArray(int code, Datum arg)
 {
 	Assert(MyProc != NULL);
-	ProcArrayRemove(MyProc);
+	ProcArrayRemove(MyProc, InvalidTransactionId);
 }
 
 /*

src/backend/utils/time/tqual.c

@@ -31,7 +31,7 @@
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/utils/time/tqual.c,v 1.104 2007/08/14 17:35:18 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/utils/time/tqual.c,v 1.105 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -71,10 +71,14 @@ Snapshot	LatestSnapshot = NULL;
  */
 Snapshot	ActiveSnapshot = NULL;
 
-/* These are updated by GetSnapshotData: */
+/*
-TransactionId TransactionXmin = InvalidTransactionId;
+ * These are updated by GetSnapshotData.  We initialize them this way
-TransactionId RecentXmin = InvalidTransactionId;
+ * for the convenience of TransactionIdIsInProgress: even in bootstrap
-TransactionId RecentGlobalXmin = InvalidTransactionId;
+ * mode, we don't want it to say that BootstrapTransactionId is in progress.
+ */
+TransactionId TransactionXmin = FirstNormalTransactionId;
+TransactionId RecentXmin = FirstNormalTransactionId;
+TransactionId RecentGlobalXmin = FirstNormalTransactionId;
 
 /* local functions */
 static bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);

src/include/access/transam.h

@@ -7,7 +7,7 @@
  * Portions Copyright (c) 1996-2007, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
- * $PostgreSQL: pgsql/src/include/access/transam.h,v 1.61 2007/08/01 22:45:09 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/access/transam.h,v 1.62 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -43,6 +43,7 @@
 #define TransactionIdEquals(id1, id2)	((id1) == (id2))
 #define TransactionIdStore(xid, dest)	(*(dest) = (xid))
 #define StoreInvalidTransactionId(dest) (*(dest) = InvalidTransactionId)
+
 /* advance a transaction ID variable, handling wraparound correctly */
 #define TransactionIdAdvance(dest)	\
 	do { \
@@ -51,6 +52,12 @@
 			(dest) = FirstNormalTransactionId; \
 	} while(0)
 
+/* back up a transaction ID variable, handling wraparound correctly */
+#define TransactionIdRetreat(dest)	\
+	do { \
+		(dest)--; \
+	} while ((dest) < FirstNormalTransactionId)
+
 
 /* ----------
  *		Object ID (OID) zero is InvalidOid.
@@ -78,8 +85,10 @@
 #define FirstNormalObjectId		16384
 
 /*
- * VariableCache is placed in shmem and used by
+ * VariableCache is a data structure in shared memory that is used to track
- * backends to get next available OID & XID.
+ * OID and XID assignment state.  For largely historical reasons, there is
+ * just one struct with different fields that are protected by different
+ * LWLocks.
  *
  * Note: xidWrapLimit and limit_datname are not "active" values, but are
  * used just to generate useful messages when xidWarnLimit or xidStopLimit
@@ -87,8 +96,15 @@
  */
 typedef struct VariableCacheData
 {
+	/*
+	 * These fields are protected by OidGenLock.
+	 */
 	Oid			nextOid;		/* next OID to assign */
 	uint32		oidCount;		/* OIDs available before must do XLOG work */
+
+	/*
+	 * These fields are protected by XidGenLock.
+	 */
 	TransactionId nextXid;		/* next XID to assign */
 
 	TransactionId oldestXid;	/* cluster-wide minimum datfrozenxid */
@@ -97,6 +113,12 @@ typedef struct VariableCacheData
 	TransactionId xidStopLimit; /* refuse to advance nextXid beyond here */
 	TransactionId xidWrapLimit; /* where the world ends */
 	NameData	limit_datname;	/* database that needs vacuumed first */
+
+	/*
+	 * These fields are protected by ProcArrayLock.
+	 */
+	TransactionId latestCompletedXid;	/* newest XID that has committed or
+										 * aborted */
 } VariableCacheData;
 
 typedef VariableCacheData *VariableCache;
@@ -127,6 +149,8 @@ extern bool TransactionIdPrecedes(TransactionId id1, TransactionId id2);
 extern bool TransactionIdPrecedesOrEquals(TransactionId id1, TransactionId id2);
 extern bool TransactionIdFollows(TransactionId id1, TransactionId id2);
 extern bool TransactionIdFollowsOrEquals(TransactionId id1, TransactionId id2);
+extern TransactionId TransactionIdLatest(TransactionId mainxid,
+										 int nxids, const TransactionId *xids);
 extern XLogRecPtr TransactionIdGetCommitLSN(TransactionId xid);
 
 /* in transam/varsup.c */

src/include/access/xact.h

@@ -7,7 +7,7 @@
  * Portions Copyright (c) 1996-2007, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
- * $PostgreSQL: pgsql/src/include/access/xact.h,v 1.89 2007/09/05 18:10:48 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/access/xact.h,v 1.90 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -178,7 +178,7 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
 extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
 extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
 
-extern void RecordTransactionCommit(void);
+extern TransactionId RecordTransactionCommit(void);
 
 extern int	xactGetCommittedChildren(TransactionId **ptr);
 

src/include/storage/procarray.h

@@ -7,7 +7,7 @@
  * Portions Copyright (c) 1996-2007, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
- * $PostgreSQL: pgsql/src/include/storage/procarray.h,v 1.16 2007/09/07 00:58:57 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/storage/procarray.h,v 1.17 2007/09/08 20:31:15 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -20,7 +20,10 @@
 extern Size ProcArrayShmemSize(void);
 extern void CreateSharedProcArray(void);
 extern void ProcArrayAdd(PGPROC *proc);
-extern void ProcArrayRemove(PGPROC *proc);
+extern void ProcArrayRemove(PGPROC *proc, TransactionId latestXid);
+
+extern void ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid);
+extern void ProcArrayClearTransaction(PGPROC *proc);
 
 extern bool TransactionIdIsInProgress(TransactionId xid);
 extern bool TransactionIdIsActive(TransactionId xid);
@@ -41,6 +44,7 @@ extern int	CountUserBackends(Oid roleid);
 extern bool CheckOtherDBBackends(Oid databaseId);
 
 extern void XidCacheRemoveRunningXids(TransactionId xid,
-						  int nxids, TransactionId *xids);
+									  int nxids, const TransactionId *xids,
+									  TransactionId latestXid);
 
 #endif   /* PROCARRAY_H */