rdelaage
/
postgresql
mirror of https://git.postgresql.org/git/postgresql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
postgresql/doc/TODO.detail/thread

333 lines
12 KiB
Plaintext
Raw Blame History

From mscott@sacadia.com Wed Nov 15 14:50:19 2000
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583
for <pgman@candle.pha.pa.us>; Wed, 15 Nov 2000 14:50:13 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998;
Wed, 15 Nov 2000 11:35:33 -0800 (PST)
Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>,
Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Please help with some advice
Message-ID: <Pine.GSO.4.10.10011151053260.9940-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: ORr
Dear Sirs,
I have been lurking on the PostgreSQL hackers list for about 3 months now
and your names comes up more than any with helpful info about the project
so I was hoping you could help me.
Let me cut to the chase. I have been experimenting with 7.0.2 source to
see if I could create a mutlti-threaded version of the backend so
I could link directly from java ( I have a fe<->be protocol that I use for
my apps). Needless to say I got into much more than I bargained for. I
now have a version that works and it has some nice benefits that are very
helpful to a project that I am working on. What I gained was
prepared statements outside of spi
batched commits (fsync)
one connection per thread
multiple threads per process
multiple processes per installation
I never really intended for anyone else to see the work so I drifted
pretty far from the original code. I also ended up using Solaris threads
rather than pthreads, I did my own implementation of the bufmgr.c and
gram.y, and used Solaris implementation of mutex in place of S_LOCK and
TAS. I grabbed all global variables and put them in an environment
variable that is thread local. I also did some really stupid
things like making TransactionId uint64 and making all my inserts use the
same oid.
My question is this. I would like to get some critical feedback and
suggestions about the work from others. What is the best way to go about
this? I thought about trying to create a project on greatbridge.org
but I am rather new to open source and the code needs commented properly
and cleaned up before too many try and look at it.
Any suggestions would be greatly appreciated.
Thanks in advance,
Myron Scott
From mscott@sacadia.com Thu Nov 16 17:19:45 2000
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:19:43 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449;
Thu, 16 Nov 2000 14:05:15 -0800 (PST)
Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: Please help with some advice
In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us>
Message-ID: <Pine.GSO.4.10.10011161401570.11441-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: OR
Bruce Momjian wrote:
>I am curious how you isolated each thread. It seems we pretty much
>assume all our memory is controlled by a single query in the process.
I moved all global variables to a thread global variable which is accessed
by the method GetEnv(). Which looks like this
Env* GetEnv(void) {
Env* env;
thr_getspecific(*envkey,(void*)&env);
return env;
}
The Env struct includes the CurrentMemoryContext, TopMemoryContext,
PortalHeapMemory for each instance of a connection (one thread per
connection). So, for example,
EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
void
EndPortalAllocMode()
{
PortalHeapMemory context;
AssertState(PortalManagerEnabled);
AssertState(IsA(GetEnv()->CurrentMemoryContext,
PortalHeapMemory));
context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
AssertState(PointerIsValid(context->block)); /* XXX
Trap(...) */
/* free current mode */
AllocSetReset(&HEAPMEMBLOCK(context)->setData);
MemoryContextFree((MemoryContext)
PortalHeapMemoryGetVariableMemory(context),
context->block);
/* restore previous mode */
context->block = FixedStackPop(&context->stackData);
}
From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000
Received: from sectorbase2.sectorbase.com ([208.48.122.131])
by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:23:21 -0500 (EST)
Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21)
id <V8XQB5RW>; Thu, 16 Nov 2000 14:05:24 -0800
Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com>
From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
To: "'Myron Scott'" <mscott@sacadia.com>,
Bruce Momjian
<pgman@candle.pha.pa.us>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Subject: RE: Please help with some advice
Date: Thu, 16 Nov 2000 14:09:30 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2650.21)
Content-Type: text/plain;
charset="iso-8859-1"
Status: ORr
I think the question do we want to make backend multy-threaded
should be discussed in hackers.
Vadim
> -----Original Message-----
> From: Myron Scott [mailto:mscott@sacadia.com]
> Sent: Thursday, November 16, 2000 2:05 PM
> To: Bruce Momjian
> Cc: Mikheev, Vadim; Tom Lane
> Subject: Re: Please help with some advice
>
>
> Bruce Momjian wrote:
>
> >I am curious how you isolated each thread. It seems we pretty much
> >assume all our memory is controlled by a single query in the process.
>
>
>
> I moved all global variables to a thread global variable
> which is accessed
> by the method GetEnv(). Which looks like this
>
> Env* GetEnv(void) {
> Env* env;
> thr_getspecific(*envkey,(void*)&env);
> return env;
> }
>
> The Env struct includes the CurrentMemoryContext, TopMemoryContext,
> PortalHeapMemory for each instance of a connection (one thread per
> connection). So, for example,
> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
>
> void
> EndPortalAllocMode()
> {
> PortalHeapMemory context;
>
> AssertState(PortalManagerEnabled);
> AssertState(IsA(GetEnv()->CurrentMemoryContext,
> PortalHeapMemory));
>
> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
> AssertState(PointerIsValid(context->block)); /* XXX
> Trap(...) */
>
> /* free current mode */
> AllocSetReset(&HEAPMEMBLOCK(context)->setData);
> MemoryContextFree((MemoryContext)
> PortalHeapMemoryGetVariableMemory(context),
> context->block);
>
> /* restore previous mode */
> context->block = FixedStackPop(&context->stackData);
> }
>
>
>
From mscott@sacadia.com Thu Nov 16 22:16:38 2000
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 22:16:36 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874;
Thu, 16 Nov 2000 19:04:48 -0800 (PST)
Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: Please help with some advice
In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us>
Message-ID: <Pine.GSO.4.10.10011161904140.11870-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: ORr
Thanks very much, I will post to hackers.
Myron
From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 00:30:19 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335;
Tue, 2 Jan 2001 00:30:45 -0500 (EST)
(envelope-from pgsql-hackers-owner+M2691@postgresql.org)
Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged))
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232
for <pgsql-hackers@postgresql.org>; Tue, 2 Jan 2001 00:30:32 -0500 (EST)
(envelope-from mscott@sacadia.com)
Received: from [206.112.108.0] (HELO sacadia.com)
by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2)
with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500
Message-ID: <3A5167DB.3050807@sacadia.com>
Date: Mon, 01 Jan 2001 21:32:11 -0800
From: Myron Scott <mscott@sacadia.com>
Reply-To: mscott@sacadia.com
User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0
X-Accept-Language: en
MIME-Version: 1.0
To: "Ross J. Reedstrom" <reedstrm@rice.edu>
CC: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Using Threads?
References: <004401c058fd$fd498d40$f2356880@tracy> <Pine.GSO.4.10.10012032351040.28161-100000@goldengate.kojoworldwide.com.> <20001204113307.B5871@rice.edu>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
For anyone interested,
I have posted my multi-threaded version of PostgreSQL here.
http://www.sacadia.com/mtpg.html
It is based on 7.0.2 and the TAO CORBA ORB which is here.
http://www.cs.wustl.edu/~schmidt/TAO.html
Myron Scott
mkscott@sacadia.com
From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001
Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 03:02:27 -0500 (EST)
Received: (from bright@localhost)
by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623;
Tue, 2 Jan 2001 00:02:31 -0800 (PST)
Date: Tue, 2 Jan 2001 00:02:31 -0800
From: Alfred Perlstein <bright@wintelcom.net>
To: Bruce Momjian <pgman@candle.pha.pa.us>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten
Message-ID: <20010102000230.C19572@fw.wintelcom.net>
References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500
Status: OR
* Bruce Momjian <pgman@candle.pha.pa.us> [010101 23:59] wrote:
> > Alfred Perlstein <bright@wintelcom.net> writes:
> > > One trick that may help is calling sched_yield(2) on a lock miss,
> > > it's a POSIX call and quite new so you'd need a 'configure' test
> > > for it.
> >
> > The author of the current s_lock code seems to have thought that
> > select() with a zero delay would do the equivalent of sched_yield().
> > I'm not sure if that's true on very many kernels, if indeed any...
> >
> > I doubt we could buy much by depending on sched_yield(); if you want
> > to assume POSIX facilities, ISTM you might as well go for user-space
> > semaphores and forget the whole TAS mechanism.
>
>
> Another issue is that sched_yield brings in the pthreads library/hooks
> on some OS's, which we certainly want to avoid.
I know it's a major undertaking, but since the work is sort of done,
have you guys considered the port to solaris threads and seeing about
making a pthreads port of that?
I know it would probably get you considerable gains under Windows
at the expense of dropping some really really legacy system.
Or you could do what apache (is rumored) does and have it do either
threads or processes or both...
--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."
Reference in New Issue View Git Blame Copy Permalink