mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-08-22 03:13:25 +02:00
1430 lines
63 KiB
Plaintext
1430 lines
63 KiB
Plaintext
From mscott@sacadia.com Wed Nov 15 14:50:19 2000
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583
|
|
for <pgman@candle.pha.pa.us>; Wed, 15 Nov 2000 14:50:13 -0500 (EST)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998;
|
|
Wed, 15 Nov 2000 11:35:33 -0800 (PST)
|
|
Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>,
|
|
Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>
|
|
Subject: Please help with some advice
|
|
Message-ID: <Pine.GSO.4.10.10011151053260.9940-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Status: ORr
|
|
|
|
Dear Sirs,
|
|
|
|
I have been lurking on the PostgreSQL hackers list for about 3 months now
|
|
and your names comes up more than any with helpful info about the project
|
|
so I was hoping you could help me.
|
|
|
|
Let me cut to the chase. I have been experimenting with 7.0.2 source to
|
|
see if I could create a mutlti-threaded version of the backend so
|
|
I could link directly from java ( I have a fe<->be protocol that I use for
|
|
my apps). Needless to say I got into much more than I bargained for. I
|
|
now have a version that works and it has some nice benefits that are very
|
|
helpful to a project that I am working on. What I gained was
|
|
|
|
prepared statements outside of spi
|
|
batched commits (fsync)
|
|
one connection per thread
|
|
multiple threads per process
|
|
multiple processes per installation
|
|
|
|
I never really intended for anyone else to see the work so I drifted
|
|
pretty far from the original code. I also ended up using Solaris threads
|
|
rather than pthreads, I did my own implementation of the bufmgr.c and
|
|
gram.y, and used Solaris implementation of mutex in place of S_LOCK and
|
|
TAS. I grabbed all global variables and put them in an environment
|
|
variable that is thread local. I also did some really stupid
|
|
things like making TransactionId uint64 and making all my inserts use the
|
|
same oid.
|
|
|
|
My question is this. I would like to get some critical feedback and
|
|
suggestions about the work from others. What is the best way to go about
|
|
this? I thought about trying to create a project on greatbridge.org
|
|
but I am rather new to open source and the code needs commented properly
|
|
and cleaned up before too many try and look at it.
|
|
|
|
Any suggestions would be greatly appreciated.
|
|
|
|
|
|
Thanks in advance,
|
|
|
|
Myron Scott
|
|
|
|
|
|
|
|
From mscott@sacadia.com Thu Nov 16 17:19:45 2000
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315
|
|
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:19:43 -0500 (EST)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449;
|
|
Thu, 16 Nov 2000 14:05:15 -0800 (PST)
|
|
Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>
|
|
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
|
|
Subject: Re: Please help with some advice
|
|
In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us>
|
|
Message-ID: <Pine.GSO.4.10.10011161401570.11441-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Status: OR
|
|
|
|
Bruce Momjian wrote:
|
|
|
|
>I am curious how you isolated each thread. It seems we pretty much
|
|
>assume all our memory is controlled by a single query in the process.
|
|
|
|
|
|
I moved all global variables to a thread global variable which is accessed
|
|
by the method GetEnv(). Which looks like this
|
|
|
|
Env* GetEnv(void) {
|
|
Env* env;
|
|
thr_getspecific(*envkey,(void*)&env);
|
|
return env;
|
|
}
|
|
|
|
The Env struct includes the CurrentMemoryContext, TopMemoryContext,
|
|
PortalHeapMemory for each instance of a connection (one thread per
|
|
connection). So, for example,
|
|
EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
|
|
|
|
void
|
|
EndPortalAllocMode()
|
|
{
|
|
PortalHeapMemory context;
|
|
|
|
AssertState(PortalManagerEnabled);
|
|
AssertState(IsA(GetEnv()->CurrentMemoryContext,
|
|
PortalHeapMemory));
|
|
|
|
context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
|
|
AssertState(PointerIsValid(context->block)); /* XXX
|
|
Trap(...) */
|
|
|
|
/* free current mode */
|
|
AllocSetReset(&HEAPMEMBLOCK(context)->setData);
|
|
MemoryContextFree((MemoryContext)
|
|
PortalHeapMemoryGetVariableMemory(context),
|
|
context->block);
|
|
|
|
/* restore previous mode */
|
|
context->block = FixedStackPop(&context->stackData);
|
|
}
|
|
|
|
|
|
|
|
|
|
From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000
|
|
Received: from sectorbase2.sectorbase.com ([208.48.122.131])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562
|
|
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:23:21 -0500 (EST)
|
|
Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21)
|
|
id <V8XQB5RW>; Thu, 16 Nov 2000 14:05:24 -0800
|
|
Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com>
|
|
From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
|
|
To: "'Myron Scott'" <mscott@sacadia.com>,
|
|
Bruce Momjian
|
|
<pgman@candle.pha.pa.us>
|
|
Cc: Tom Lane <tgl@sss.pgh.pa.us>
|
|
Subject: RE: Please help with some advice
|
|
Date: Thu, 16 Nov 2000 14:09:30 -0800
|
|
MIME-Version: 1.0
|
|
X-Mailer: Internet Mail Service (5.5.2650.21)
|
|
Content-Type: text/plain;
|
|
charset="iso-8859-1"
|
|
Status: ORr
|
|
|
|
I think the question do we want to make backend multy-threaded
|
|
should be discussed in hackers.
|
|
|
|
Vadim
|
|
|
|
> -----Original Message-----
|
|
> From: Myron Scott [mailto:mscott@sacadia.com]
|
|
> Sent: Thursday, November 16, 2000 2:05 PM
|
|
> To: Bruce Momjian
|
|
> Cc: Mikheev, Vadim; Tom Lane
|
|
> Subject: Re: Please help with some advice
|
|
>
|
|
>
|
|
> Bruce Momjian wrote:
|
|
>
|
|
> >I am curious how you isolated each thread. It seems we pretty much
|
|
> >assume all our memory is controlled by a single query in the process.
|
|
>
|
|
>
|
|
>
|
|
> I moved all global variables to a thread global variable
|
|
> which is accessed
|
|
> by the method GetEnv(). Which looks like this
|
|
>
|
|
> Env* GetEnv(void) {
|
|
> Env* env;
|
|
> thr_getspecific(*envkey,(void*)&env);
|
|
> return env;
|
|
> }
|
|
>
|
|
> The Env struct includes the CurrentMemoryContext, TopMemoryContext,
|
|
> PortalHeapMemory for each instance of a connection (one thread per
|
|
> connection). So, for example,
|
|
> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
|
|
>
|
|
> void
|
|
> EndPortalAllocMode()
|
|
> {
|
|
> PortalHeapMemory context;
|
|
>
|
|
> AssertState(PortalManagerEnabled);
|
|
> AssertState(IsA(GetEnv()->CurrentMemoryContext,
|
|
> PortalHeapMemory));
|
|
>
|
|
> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
|
|
> AssertState(PointerIsValid(context->block)); /* XXX
|
|
> Trap(...) */
|
|
>
|
|
> /* free current mode */
|
|
> AllocSetReset(&HEAPMEMBLOCK(context)->setData);
|
|
> MemoryContextFree((MemoryContext)
|
|
> PortalHeapMemoryGetVariableMemory(context),
|
|
> context->block);
|
|
>
|
|
> /* restore previous mode */
|
|
> context->block = FixedStackPop(&context->stackData);
|
|
> }
|
|
>
|
|
>
|
|
>
|
|
|
|
From mscott@sacadia.com Thu Nov 16 22:16:38 2000
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638
|
|
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 22:16:36 -0500 (EST)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874;
|
|
Thu, 16 Nov 2000 19:04:48 -0800 (PST)
|
|
Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>
|
|
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
|
|
Subject: Re: Please help with some advice
|
|
In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us>
|
|
Message-ID: <Pine.GSO.4.10.10011161904140.11870-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Status: ORr
|
|
|
|
Thanks very much, I will post to hackers.
|
|
|
|
Myron
|
|
|
|
|
|
|
|
From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195
|
|
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 00:30:19 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335;
|
|
Tue, 2 Jan 2001 00:30:45 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M2691@postgresql.org)
|
|
Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged))
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232
|
|
for <pgsql-hackers@postgresql.org>; Tue, 2 Jan 2001 00:30:32 -0500 (EST)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from [206.112.108.0] (HELO sacadia.com)
|
|
by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2)
|
|
with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500
|
|
Message-ID: <3A5167DB.3050807@sacadia.com>
|
|
Date: Mon, 01 Jan 2001 21:32:11 -0800
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
Reply-To: mscott@sacadia.com
|
|
User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0
|
|
X-Accept-Language: en
|
|
MIME-Version: 1.0
|
|
To: "Ross J. Reedstrom" <reedstrm@rice.edu>
|
|
CC: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Using Threads?
|
|
References: <004401c058fd$fd498d40$f2356880@tracy> <Pine.GSO.4.10.10012032351040.28161-100000@goldengate.kojoworldwide.com.> <20001204113307.B5871@rice.edu>
|
|
Content-Type: text/plain; charset=us-ascii; format=flowed
|
|
Content-Transfer-Encoding: 7bit
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
For anyone interested,
|
|
|
|
I have posted my multi-threaded version of PostgreSQL here.
|
|
|
|
http://www.sacadia.com/mtpg.html
|
|
|
|
It is based on 7.0.2 and the TAO CORBA ORB which is here.
|
|
|
|
http://www.cs.wustl.edu/~schmidt/TAO.html
|
|
|
|
Myron Scott
|
|
mkscott@sacadia.com
|
|
|
|
|
|
|
|
From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001
|
|
Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169
|
|
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 03:02:27 -0500 (EST)
|
|
Received: (from bright@localhost)
|
|
by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623;
|
|
Tue, 2 Jan 2001 00:02:31 -0800 (PST)
|
|
Date: Tue, 2 Jan 2001 00:02:31 -0800
|
|
From: Alfred Perlstein <bright@wintelcom.net>
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>
|
|
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten
|
|
Message-ID: <20010102000230.C19572@fw.wintelcom.net>
|
|
References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us>
|
|
Mime-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
User-Agent: Mutt/1.2.5i
|
|
In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500
|
|
Status: OR
|
|
|
|
* Bruce Momjian <pgman@candle.pha.pa.us> [010101 23:59] wrote:
|
|
> > Alfred Perlstein <bright@wintelcom.net> writes:
|
|
> > > One trick that may help is calling sched_yield(2) on a lock miss,
|
|
> > > it's a POSIX call and quite new so you'd need a 'configure' test
|
|
> > > for it.
|
|
> >
|
|
> > The author of the current s_lock code seems to have thought that
|
|
> > select() with a zero delay would do the equivalent of sched_yield().
|
|
> > I'm not sure if that's true on very many kernels, if indeed any...
|
|
> >
|
|
> > I doubt we could buy much by depending on sched_yield(); if you want
|
|
> > to assume POSIX facilities, ISTM you might as well go for user-space
|
|
> > semaphores and forget the whole TAS mechanism.
|
|
>
|
|
>
|
|
> Another issue is that sched_yield brings in the pthreads library/hooks
|
|
> on some OS's, which we certainly want to avoid.
|
|
|
|
I know it's a major undertaking, but since the work is sort of done,
|
|
have you guys considered the port to solaris threads and seeing about
|
|
making a pthreads port of that?
|
|
|
|
I know it would probably get you considerable gains under Windows
|
|
at the expense of dropping some really really legacy system.
|
|
|
|
Or you could do what apache (is rumored) does and have it do either
|
|
threads or processes or both...
|
|
|
|
--
|
|
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
|
|
"I have the heart of a child; I keep it in a jar on my desk."
|
|
|
|
From pgsql-hackers-owner+M4275@postgresql.org Mon Feb 5 21:45:00 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA09262
|
|
for <pgman@candle.pha.pa.us>; Mon, 5 Feb 2001 21:44:59 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f162ixx00920;
|
|
Mon, 5 Feb 2001 21:44:59 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M4275@postgresql.org)
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f162fSx00595
|
|
for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 21:41:29 -0500 (EST)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id SAA03298
|
|
for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 18:25:05 -0800 (PST)
|
|
Date: Mon, 5 Feb 2001 18:25:05 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Using Threads?
|
|
Message-ID: <Pine.GSO.4.10.10102051823210.3289-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
I have put a new version of my multi-threaded
|
|
postgresql experiment at
|
|
|
|
http://www.sacadia.com/mtpg.html
|
|
|
|
This one actually works. I have added a server
|
|
based on omniORB, a CORBA 2.3 ORB from ATT. It
|
|
is much smaller than TAO and uses the thread per
|
|
connection model. I haven't added the java side
|
|
of the JNI interface yet but the C++ side is there.
|
|
|
|
It's still not stable but it is much better than
|
|
the last.
|
|
|
|
Myron Scott
|
|
mkscott@sacadia.com
|
|
|
|
|
|
|
|
|
|
|
|
From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
|
|
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
|
|
Tue, 6 Feb 2001 10:24:11 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M4304@postgresql.org)
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
|
|
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
|
|
Tue, 6 Feb 2001 07:05:04 -0800 (PST)
|
|
Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: Karel Zak <zakkr@zf.jcu.cz>
|
|
cc: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Using Threads
|
|
In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
|
|
Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
|
|
>
|
|
> Sorry I haven't time to see and test your experiment,
|
|
> but I have a question. How you solve memory management?
|
|
> The current mmgr is based on global variable
|
|
> CurrentMemoryContext that is very often changed and used.
|
|
> Use you for this locks? If yes it is probably problematic
|
|
> point for perfomance.
|
|
>
|
|
> Karel
|
|
>
|
|
|
|
There are many many globals I had to work around including all the memory
|
|
management stuff. I basically threw everything into and "environment"
|
|
variable which I stored in a thread specific using thr_setspecific.
|
|
|
|
Performance is acually very good for what I am doing. I was able to batch
|
|
commit transactions which cuts down on fsync calls, use prepared
|
|
statements from my client using CORBA, and the various locking calls for
|
|
the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
|
|
some performance tests for inserts
|
|
|
|
20 clients, 900 inserts per client, 1 insert per transaction, 4 different
|
|
tables.
|
|
|
|
7.0.2 About 10:52 average completion
|
|
multi-threaded 2:42 average completion
|
|
7.1beta3 1:13 average completion
|
|
|
|
If I increased the number of inserts per transaction, multi-threaded got
|
|
closer to 7.1 for inserts. I haven't tested other other types of
|
|
commands
|
|
yet.
|
|
|
|
|
|
Myron Scott
|
|
mkscott@sacadia.com
|
|
|
|
|
|
From pgsql-hackers-owner+M4313@postgresql.org Tue Feb 6 12:32:00 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA29163
|
|
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 12:31:59 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16HVox17454;
|
|
Tue, 6 Feb 2001 12:31:51 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M4313@postgresql.org)
|
|
Received: from ara.zf.jcu.cz (ara.zf.jcu.cz [160.217.161.4])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16HV6x17323
|
|
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 12:31:06 -0500 (EST)
|
|
(envelope-from zakkr@zf.jcu.cz)
|
|
Received: from localhost (zakkr@localhost)
|
|
by ara.zf.jcu.cz (8.9.3/8.9.3/Debian 8.9.3-21) with SMTP id SAA03980;
|
|
Tue, 6 Feb 2001 18:31:02 +0100
|
|
Date: Tue, 6 Feb 2001 18:31:02 +0100 (CET)
|
|
From: Karel Zak <zakkr@zf.jcu.cz>
|
|
To: Myron Scott <mscott@sacadia.com>
|
|
cc: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Using Threads
|
|
In-Reply-To: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
|
|
Message-ID: <Pine.LNX.3.96.1010206182112.3799B-100000@ara.zf.jcu.cz>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
|
|
On Tue, 6 Feb 2001, Myron Scott wrote:
|
|
|
|
> There are many many globals I had to work around including all the memory
|
|
> management stuff. I basically threw everything into and "environment"
|
|
> variable which I stored in a thread specific using thr_setspecific.
|
|
|
|
Yes, it's good. I working on multi-thread application server
|
|
(http://mape.jcu.cz) and I use for this project some things from PG (like
|
|
mmgr), I planning use same solution.
|
|
|
|
> Performance is acually very good for what I am doing. I was able to batch
|
|
> commit transactions which cuts down on fsync calls, use prepared
|
|
> statements from my client using CORBA, and the various locking calls for
|
|
> the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
|
|
> some performance tests for inserts
|
|
>
|
|
> 20 clients, 900 inserts per client, 1 insert per transaction, 4 different
|
|
> tables.
|
|
>
|
|
> 7.0.2 About 10:52 average completion
|
|
> multi-threaded 2:42 average completion
|
|
> 7.1beta3 1:13 average completion
|
|
|
|
It is very very good for time for 7.1, already look forward to 7.2! :-)
|
|
|
|
BTW, I not sure if you anytime in future will see threads in
|
|
official PostgreSQL and if you spending time on relevant things (IMHO).
|
|
|
|
Karel
|
|
|
|
|
|
|
|
|
|
|
|
|
|
From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
|
|
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
|
|
Tue, 6 Feb 2001 10:24:11 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M4304@postgresql.org)
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
|
|
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
|
|
Tue, 6 Feb 2001 07:05:04 -0800 (PST)
|
|
Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: Karel Zak <zakkr@zf.jcu.cz>
|
|
cc: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Using Threads
|
|
In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
|
|
Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
|
|
>
|
|
> Sorry I haven't time to see and test your experiment,
|
|
> but I have a question. How you solve memory management?
|
|
> The current mmgr is based on global variable
|
|
> CurrentMemoryContext that is very often changed and used.
|
|
> Use you for this locks? If yes it is probably problematic
|
|
> point for perfomance.
|
|
>
|
|
> Karel
|
|
>
|
|
|
|
There are many many globals I had to work around including all the memory
|
|
management stuff. I basically threw everything into and "environment"
|
|
variable which I stored in a thread specific using thr_setspecific.
|
|
|
|
Performance is acually very good for what I am doing. I was able to batch
|
|
commit transactions which cuts down on fsync calls, use prepared
|
|
statements from my client using CORBA, and the various locking calls for
|
|
the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
|
|
some performance tests for inserts
|
|
|
|
20 clients, 900 inserts per client, 1 insert per transaction, 4 different
|
|
tables.
|
|
|
|
7.0.2 About 10:52 average completion
|
|
multi-threaded 2:42 average completion
|
|
7.1beta3 1:13 average completion
|
|
|
|
If I increased the number of inserts per transaction, multi-threaded got
|
|
closer to 7.1 for inserts. I haven't tested other other types of
|
|
commands
|
|
yet.
|
|
|
|
|
|
Myron Scott
|
|
mkscott@sacadia.com
|
|
|
|
|
|
From lamar.owen@wgcr.org Thu Jun 28 11:14:10 2001
|
|
Return-path: <lamar.owen@wgcr.org>
|
|
Received: from www.wgcr.org (IDENT:root@www.wgcr.org [206.74.232.194])
|
|
by candle.pha.pa.us (8.10.1/8.10.1) with ESMTP id f5SFE9U18758
|
|
for <pgman@candle.pha.pa.us>; Thu, 28 Jun 2001 11:14:09 -0400 (EDT)
|
|
Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3])
|
|
by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id LAA11879;
|
|
Thu, 28 Jun 2001 11:14:14 -0400
|
|
Content-Type: text/plain;
|
|
charset="iso-8859-1"
|
|
From: Lamar Owen <lamar.owen@wgcr.org>
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>
|
|
Subject: Process weight (was:Re: [GENERAL] Re: Red Hat to support PostgreSQL)
|
|
Date: Thu, 28 Jun 2001 11:14:09 -0400
|
|
X-Mailer: KMail [version 1.2]
|
|
References: <200106272258.f5RMwIb26959@candle.pha.pa.us>
|
|
In-Reply-To: <200106272258.f5RMwIb26959@candle.pha.pa.us>
|
|
MIME-Version: 1.0
|
|
Message-ID: <01062811140902.01118@lowen.wgcr.org>
|
|
Content-Transfer-Encoding: 8bit
|
|
Status: ORr
|
|
|
|
On Wednesday 27 June 2001 18:58, Bruce Momjian wrote:
|
|
> > I had almost given up on using Postgres for this system because under
|
|
> > Solaris, it just couldn't cut it (MySQL could do the work with one CPU
|
|
> > while Postgres took up even more CPU and required *both* CPUs to be
|
|
> > enabled), but when we moved the system to a Linux box, things worked
|
|
> > much better.
|
|
|
|
> Ah, back to a PostgreSQL topic. :-)
|
|
|
|
> My guess on this one is that Solaris is slower for PostgreSQL because
|
|
> process switching is _much_ heavier on Solaris than other OS's. This is
|
|
> because of the way they implemented processes in SVr4. They got quite
|
|
> heavy, almost requiring kernel threads so you weren't switching
|
|
> processes all the time.
|
|
|
|
Now, the question of the week:
|
|
Is supporting a thread model for an inefficient OS a desirable thing to do,
|
|
when more efficient OS kernels are available such as FreeBSD 4.x and Linux
|
|
2.4? My opinion is that our existing model, when used with a
|
|
connection-pooling frontend, is rather efficient. (Yes, I use a
|
|
connection-pooling frontend. Performance is rather nice, and I don't have to
|
|
have a full backend spawned for every page hit.)
|
|
|
|
In fact, on a Linux box threads show as processes. While I know that the
|
|
kernel actually supports themin a slightly different manner than processes,
|
|
they have more similarities than differences.
|
|
|
|
However, even on OS's where threads are supported, the mechanism to support
|
|
those threads must be an efficient one -- not all pthreads libraries are
|
|
created equal. Many are frontends (expensive ones, at that) for plain old
|
|
processes.
|
|
|
|
Does anyone know of a resource that details the 'weight' of processes for our
|
|
supported platforms? [reply off-list -- I'll be glad to summarize responses
|
|
to HACKERS, ADMIN, or PORTS, as appropriate, if desired.]
|
|
--
|
|
Lamar Owen
|
|
WGCR Internet Radio
|
|
1 Peter 4:11
|
|
|
|
From pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:25:32 2001
|
|
Return-path: <pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLPWo07589
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:25:32 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLPf405606
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 16:25:41 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QKj3h82020
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 16:45:03 -0400 (EDT)
|
|
(envelope-from markw@mohawksoft.com)
|
|
Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1])
|
|
by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id QAA23693;
|
|
Wed, 26 Sep 2001 16:43:02 -0400
|
|
Message-ID: <3BB23DD6.E86AF327@mohawksoft.com>
|
|
Date: Wed, 26 Sep 2001 16:43:02 -0400
|
|
From: mlw <markw@mohawksoft.com>
|
|
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686)
|
|
X-Accept-Language: en
|
|
MIME-Version: 1.0
|
|
To: "D. Hageman" <dhageman@dracken.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
References: <Pine.LNX.4.33.0109261330030.1906-100000@typhon.dracken.com>
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Transfer-Encoding: 7bit
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
"D. Hageman" wrote:
|
|
|
|
> The plan for the new spinlocks does look like it has some potential. My
|
|
> only comment in regards to permformance when we start looking at SMP
|
|
> machines is ... it is my belief that getting a true threaded backend may
|
|
> be the only way to get the full potential out of SMP machines. I see that
|
|
> is one of the things to experiment with on the TODO list and I have seen
|
|
> some people have messed around already with this using Solaris threads.
|
|
> It should probably be attempted with pthreads if PostgreSQL is going to
|
|
> keep some resemblance of cross-platform compatibility. At that time, it
|
|
> would probably be easier to go in and clean up some stuff for the
|
|
> implementation of other TODO items (put in the base framework for more
|
|
> complex future items) as threading the backend would take a little bit of
|
|
> ideology shift.
|
|
|
|
I can only think of two objectives for threading. (1) running the various
|
|
connections in their own thread instead of their own process. (2) running
|
|
complex queries across multiple threads.
|
|
|
|
For item (1) I see no value to this. It is a lot of work with no tangible
|
|
benefit. If you have an old fashion pthreads implementation, it will hurt
|
|
performance because are scheduled within the single process's time slice.. If
|
|
you have a newer kernel scheduled implementation, then you will have the same
|
|
scheduling as separate processes. The only thing you will need to do is
|
|
switch your brain from figuring out how to share data, to trying to figure
|
|
out how to isolate data. A multithreaded implementation lacks many of the
|
|
benefits and robustness of a multiprocess implementation.
|
|
|
|
For item (2) I can see how that could speed up queries in a low utilization
|
|
system, and that would be cool, but in a server that is under load, threading
|
|
the queries probably be less efficient.
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
|
|
|
|
From pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:40:26 2001
|
|
Return-path: <pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMePo13437
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 18:40:25 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMeZ417944
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:40:35 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from foghorn.airs.com (foghorn.airs.com [63.201.54.26])
|
|
by postgresql.org (8.11.3/8.11.4) with SMTP id f8QM59h01247
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 18:05:09 -0400 (EDT)
|
|
(envelope-from ian@airs.com)
|
|
Received: (qmail 10089 invoked by uid 10); 26 Sep 2001 22:04:49 -0000
|
|
Received: (qmail 6837 invoked by uid 269); 26 Sep 2001 22:04:41 -0000
|
|
Mail-Followup-To: markw@mohawksoft.com,
|
|
pgsql-hackers@postgresql.org,
|
|
dhageman@dracken.com
|
|
To: "D. Hageman" <dhageman@dracken.com>
|
|
cc: mlw <markw@mohawksoft.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
References: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
|
|
From: Ian Lance Taylor <ian@airs.com>
|
|
Date: 26 Sep 2001 15:04:41 -0700
|
|
In-Reply-To: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
|
|
Message-ID: <si8zf1vcau.fsf@daffy.airs.com>
|
|
Lines: 45
|
|
User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
"D. Hageman" <dhageman@dracken.com> writes:
|
|
|
|
> > you have a newer kernel scheduled implementation, then you will have the same
|
|
> > scheduling as separate processes. The only thing you will need to do is
|
|
> > switch your brain from figuring out how to share data, to trying to figure
|
|
> > out how to isolate data. A multithreaded implementation lacks many of the
|
|
> > benefits and robustness of a multiprocess implementation.
|
|
>
|
|
> Save for the fact that the kernel can switch between threads faster then
|
|
> it can switch processes considering threads share the same address space,
|
|
> stack, code, etc. If need be sharing the data between threads is much
|
|
> easier then sharing between processes.
|
|
|
|
When using a kernel threading model, it's not obvious to me that the
|
|
kernel will switch between threads much faster than it will switch
|
|
between processes. As far as I can see, the only potential savings is
|
|
not reloading the pointers to the page tables. That is not nothing,
|
|
but it is also not a lot.
|
|
|
|
> I can't comment on the "isolate data" line. I am still trying to figure
|
|
> that one out.
|
|
|
|
Sometimes you need data which is specific to a particular thread.
|
|
Basically, you have to look at every global variable in the Postgres
|
|
backend, and determine whether to share it among all threads or to
|
|
make it thread-specific. In other words, you have to take extra steps
|
|
to isolate the data within the thread. This is the reverse of the
|
|
current situation, in which you have to take extra steps to share data
|
|
among all backend processes.
|
|
|
|
> That last line is a troll if I every saw it ;-) I will agree that threads
|
|
> isn't for everything and that it has costs just like everything else. Let
|
|
> me stress that last part - like everything else. Certain costs exist in
|
|
> the present model, nothing is - how should we say ... perfect.
|
|
|
|
When writing in C, threading inevitably loses robustness. Erratic
|
|
behaviour by one thread, perhaps in a user defined function, can
|
|
subtly corrupt the entire system, rather than just that thread. Part
|
|
of defensive programming is building barriers between different parts
|
|
of a system. Process boundaries are a powerful barrier.
|
|
|
|
(Actually, though, Postgres is already vulnerable to erratic behaviour
|
|
because any backend process can corrupt the shared buffer pool.)
|
|
|
|
Ian
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
|
|
|
|
From pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:54:58 2001
|
|
Return-path: <pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMsvo14061
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 18:54:57 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMt7420740
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:55:07 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QMOPh04333
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 18:24:26 -0400 (EDT)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id PAA00633
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 15:03:00 -0700 (PDT)
|
|
Date: Wed, 26 Sep 2001 15:03:00 -0700 (PDT)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com>
|
|
Message-ID: <Pine.GSO.4.10.10109261428340.563-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
|
|
|
|
On Wed, 26 Sep 2001, mlw wrote:
|
|
|
|
> I can only think of two objectives for threading. (1) running the various
|
|
> connections in their own thread instead of their own process. (2) running
|
|
> complex queries across multiple threads.
|
|
>
|
|
|
|
I did a multi-threaded version of 7.0.2 using Solaris threads about a year
|
|
ago in order to try
|
|
and get multiple backend connections working under one java process using
|
|
jni. I used the thread per connection model.
|
|
|
|
I eventually got it working, but it was/is very messy ( there were global
|
|
variables everywhere! ). Anyway, I was able to get a pretty good speed up
|
|
on inserts by scheduling buffer writes from multiple connections on one
|
|
common writing thread.
|
|
|
|
I also got some other features that were important to me at the time.
|
|
|
|
1. True prepared statements under java with bound input and output
|
|
variables
|
|
2. Better system utilization
|
|
a. fewer Solaris lightweight processes mapped to threads.
|
|
b. Fewer open files per postgres installation
|
|
3. Automatic vacuums when system activity is low by a daemon thread.
|
|
|
|
but there were some drawbacks... One rogue thread or bad user
|
|
function could take down all connections for that process. This
|
|
was and seems to still be the major drawback to using threads.
|
|
|
|
|
|
Myron Scott
|
|
mscott@sacadia.com
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 5: Have you checked our extensive FAQ?
|
|
|
|
http://www.postgresql.org/users-lounge/docs/faq.html
|
|
|
|
From pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:45:26 2001
|
|
Return-path: <pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLjQo08483
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:45:26 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLjY409914
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 16:45:35 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from typhon.dracken.com (dv07m61.lawrence.ks.us [24.124.61.35])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QLGDh91021
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 17:16:13 -0400 (EDT)
|
|
(envelope-from dhageman@dracken.com)
|
|
Received: from localhost (dhageman@localhost)
|
|
by typhon.dracken.com (8.11.4/8.11.4) with ESMTP id f8QLEMY01973;
|
|
Wed, 26 Sep 2001 16:14:22 -0500
|
|
X-Authentication-Warning: typhon.dracken.com: dhageman owned process doing -bs
|
|
Date: Wed, 26 Sep 2001 16:14:22 -0500 (CDT)
|
|
From: "D. Hageman" <dhageman@dracken.com>
|
|
To: mlw <markw@mohawksoft.com>
|
|
cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com>
|
|
Message-ID: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: ORr
|
|
|
|
On Wed, 26 Sep 2001, mlw wrote:
|
|
>
|
|
> I can only think of two objectives for threading. (1) running the various
|
|
> connections in their own thread instead of their own process. (2) running
|
|
> complex queries across multiple threads.
|
|
>
|
|
> For item (1) I see no value to this. It is a lot of work with no tangible
|
|
> benefit. If you have an old fashion pthreads implementation, it will hurt
|
|
> performance because are scheduled within the single process's time slice..
|
|
|
|
Old fashion ... as in a userland library that implements POSIX threads?
|
|
Well, I would agree. However, most *modern* implementations are done in
|
|
the kernel or kernel and userland coop model and don't have this
|
|
limitation (as you mention later in your e-mail). You have kinda hit on
|
|
one of my gripes about computers in general. At what point in time does
|
|
one say something is obsolete or too old to support anymore - that it
|
|
hinders progress instead of adding a "feature"?
|
|
|
|
> you have a newer kernel scheduled implementation, then you will have the same
|
|
> scheduling as separate processes. The only thing you will need to do is
|
|
> switch your brain from figuring out how to share data, to trying to figure
|
|
> out how to isolate data. A multithreaded implementation lacks many of the
|
|
> benefits and robustness of a multiprocess implementation.
|
|
|
|
Save for the fact that the kernel can switch between threads faster then
|
|
it can switch processes considering threads share the same address space,
|
|
stack, code, etc. If need be sharing the data between threads is much
|
|
easier then sharing between processes.
|
|
|
|
I can't comment on the "isolate data" line. I am still trying to figure
|
|
that one out.
|
|
|
|
That last line is a troll if I every saw it ;-) I will agree that threads
|
|
isn't for everything and that it has costs just like everything else. Let
|
|
me stress that last part - like everything else. Certain costs exist in
|
|
the present model, nothing is - how should we say ... perfect.
|
|
|
|
> For item (2) I can see how that could speed up queries in a low utilization
|
|
> system, and that would be cool, but in a server that is under load, threading
|
|
> the queries probably be less efficient.
|
|
|
|
Well, I don't follow your logic and you didn't give any substance to back
|
|
up your claim. I am willing to listen.
|
|
|
|
Another thought ... Oracle uses threads doesn't it or at least it has a
|
|
single processor and multi-processor version last time I knew ... which do
|
|
they claim is better? (Not saying that Oracle's proclimation of what is
|
|
good and what is not matters, but it is good for another view point).
|
|
|
|
--
|
|
//========================================================\\
|
|
|| D. Hageman <dhageman@dracken.com> ||
|
|
\\========================================================//
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
|
|
|
|
From pgsql-hackers-owner+M13607=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 19:14:59 2001
|
|
Return-path: <pgsql-hackers-owner+M13607=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QNExo15536
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 19:14:59 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QNF8423944
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 18:15:09 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13607=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from belphigor.mcnaught.org ([216.151.155.121])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QMe3h07256
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 18:40:04 -0400 (EDT)
|
|
(envelope-from doug@wireboard.com)
|
|
Received: (from doug@localhost)
|
|
by belphigor.mcnaught.org (8.11.6/8.9.3) id f8QMdkB05502;
|
|
Wed, 26 Sep 2001 18:39:46 -0400
|
|
X-Authentication-Warning: belphigor.mcnaught.org: doug set sender to doug@wireboard.com using -f
|
|
To: "D. Hageman" <dhageman@dracken.com>
|
|
cc: mlw <markw@mohawksoft.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
References: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
|
|
From: Doug McNaught <doug@wireboard.com>
|
|
Date: 26 Sep 2001 18:39:44 -0400
|
|
In-Reply-To: "D. Hageman"'s message of "Wed, 26 Sep 2001 16:14:22 -0500 (CDT)"
|
|
Message-ID: <m3y9n11sr3.fsf@belphigor.mcnaught.org>
|
|
Lines: 26
|
|
User-Agent: Gnus/5.0806 (Gnus v5.8.6) XEmacs/21.1 (20 Minutes to Nikko)
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
"D. Hageman" <dhageman@dracken.com> writes:
|
|
|
|
> Save for the fact that the kernel can switch between threads faster then
|
|
> it can switch processes considering threads share the same address space,
|
|
> stack, code, etc. If need be sharing the data between threads is much
|
|
> easier then sharing between processes.
|
|
|
|
This depends on your system. Solaris has a huge difference between
|
|
thread and process context switch times, whereas Linux has very little
|
|
difference (and in fact a Linux process context switch is about as
|
|
fast as a Solaris thread switch on the same hardware--Solaris is just
|
|
a pig when it comes to process context switching).
|
|
|
|
> I can't comment on the "isolate data" line. I am still trying to figure
|
|
> that one out.
|
|
|
|
I think his point is one of clarity and maintainability. When a
|
|
task's data is explicitly shared (via shared memory of some sort) it's
|
|
fairly clear when you're accessing shared data and need to worry about
|
|
locking. Whereas when all data is shared by default (as with threads)
|
|
it's very easy to miss places where threads can step on each other.
|
|
|
|
-Doug
|
|
--
|
|
In a world of steel-eyed death, and men who are fighting to be warm,
|
|
Come in, she said, I'll give you shelter from the storm. -Dylan
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 2: you can get off all lists at once with the unregister command
|
|
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
|
|
|
|
From pgsql-hackers-owner+M13611=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 21:05:02 2001
|
|
Return-path: <pgsql-hackers-owner+M13611=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R152o22010
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 21:05:02 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R158430261
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 20:05:08 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13611=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from sss.pgh.pa.us ([192.204.191.242])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R0lgh29430
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 20:47:42 -0400 (EDT)
|
|
(envelope-from tgl@sss.pgh.pa.us)
|
|
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
|
|
by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f8R0kpK14707;
|
|
Wed, 26 Sep 2001 20:46:51 -0400 (EDT)
|
|
To: Ian Lance Taylor <ian@airs.com>
|
|
cc: "D. Hageman" <dhageman@dracken.com>, mlw <markw@mohawksoft.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <si8zf1vcau.fsf@daffy.airs.com>
|
|
References: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com> <si8zf1vcau.fsf@daffy.airs.com>
|
|
Comments: In-reply-to Ian Lance Taylor <ian@airs.com>
|
|
message dated "26 Sep 2001 15:04:41 -0700"
|
|
Date: Wed, 26 Sep 2001 20:46:51 -0400
|
|
Message-ID: <14704.1001551611@sss.pgh.pa.us>
|
|
From: Tom Lane <tgl@sss.pgh.pa.us>
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
Ian Lance Taylor <ian@airs.com> writes:
|
|
> (Actually, though, Postgres is already vulnerable to erratic behaviour
|
|
> because any backend process can corrupt the shared buffer pool.)
|
|
|
|
Not to mention the other parts of shared memory.
|
|
|
|
Nonetheless, our experience has been that cross-backend failures due to
|
|
memory clobbers in shared memory are very infrequent --- certainly far
|
|
less often than we see localized-to-a-backend crashes. Probably this is
|
|
because the shared memory is (a) small compared to the rest of the
|
|
address space and (b) only accessed by certain specific modules within
|
|
Postgres.
|
|
|
|
I'm convinced that switching to a thread model would result in a
|
|
significant degradation in our ability to recover from coredump-type
|
|
failures, even given the (implausible) assumption that we introduce no
|
|
new bugs during the conversion. I'm also *un*convinced that such a
|
|
conversion will yield significant performance benefits, unless we
|
|
introduce additional cross-thread dependencies (and more fragility
|
|
and lock contention) by tactics such as sharing catalog caches across
|
|
threads.
|
|
|
|
regards, tom lane
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 3: if posting/reading through Usenet, please send an appropriate
|
|
subscribe-nomail command to majordomo@postgresql.org so that your
|
|
message can get through to the mailing list cleanly
|
|
|
|
From pgsql-hackers-owner+M13616=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 23:10:52 2001
|
|
Return-path: <pgsql-hackers-owner+M13616=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R3Aqo03180
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 23:10:52 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R3B3438816
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 22:11:03 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13616=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R2vCh48923
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 22:57:12 -0400 (EDT)
|
|
(envelope-from alex@pilosoft.com)
|
|
Received: from localhost (alexmail@localhost)
|
|
by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id WAA27630;
|
|
Wed, 26 Sep 2001 22:58:41 -0400 (EDT)
|
|
Date: Wed, 26 Sep 2001 22:58:41 -0400 (EDT)
|
|
From: Alex Pilosov <alex@pilosoft.com>
|
|
To: "D. Hageman" <dhageman@dracken.com>
|
|
cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <Pine.LNX.4.33.0109261733050.2225-100000@typhon.dracken.com>
|
|
Message-ID: <Pine.BSO.4.10.10109262249480.14740-100000@spider.pilosoft.com>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
On Wed, 26 Sep 2001, D. Hageman wrote:
|
|
|
|
> > > Save for the fact that the kernel can switch between threads faster then
|
|
> > > it can switch processes considering threads share the same address space,
|
|
> > > stack, code, etc. If need be sharing the data between threads is much
|
|
> > > easier then sharing between processes.
|
|
> >
|
|
> > When using a kernel threading model, it's not obvious to me that the
|
|
> > kernel will switch between threads much faster than it will switch
|
|
> > between processes. As far as I can see, the only potential savings is
|
|
> > not reloading the pointers to the page tables. That is not nothing,
|
|
> > but it is also
|
|
<major snippage>
|
|
> > > I can't comment on the "isolate data" line. I am still trying to figure
|
|
> > > that one out.
|
|
> >
|
|
> > Sometimes you need data which is specific to a particular thread.
|
|
>
|
|
> When you need data that is specific to a thread you use a TSD (Thread
|
|
> Specific Data).
|
|
Which Linux does not support with a vengeance, to my knowledge.
|
|
|
|
As a matter of fact, quote from Linus on the matter was something like
|
|
"Solution to slow process switching is fast process switching, not another
|
|
kernel abstraction [referring to threads and TSD]". TSDs make
|
|
implementation of thread switching complex, and fork() complex.
|
|
|
|
The question about threads boils down to: Is there far more data that is
|
|
shared than unshared? If yes, threads are better, if not, you'll be
|
|
abusing TSD and slowing things down.
|
|
|
|
I believe right now, postgresql' model of sharing only things that need to
|
|
be shared is pretty damn good. The only slight problem is overhead of
|
|
forking another backend, but its still _fast_.
|
|
|
|
IMHO, threads would not bring large improvement to postgresql.
|
|
|
|
Actually, if I remember, there was someone who ported postgresql (I think
|
|
it was 6.5) to be multithreaded with major pain, because the requirement
|
|
was to integrate with CORBA. I believe that person posted some benchmarks
|
|
which were essentially identical to non-threaded postgres...
|
|
|
|
-alex
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 6: Have you searched our list archives?
|
|
|
|
http://archives.postgresql.org
|
|
|
|
From pgsql-hackers-owner+M13619=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 00:32:55 2001
|
|
Return-path: <pgsql-hackers-owner+M13619=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R4Wto07075
|
|
for <pgman@candle.pha.pa.us>; Thu, 27 Sep 2001 00:32:55 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R4X7444942
|
|
for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 23:33:07 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13619=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from sss.pgh.pa.us ([192.204.191.242])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R4Jsh61257
|
|
for <pgsql-hackers@postgresql.org>; Thu, 27 Sep 2001 00:19:54 -0400 (EDT)
|
|
(envelope-from tgl@sss.pgh.pa.us)
|
|
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
|
|
by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f8R4JLK15406;
|
|
Thu, 27 Sep 2001 00:19:21 -0400 (EDT)
|
|
To: "D. Hageman" <dhageman@dracken.com>
|
|
cc: Alex Pilosov <alex@pilosoft.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <Pine.LNX.4.33.0109262224040.1173-100000@typhon.dracken.com>
|
|
References: <Pine.LNX.4.33.0109262224040.1173-100000@typhon.dracken.com>
|
|
Comments: In-reply-to "D. Hageman" <dhageman@dracken.com>
|
|
message dated "Wed, 26 Sep 2001 22:41:39 -0500"
|
|
Date: Thu, 27 Sep 2001 00:19:20 -0400
|
|
Message-ID: <15403.1001564360@sss.pgh.pa.us>
|
|
From: Tom Lane <tgl@sss.pgh.pa.us>
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
"D. Hageman" <dhageman@dracken.com> writes:
|
|
> If you look at Myron Scott's post today you will see that it had other
|
|
> advantages going for it (like auto-vacuum!) and disadvantages ... rogue
|
|
> thread corruption (already debated today).
|
|
|
|
But note that Myron did a number of things that are (IMHO) orthogonal
|
|
to process-to-thread conversion, such as adding prepared statements,
|
|
a separate thread/process/whateveryoucallit for buffer writing, ditto
|
|
for vacuuming, etc. I think his results cannot be taken as indicative
|
|
of the benefits of threads per se --- these other things could be
|
|
implemented in a pure process model too, and we have no data with which
|
|
to estimate which change bought how much.
|
|
|
|
Threading certainly should reduce the context switch time, but this
|
|
comes at the price of increased overhead within each context (since
|
|
access to thread-local variables is not free). It's by no means
|
|
obvious that there's a net win there.
|
|
|
|
regards, tom lane
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 6: Have you searched our list archives?
|
|
|
|
http://archives.postgresql.org
|
|
|
|
From pgsql-hackers-owner+M13621=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 01:59:44 2001
|
|
Return-path: <pgsql-hackers-owner+M13621=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R5xio11898
|
|
for <pgman@candle.pha.pa.us>; Thu, 27 Sep 2001 01:59:44 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R5xi449748
|
|
for <pgman@candle.pha.pa.us>; Thu, 27 Sep 2001 00:59:45 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13621=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R5joh75612
|
|
for <pgsql-hackers@postgresql.org>; Thu, 27 Sep 2001 01:45:50 -0400 (EDT)
|
|
(envelope-from mscott@sacadia.com)
|
|
Received: from localhost (localhost [127.0.0.1])
|
|
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id WAA01144
|
|
for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 22:24:29 -0700 (PDT)
|
|
Date: Wed, 26 Sep 2001 22:24:29 -0700 (PDT)
|
|
From: Myron Scott <mscott@sacadia.com>
|
|
X-Sender: mscott@goldengate.kojoworldwide.com.
|
|
To: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
In-Reply-To: <15403.1001564360@sss.pgh.pa.us>
|
|
Message-ID: <Pine.GSO.4.10.10109262146500.1111-100000@goldengate.kojoworldwide.com.>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
|
|
|
|
> But note that Myron did a number of things that are (IMHO) orthogonal
|
|
|
|
yes, I did :)
|
|
|
|
> to process-to-thread conversion, such as adding prepared statements,
|
|
> a separate thread/process/whateveryoucallit for buffer writing, ditto
|
|
> for vacuuming, etc. I think his results cannot be taken as indicative
|
|
> of the benefits of threads per se --- these other things could be
|
|
> implemented in a pure process model too, and we have no data with which
|
|
> to estimate which change bought how much.
|
|
>
|
|
|
|
If you are comparing just process vs. thread, I really don't think I
|
|
gained much for performance and ended up with some pretty unmanageable
|
|
code.
|
|
|
|
The one thing that led to most of the gains was scheduling all the writes
|
|
to one thread which, as noted by Tom, you could do on the process model.
|
|
Besides, Most of the advantage in doing this was taken away with the
|
|
addition of WAL in 7.1.
|
|
|
|
The other real gain that I saw with threading was limiting the number of
|
|
open files but
|
|
that led me to alter much of the file manager in order to synchronize
|
|
access to the files which probably slowed things a bit.
|
|
|
|
To be honest, I don't think I, personally,
|
|
would try this again. I went pretty far off
|
|
the beaten path with this thing. It works well for what I am doing
|
|
( a limited number of SQL statements run many times over ) but there
|
|
probably was a better way. I'm thinking now that I should have tried to
|
|
add a CORBA interface for connections. I would have been able to
|
|
accomplish my original goals without creating a deadend for myself.
|
|
|
|
|
|
Thanks all for a great project,
|
|
|
|
Myron
|
|
mscott@sacadia.com
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 2: you can get off all lists at once with the unregister command
|
|
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
|
|
|
|
From pgsql-hackers-owner+M13632=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 10:21:22 2001
|
|
Return-path: <pgsql-hackers-owner+M13632=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8RELLo08607
|
|
for <pgman@candle.pha.pa.us>; Thu, 27 Sep 2001 10:21:21 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8RELP487000
|
|
for <pgman@candle.pha.pa.us>; Thu, 27 Sep 2001 09:21:26 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13632=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f8RE49h21870
|
|
for <pgsql-hackers@postgresql.org>; Thu, 27 Sep 2001 10:04:09 -0400 (EDT)
|
|
(envelope-from markw@mohawksoft.com)
|
|
Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1])
|
|
by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id KAA24417;
|
|
Thu, 27 Sep 2001 10:02:06 -0400
|
|
Message-ID: <3BB3315D.EC99FF65@mohawksoft.com>
|
|
Date: Thu, 27 Sep 2001 10:02:05 -0400
|
|
From: mlw <markw@mohawksoft.com>
|
|
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686)
|
|
X-Accept-Language: en
|
|
MIME-Version: 1.0
|
|
To: "D. Hageman" <dhageman@dracken.com>
|
|
cc: Ian Lance Taylor <ian@airs.com>,
|
|
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Spinlock performance improvement proposal
|
|
References: <Pine.LNX.4.33.0109261733050.2225-100000@typhon.dracken.com>
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Transfer-Encoding: 7bit
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
"D. Hageman" wrote:
|
|
|
|
> On 26 Sep 2001, Ian Lance Taylor wrote:
|
|
> >
|
|
> > > Save for the fact that the kernel can switch between threads faster then
|
|
> > > it can switch processes considering threads share the same address space,
|
|
> > > stack, code, etc. If need be sharing the data between threads is much
|
|
> > > easier then sharing between processes.
|
|
> >
|
|
> > When using a kernel threading model, it's not obvious to me that the
|
|
> > kernel will switch between threads much faster than it will switch
|
|
> > between processes. As far as I can see, the only potential savings is
|
|
> > not reloading the pointers to the page tables. That is not nothing,
|
|
> > but it is also not a lot.
|
|
>
|
|
> It is my understanding that avoiding a full context switch of the
|
|
> processor can be of a significant advantage. This is especially important
|
|
> on processor architectures that can be kinda slow at doing it (x86). I
|
|
> will admit that most modern kernels have features that assist software
|
|
> packages utilizing the forking model (copy on write for instance). It is
|
|
> also my impression that these do a good job. I am the kind of guy that
|
|
> looks towards the future (as in a year, year and half or so) and say that
|
|
> processors will hopefully get faster at context switching and more and
|
|
> more kernels will implement these algorithms to speed up the forking
|
|
> model. At the same time, I see more and more processors being shoved into
|
|
> a single box and it appears that the threads model works better on these
|
|
> type of systems.
|
|
|
|
"context" switching happens all the time on a multitasking system. On the x86
|
|
processor, a context switch happens when you call into the kernel. You have to go
|
|
through a call-gate to get to a lower privilege ring. "context" switching is very
|
|
fast. The operating system dictates how heavy or light a process switch is. Under
|
|
Linux (and I believe FreeBSD with Linux threads, or version 4.x ) threads and
|
|
processes are virtually identical. The only difference is that the virtual memory
|
|
pages are not "copy on write." Process vs thread scheduling is also virtually
|
|
identical.
|
|
|
|
If you look to the future, then you should accept that process switching should
|
|
become more efficient as the operating systems improve.
|
|
|
|
>
|
|
> > > I can't comment on the "isolate data" line. I am still trying to figure
|
|
> > > that one out.
|
|
> >
|
|
> > Sometimes you need data which is specific to a particular thread.
|
|
>
|
|
> When you need data that is specific to a thread you use a TSD (Thread
|
|
> Specific Data).
|
|
|
|
Yes, but Postgres has many global variables. The assumption has always been that
|
|
it is a stand-alone process with an explicitly shared paradigm, not implicitly.
|
|
|
|
>
|
|
> > Basically, you have to look at every global variable in the Postgres
|
|
> > backend, and determine whether to share it among all threads or to
|
|
> > make it thread-specific.
|
|
>
|
|
> Yes, if one was to implement threads into PostgreSQL I would think that
|
|
> some re-writing would be in order of several areas. Like I said before,
|
|
> give a person a chance to restructure things so future TODO items wouldn't
|
|
> be so hard to implement. Personally, I like to stay away from global
|
|
> variables as much as possible. They just get you into trouble.
|
|
|
|
In real live software, software which lives from year to year with active
|
|
development, things do get messy. There are always global variables involved in a
|
|
program. Efforts, of course, should be made to keep them to a minimum, but the
|
|
reality is that they always happen.
|
|
|
|
Also, the very structure of function calls may need to change when going from a
|
|
process model to a threaded model. Functions never before reentrant are now be
|
|
reentrant, think about that. That is a huge undertaking. Every single function
|
|
may need to be examined for thread safety, with little benefit.
|
|
|
|
>
|
|
> > > That last line is a troll if I every saw it ;-) I will agree that threads
|
|
> > > isn't for everything and that it has costs just like everything else. Let
|
|
> > > me stress that last part - like everything else. Certain costs exist in
|
|
> > > the present model, nothing is - how should we say ... perfect.
|
|
> >
|
|
> > When writing in C, threading inevitably loses robustness. Erratic
|
|
> > behaviour by one thread, perhaps in a user defined function, can
|
|
> > subtly corrupt the entire system, rather than just that thread. Part
|
|
> > of defensive programming is building barriers between different parts
|
|
> > of a system. Process boundaries are a powerful barrier.
|
|
>
|
|
> I agree with everything you wrote above except for the first line. My
|
|
> only comment is that process boundaries are only *truely* a powerful
|
|
> barrier if the processes are different pieces of code and are not
|
|
> dependent on each other in crippling ways. Forking the same code with the
|
|
> bug in it - and only 1 in 5 die - is still 4 copies of buggy code running
|
|
> on your system ;-)
|
|
|
|
This is simply not true. All software has bugs, it is an undeniable fact. Some
|
|
bugs are more likely to be hit than others. 5 processes , when one process hits a
|
|
bug, that does not mean the other 4 will hit the same bug. Obscure bugs kill
|
|
software all the time, the trick is to minimize the impact. Software is not
|
|
perfect, assuming it can be is a mistake.
|
|
|
|
|
|
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 5: Have you checked our extensive FAQ?
|
|
|
|
http://www.postgresql.org/users-lounge/docs/faq.html
|
|
|