From 6e5e78241b6adc2e0686d5461b3122f9d9d7766f Mon Sep 17 00:00:00 2001 From: Bruce Momjian Date: Sun, 1 Aug 2004 04:53:38 +0000 Subject: [PATCH] Remove pool discussion. All old stuff. --- doc/TODO.detail/pool | 1423 ------------------------------------------ 1 file changed, 1423 deletions(-) delete mode 100644 doc/TODO.detail/pool diff --git a/doc/TODO.detail/pool b/doc/TODO.detail/pool deleted file mode 100644 index e154f5638a..0000000000 --- a/doc/TODO.detail/pool +++ /dev/null @@ -1,1423 +0,0 @@ -From pgsql-hackers-owner+M4897@hub.org Wed Jul 12 00:15:33 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA06129 - for ; Wed, 12 Jul 2000 00:15:32 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C4FiW14410; - Wed, 12 Jul 2000 00:15:44 -0400 (EDT) -Received: from onyx-technologies.com (iron.onyx-technologies.com [216.205.44.194] (may be forged)) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C4ECW07902 - for ; Wed, 12 Jul 2000 00:14:12 -0400 (EDT) -Received: from onyx-technologies.com (collins.onyx-technologies.com [192.168.188.10]) - by onyx-technologies.com (8.9.2/8.9.0) with ESMTP id AAA14868 - for ; Wed, 12 Jul 2000 00:11:43 -0400 (EDT) -Message-ID: <396BE1B6.F755C5CE@onyx-technologies.com> -Date: Tue, 11 Jul 2000 23:10:46 -0400 -From: Jeffery Collins -X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.14-15mdk i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: ORr - -It seems like a first step would be to just have postmaster cache unused -connections. In other words if a client closes a connection, postmaster -keeps the connection and the child process around for the next connect -request. This has many of your advantages, but not all. However, it seems -like it would be simpler than attempting to multiplex a connection between -multiple clients. - -Jeff - -> -> Alfred Perlstein wrote: -> > -> > In an effort to complicate the postmaster beyond recognition I'm -> > proposing an idea that I hope can be useful to the developers. -> > -> > Connection pooling: -> > -> > The idea is to have the postmaster multiplex and do hand-offs of -> > database connections to other postgresql processes when the max -> > connections has been exceeded. -> > -> > This allows several gains: -> > -> > 1) Postgresql can support a large number of connections without -> > requiring a large amount of processes to do so. -> > -> > 2) Connection startup/finish will be cheaper because Postgresql -> > processes will not exit and need to reninit things such as shared -> > memory attachments and file opens. This will also reduce the load -> > on the supporting operating system and make postgresql much 'cheaper' -> > to run on systems that don't support the fork() model of execution -> > gracefully. -> > -> > 3) Long running connections can be preempted at transaction boundries -> > allowing other connections to gain process timeslices from the -> > connection pool. -> > -> > The idea is to make the postmaster that accepts connections a broker -> > for the connections. It will dole out descriptors using file -> > descriptor passing to children. If there's a demand for connections -> > meaning that all the postmasters are busy and there are pending -> > connections the postmaster can ask for a yeild on one of the -> > connections. -> > -> > A yeild involves the child postgresql process passing back the -> > client connection at a transaction boundry (between transactions) -> > so it can later be given to another (perhaps the same) child process. -> > -> > I spoke with Bruce briefly about this and he suggested that system -> > tables containing unique IDs could be used to identify passed -> > connections to the children and back to the postmaster. -> > -> > When a handoff occurs, the descriptor along with an ID referencing -> > things like temp tables and enviornment variables and authentication -> > information could be handed out as well allowing the child to resume -> > service to the interrupted connection. -> > -> > I really don't have the knowledge of Postgresql internals to -> > accomplish this, but the concepts are simple and the gains would -> > seem to be very high. -> > -> > Comments? -> > -> > -- -> > -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] -> > "I have the heart of a child; I keep it in a jar on my desk." - - -From pgsql-hackers-owner+M4904@hub.org Wed Jul 12 01:24:09 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06757 - for ; Wed, 12 Jul 2000 01:24:08 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C5OLW65679; - Wed, 12 Jul 2000 01:24:21 -0400 (EDT) -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C5MkW61040 - for ; Wed, 12 Jul 2000 01:22:46 -0400 (EDT) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Md429901; - Tue, 11 Jul 2000 22:22:39 -0700 (PDT) -Date: Tue, 11 Jul 2000 22:22:39 -0700 -From: Alfred Perlstein -To: Chris Bitmead -Cc: pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -Message-ID: <20000711222239.X25571@fw.wintelcom.net> -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2i -In-Reply-To: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au>; from chrisb@nimrod.itg.telstra.com.au on Wed, Jul 12, 2000 at 01:48:20PM +1000 -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -* Chris Bitmead [000711 20:53] wrote: -> -> Seems a lot trickier than you think. A backend can only be running -> one transaction at a time, so you'd have to keep track of which backends -> are in the middle of a transaction. I can imagine race conditions here. -> And backends can have contexts that are set by various clients using -> SET and friends. Then you'd have to worry about authentication each -> time. And you'd have to have algorithms for cleaning up old processes -> and/or dead processes. It all really sounds a bit hard. - -The backends can simply inform the postmaster when they are ready -either because they are done with a connection or because they -have just closed a transaction. - -All the state (auth/temp tables) can be held in the system tables. - -It's complicated, but no where on the order of something like -a new storage manager. - --Alfred - -From bright@fw.wintelcom.net Wed Jul 12 01:34:30 2000 -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06793 - for ; Wed, 12 Jul 2000 01:34:29 -0400 (EDT) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; - Tue, 11 Jul 2000 22:35:01 -0700 (PDT) -Date: Tue, 11 Jul 2000 22:35:00 -0700 -From: Alfred Perlstein -To: Bruce Momjian -Cc: Jeffery Collins , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -Message-ID: <20000711223500.Z25571@fw.wintelcom.net> -References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2i -In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 -Status: OR - -* Bruce Momjian [000711 21:31] wrote: -> > It seems like a first step would be to just have postmaster cache unused -> > connections. In other words if a client closes a connection, postmaster -> > keeps the connection and the child process around for the next connect -> > request. This has many of your advantages, but not all. However, it seems -> > like it would be simpler than attempting to multiplex a connection between -> > multiple clients. -> > -> -> This does seem like a good optimization. - -I'm not sure if the postmaster is needed besideds just to fork/exec -the backend, if so then when a backend finishes it can just call -accept() on the listening socket inherited from the postmaster to -get the next incomming connection. - --Alfred - -From pgsql-hackers-owner+M4906@hub.org Wed Jul 12 01:36:44 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06806 - for ; Wed, 12 Jul 2000 01:36:44 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C5akW94517; - Wed, 12 Jul 2000 01:36:46 -0400 (EDT) -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C5ZCW88503 - for ; Wed, 12 Jul 2000 01:35:12 -0400 (EDT) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; - Tue, 11 Jul 2000 22:35:01 -0700 (PDT) -Date: Tue, 11 Jul 2000 22:35:00 -0700 -From: Alfred Perlstein -To: Bruce Momjian -Cc: Jeffery Collins , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -Message-ID: <20000711223500.Z25571@fw.wintelcom.net> -References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2i -In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -* Bruce Momjian [000711 21:31] wrote: -> > It seems like a first step would be to just have postmaster cache unused -> > connections. In other words if a client closes a connection, postmaster -> > keeps the connection and the child process around for the next connect -> > request. This has many of your advantages, but not all. However, it seems -> > like it would be simpler than attempting to multiplex a connection between -> > multiple clients. -> > -> -> This does seem like a good optimization. - -I'm not sure if the postmaster is needed besideds just to fork/exec -the backend, if so then when a backend finishes it can just call -accept() on the listening socket inherited from the postmaster to -get the next incomming connection. - --Alfred - -From pgsql-hackers-owner+M4907@hub.org Wed Jul 12 01:55:39 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06881 - for ; Wed, 12 Jul 2000 01:55:38 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C5tnW34576; - Wed, 12 Jul 2000 01:55:49 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C5rfW28119 - for ; Wed, 12 Jul 2000 01:53:42 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id BAA21895; - Wed, 12 Jul 2000 01:52:56 -0400 (EDT) -To: Chris Bitmead -cc: Alfred Perlstein , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -In-reply-to: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> -Comments: In-reply-to Chris Bitmead - message dated "Wed, 12 Jul 2000 13:48:20 +1000" -Date: Wed, 12 Jul 2000 01:52:56 -0400 -Message-ID: <21892.963381176@sss.pgh.pa.us> -From: Tom Lane -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -Chris Bitmead writes: -> Seems a lot trickier than you think. A backend can only be running -> one transaction at a time, so you'd have to keep track of which backends -> are in the middle of a transaction. I can imagine race conditions here. - -Aborting out of a transaction is no problem; we have code for that -anyway. More serious problems: - -* We have no code for reassigning a backend to a different database, - so the pooling would have to be per-database. - -* AFAIK there is no portable way to pass a socket connection from the - postmaster to an already-existing backend process. If you do a - fork() then the connection is inherited ... otherwise you've got a - problem. (You could work around this if the postmaster relays - every single byte in both directions between client and backend, - but the performance problems with that should be obvious.) - -> And backends can have contexts that are set by various clients using -> SET and friends. - -Resetting SET variables would be a problem, and there's also the -assigned user name to be reset. This doesn't seem impossible, but -it does seem tedious and error-prone. (OTOH, Peter E's recent work -on guc.c might have unified option-handling enough to bring it -within reason.) - -The killer problem here is that you can't hand off a connection -accepted by the postmaster to a backend except by fork() --- at least -not with methods that work on a wide variety of Unixen. Unless someone -has a way around that, I think the idea is dead in the water; the lesser -issues don't matter. - - regards, tom lane - -From pgsql-hackers-owner+M4910@hub.org Wed Jul 12 02:24:16 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11184 - for ; Wed, 12 Jul 2000 02:24:15 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C6OAW98187; - Wed, 12 Jul 2000 02:24:10 -0400 (EDT) -Received: from acheron.rime.com.au (root@albatr.lnk.telstra.net [139.130.54.222]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C6MZW95741 - for ; Wed, 12 Jul 2000 02:22:36 -0400 (EDT) -Received: from oberon (Oberon.rime.com.au [203.8.195.100]) - by acheron.rime.com.au (8.9.3/8.9.3) with SMTP id QAA12845; - Wed, 12 Jul 2000 16:16:23 +1000 -Message-Id: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> -X-Sender: pjw@mail.rhyme.com.au -X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.5 (32) -Date: Wed, 12 Jul 2000 16:22:10 +1000 -To: Tom Lane , - Chris Bitmead -From: Philip Warner -Subject: Re: [HACKERS] Connection pooling. -Cc: Alfred Perlstein , pgsql-hackers@hub.org -In-Reply-To: <21892.963381176@sss.pgh.pa.us> -References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> - <20000711185318.W25571@fw.wintelcom.net> - <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> -Mime-Version: 1.0 -Content-Type: text/plain; charset="us-ascii" -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -At 01:52 12/07/00 -0400, Tom Lane wrote: -> ->The killer problem here is that you can't hand off a connection ->accepted by the postmaster to a backend except by fork() --- at least ->not with methods that work on a wide variety of Unixen. Unless someone ->has a way around that, I think the idea is dead in the water; the lesser ->issues don't matter. -> - -My understanding of pg client interfaces is that the client uses ont of the -pg interface libraries to make a connection to the db; they specify host & -port and get back some kind of connection object. - -What stops the interface library from using the host & port to talk to the -postmaster, find the host & port the spare db server, then connect directly -to the server? This second connection is passed back in the connection object. - -When the client disconnects from the server, it tells the postmaster it's -available again etc. - -ie. in very rough terms: - - client calls interface to connect - - interface talks to postmaster on port 5432, says "I want a server for -xyz db" - - postmaster replies with "Try port ABCD" OR "no servers available" - postmaster marks the nominated server as 'used' - postmaster disconnects from client - - interface connects to port ABCD as per normal protocols - interface fills in connection object & returns - - ...client does some work... - - client disconnects - - db server tells postmaster it's available again. - - -There would also need to be timeout code to handle the case where the -interface did not do the second connect. - -You could also have the interface allocate a port send it's number to the -postmaster then listen on it, but I think that would represent a potential -security hole. - - ----------------------------------------------------------------- -Philip Warner | __---_____ -Albatross Consulting Pty. Ltd. |----/ - \ -(A.C.N. 008 659 498) | /(@) ______---_ -Tel: (+61) 0500 83 82 81 | _________ \ -Fax: (+61) 0500 83 82 82 | ___________ | -Http://www.rhyme.com.au | / \| - | --________-- -PGP key available upon request, | / -and from pgp5.ai.mit.edu:11371 |/ - -From pgsql-hackers-owner+M4912@hub.org Wed Jul 12 02:32:21 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11228 - for ; Wed, 12 Jul 2000 02:32:20 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C6WWW18412; - Wed, 12 Jul 2000 02:32:32 -0400 (EDT) -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C6UwW16062 - for ; Wed, 12 Jul 2000 02:30:58 -0400 (EDT) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id e6C6Uov01852; - Tue, 11 Jul 2000 23:30:50 -0700 (PDT) -Date: Tue, 11 Jul 2000 23:30:49 -0700 -From: Alfred Perlstein -To: Tom Lane -Cc: Chris Bitmead , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -Message-ID: <20000711233049.A25571@fw.wintelcom.net> -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2i -In-Reply-To: <21892.963381176@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 01:52:56AM -0400 -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -* Tom Lane [000711 22:53] wrote: -> Chris Bitmead writes: -> > Seems a lot trickier than you think. A backend can only be running -> > one transaction at a time, so you'd have to keep track of which backends -> > are in the middle of a transaction. I can imagine race conditions here. -> -> Aborting out of a transaction is no problem; we have code for that -> anyway. More serious problems: -> -> * We have no code for reassigning a backend to a different database, -> so the pooling would have to be per-database. - -That would need to be fixed. How difficult would that be? - -> * AFAIK there is no portable way to pass a socket connection from the -> postmaster to an already-existing backend process. If you do a -> fork() then the connection is inherited ... otherwise you've got a -> problem. (You could work around this if the postmaster relays -> every single byte in both directions between client and backend, -> but the performance problems with that should be obvious.) - -no, see below. - -> > And backends can have contexts that are set by various clients using -> > SET and friends. -> -> Resetting SET variables would be a problem, and there's also the -> assigned user name to be reset. This doesn't seem impossible, but -> it does seem tedious and error-prone. (OTOH, Peter E's recent work -> on guc.c might have unified option-handling enough to bring it -> within reason.) - -What can be done is that each incomming connection can be assigned an -ID into a system table. As options are added the system would assign -them to key-value pairs in this table. Once someone detects that the -remote side has closed the connection the data can be destroyed, but -until then along with the descriptor passing the ID of the client -as an index into the table can be passed for the backend to fetch. - -> The killer problem here is that you can't hand off a connection -> accepted by the postmaster to a backend except by fork() --- at least -> not with methods that work on a wide variety of Unixen. Unless someone -> has a way around that, I think the idea is dead in the water; the lesser -> issues don't matter. - -The code has been around since 4.2BSD, it takes a bit of #ifdef to -get it right on all systems but it's not impossible, have a look at -http://www.fhttpd.org/ for a web server that does this in a portable -fashion. - -I should have a library whipped up for you guys really soon now -to handle the descriptor and message passing. - --- --Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] -"I have the heart of a child; I keep it in a jar on my desk." - -From pgsql-hackers-owner+M4913@hub.org Wed Jul 12 03:06:54 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11529 - for ; Wed, 12 Jul 2000 03:06:53 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C76ZW95615; - Wed, 12 Jul 2000 03:06:35 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C74gW93358 - for ; Wed, 12 Jul 2000 03:04:42 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22136; - Wed, 12 Jul 2000 03:04:13 -0400 (EDT) -To: Alfred Perlstein -cc: Chris Bitmead , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -In-reply-to: <20000711233049.A25571@fw.wintelcom.net> -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> -Comments: In-reply-to Alfred Perlstein - message dated "Tue, 11 Jul 2000 23:30:49 -0700" -Date: Wed, 12 Jul 2000 03:04:13 -0400 -Message-ID: <22133.963385453@sss.pgh.pa.us> -From: Tom Lane -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -Alfred Perlstein writes: -> * Tom Lane [000711 22:53] wrote: ->> The killer problem here is that you can't hand off a connection ->> accepted by the postmaster to a backend except by fork() --- at least ->> not with methods that work on a wide variety of Unixen. - -> The code has been around since 4.2BSD, it takes a bit of #ifdef to -> get it right on all systems but it's not impossible, have a look at -> http://www.fhttpd.org/ for a web server that does this in a portable -> fashion. - -I looked at this to see if it would teach me something I didn't know. -It doesn't. It depends on sendmsg() which is a BSD-ism and not very -portable. - - regards, tom lane - -From pgsql-hackers-owner+M4914@hub.org Wed Jul 12 03:12:40 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11597 - for ; Wed, 12 Jul 2000 03:12:39 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C7CjW13459; - Wed, 12 Jul 2000 03:12:45 -0400 (EDT) -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C7B8W07036 - for ; Wed, 12 Jul 2000 03:11:08 -0400 (EDT) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id e6C79lE02841; - Wed, 12 Jul 2000 00:09:47 -0700 (PDT) -Date: Wed, 12 Jul 2000 00:09:47 -0700 -From: Alfred Perlstein -To: Tom Lane -Cc: Chris Bitmead , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -Message-ID: <20000712000947.D25571@fw.wintelcom.net> -References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> <22133.963385453@sss.pgh.pa.us> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2i -In-Reply-To: <22133.963385453@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 03:04:13AM -0400 -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -* Tom Lane [000712 00:04] wrote: -> Alfred Perlstein writes: -> > * Tom Lane [000711 22:53] wrote: -> >> The killer problem here is that you can't hand off a connection -> >> accepted by the postmaster to a backend except by fork() --- at least -> >> not with methods that work on a wide variety of Unixen. -> -> > The code has been around since 4.2BSD, it takes a bit of #ifdef to -> > get it right on all systems but it's not impossible, have a look at -> > http://www.fhttpd.org/ for a web server that does this in a portable -> > fashion. -> -> I looked at this to see if it would teach me something I didn't know. -> It doesn't. It depends on sendmsg() which is a BSD-ism and not very -> portable. - -It's also specified by Posix.1g if that means anything. - --Alfred - -From pgsql-hackers-owner+M4916@hub.org Wed Jul 12 03:49:58 2000 -Received: from hub.org (root@hub.org [216.126.84.1]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11736 - for ; Wed, 12 Jul 2000 03:49:58 -0400 (EDT) -Received: from hub.org (majordom@localhost [127.0.0.1]) - by hub.org (8.10.1/8.10.1) with SMTP id e6C7oBW95547; - Wed, 12 Jul 2000 03:50:11 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) - by hub.org (8.10.1/8.10.1) with ESMTP id e6C7mPW93141 - for ; Wed, 12 Jul 2000 03:48:25 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22297; - Wed, 12 Jul 2000 03:47:37 -0400 (EDT) -To: Philip Warner -cc: Chris Bitmead , - Alfred Perlstein , pgsql-hackers@hub.org -Subject: Re: [HACKERS] Connection pooling. -In-reply-to: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> -References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> -Comments: In-reply-to Philip Warner - message dated "Wed, 12 Jul 2000 16:22:10 +1000" -Date: Wed, 12 Jul 2000 03:47:37 -0400 -Message-ID: <22294.963388057@sss.pgh.pa.us> -From: Tom Lane -X-Mailing-List: pgsql-hackers@postgresql.org -Precedence: bulk -Sender: pgsql-hackers-owner@hub.org -Status: OR - -Philip Warner writes: -> What stops the interface library from using the host & port to talk to -> the postmaster, find the host & port the spare db server, then connect -> directly to the server? - -You're assuming that we can change the on-the-wire protocol freely and -only the API presented by the client libraries matters. In a perfect -world that might be true, but reality is that we can't change the wire -protocol easily. If we do, it breaks all existing precompiled clients. -Updating clients can be an extremely painful experience in multiple- -machine installations. - -Also, we don't have just one set of client libraries to fix. There are -at least three client-side implementations that don't depend on libpq. - -We have done protocol updates in the past --- in fact I was responsible -for the last one. (And I'm still carrying the scars, which is why I'm -not too enthusiastic about another one.) It's not impossible, but it -needs more evidence than "this should speed up connections by -I-don't-know-how-much"... - -It might also be worth pointing out that the goal was to speed up the -end-to-end connection time. Redirecting as you suggest is not free -(at minimum it would appear to require two TCP connection setups and two -authentication cycles). What evidence have you got that it'd be faster -than spawning a new backend? - -I tend to agree with the opinion that connection-pooling on the client -side offers more bang for the buck than we could hope to get by doing -surgery on the postmaster/backend setup. - -Also, to return to the original point, AFAIK we have not tried hard -to cut the backend startup time, other than the work that was done -a year or so back to eliminate exec() of a separate executable. -It'd be worth looking to see what could be done there with zero -impact on existing clients. - - regards, tom lane - -From pgsql-hackers-owner+M16940@postgresql.org Sun Dec 23 23:06:28 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBO46R429655 - for ; Sun, 23 Dec 2001 23:06:27 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBO40oN57016; - Sun, 23 Dec 2001 22:00:50 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16940@postgresql.org) -Received: from relay.pair.com (relay1.pair.com [209.68.1.20]) - by postgresql.org (8.11.3/8.11.4) with SMTP id fBID0um78493 - for ; Tue, 18 Dec 2001 08:00:56 -0500 (EST) - (envelope-from ml@augustz.com) -Received: (qmail 79914 invoked from network); 18 Dec 2001 13:00:58 -0000 -Received: from acz01997-2.pomona.edu (HELO Microsoft) (134.173.91.3) - by relay1.pair.com with SMTP; 18 Dec 2001 13:00:58 -0000 -X-pair-Authenticated: 134.173.91.3 -From: "August Zajonc" -To: -Subject: [HACKERS] Connection Pooling, a year later -Date: Tue, 18 Dec 2001 05:00:57 -0800 -Message-ID: -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -X-Priority: 3 (Normal) -X-MSMail-Priority: Normal -X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) -Importance: Normal -X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -I feel there was a reasonably nice client side attempt at this using a -worker pool model or something. Can't seem to track it down at this moment. -Also would spread queries in different ways to get a hot backup equivalent -etc. It was slick. - -The key is that pgsql be able to support a very significant number of -transactions. Be neat to see some numbers on your attempt. - -Site I used to run had 6 front end webservers running PHP apps. Each -persistent connection (a requirement to avoid overhead of set-up/teardowns) -lived as long as the httpd process lived, even if idle. That meant at 250 -processes per server we had a good 1500 connections clicking over. Our -feeling was that rather than growing to 3,000 connections as the frontend -grew, why not pool those connections off each machine down to perhaps -75/machine worker threads that actually did the work. - -Looks like that's not an issue if these backends suck up few resources. -Doing something similar with MySQL we'd experiance problems if we got into -the 2,000 connection range. (kernel/system limits bumped plenty high). - -While we are on TODO's I would like to point out that some way to fully -vacume (ie recover deleted and changed) while a db is in full swing is -critical to larger installtions. We did 2 billion queries between reboots on -a quad zeon MySQL box, and those are real user based queries not data loads -or anything like that. At 750-1000 queries/second bringing the database down -or seriously degrading its performance is not a good option. - -Enjoy playing with pgsql as always.... - -- AZ - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From dhogaza@pacifier.com Tue Dec 18 11:15:06 2001 -Return-path: -Received: from asteroid.pacifier.com ([199.2.117.154]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGF5419342 - for ; Tue, 18 Dec 2001 11:15:05 -0500 (EST) -Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) - by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGEGe29925; - Tue, 18 Dec 2001 08:14:17 -0800 (PST) -Message-ID: <3C1F6B81.10500@pacifier.com> -Date: Tue, 18 Dec 2001 08:14:57 -0800 -From: Don Baccus -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: Bruce Momjian -cc: mlw , owensmk@earthlink.net, - pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] Connection Pooling, a year later -References: <200112180342.fBI3g4s23880@candle.pha.pa.us> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Status: OR - -Bruce Momjian wrote: - - -> It would just be nice to have it done internally rather than have all -> the clients do it, iff it can be done cleanly. - -Serious client applications that need it already do it. Firing up an -Oracle or most other db's isn't that lightweight a deal, either, it's -not useful only for PG.. - -Personally I'd just view it as getting in the way, but then I use a -webserver that's provided connection pooling for client threads for the -last seven years ... - -I agree with Tom that the client seems to be the best place to do this. - -Among other things it isn't that difficult. If you know how to fire up -one connection, you know how to fire up N of them and adding logic to -pool them afterwards is easy enough. --- -Don Baccus -Portland, OR -http://donb.photo.net, http://birdnotes.net, http://openacs.org - - -From dhogaza@pacifier.com Tue Dec 18 11:24:33 2001 -Return-path: -Received: from asteroid.pacifier.com ([199.2.117.154]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGOW421363 - for ; Tue, 18 Dec 2001 11:24:33 -0500 (EST) -Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) - by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGNne00442; - Tue, 18 Dec 2001 08:23:49 -0800 (PST) -Message-ID: <3C1F6DBF.2040000@pacifier.com> -Date: Tue, 18 Dec 2001 08:24:31 -0800 -From: Don Baccus -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: Bruce Momjian -cc: mlw , owensmk@earthlink.net, - pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] Connection Pooling, a year later -References: <200112180357.fBI3vBm24991@candle.pha.pa.us> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Status: OR - -Bruce Momjian wrote: - - -> Yes, that is assuming you are using PHP. If you are using something -> else, you connection pooling in there too. All those client interfaces -> reimplementing connection pooling seems like a waste to me. - - -Effective pooling's pretty specific to your environment, though, so any -general mechanism would have to provide a wide-ranging suite of -parameters governing the number to pool, how long each handle should -live, what to do if a handle's released by a client while in the midst -of a transaction (AOLserver rolls back the transaction, other clients -might want to do something else, i.e. fire a callback or the like), etc etc. - -I think it would be fairly complex and for those high-throughput -applications already written with client-side pooling no improvement. - -And those are the only applications that need it. - --- -Don Baccus -Portland, OR -http://donb.photo.net, http://birdnotes.net, http://openacs.org - - -From pgsql-hackers-owner+M16726@postgresql.org Tue Dec 18 11:48:16 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGmG422658 - for ; Tue, 18 Dec 2001 11:48:16 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBIGkFN40986; - Tue, 18 Dec 2001 10:46:15 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16726@postgresql.org) -Received: from comet.pacifier.com ([199.2.117.155]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBIGYZm93404 - for ; Tue, 18 Dec 2001 11:34:35 -0500 (EST) - (envelope-from dhogaza@pacifier.com) -Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) - by comet.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGXCX29823; - Tue, 18 Dec 2001 08:33:12 -0800 (PST) -Message-ID: <3C1F6FF1.9030606@pacifier.com> -Date: Tue, 18 Dec 2001 08:33:53 -0800 -From: Don Baccus -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: Mark Pritchard -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Connection Pooling, a year later -References: -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Mark Pritchard wrote: - ->>I think it is the startup cost that most people want to avoid, and our's ->>is higher than most db's that use threads; at least I think so. ->> ->>It would just be nice to have it done internally rather than have all ->>the clients do it, iff it can be done cleanly. ->> -> -> I'd add that client side connection pooling isn't effective in some cases -> anyway - one application we work with has 4 physical application servers -> running around 6 applications. Each of the applications was written by a -> different vendor, and thus a pool size of five gives you 120 open -> connections. - -Tuning a central pooling mechanism to run well in this kind of situation -isn't going to be a trivial task, either. The next thing you'll want is -some way to prioritize the various clients so your more serious -applications have a better chance of getting a pool. - -Or you'll want to set up subpools so they don't compete with each other, -in effect replicating what's done now, but adding more complexity to the -central service. - --- -Don Baccus -Portland, OR -http://donb.photo.net, http://birdnotes.net, http://openacs.org - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From oleg@sai.msu.su Tue Dec 18 12:05:51 2001 -Return-path: -Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIH5h423591 - for ; Tue, 18 Dec 2001 12:05:43 -0500 (EST) -Received: from ra (ra [158.250.29.2]) - by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id UAA18592; - Tue, 18 Dec 2001 20:05:26 +0300 (GMT) -Date: Tue, 18 Dec 2001 20:05:26 +0300 (GMT) -From: Oleg Bartunov -X-X-Sender: -To: Don Baccus -cc: Bruce Momjian , mlw , - , , - Tom Lane -Subject: Re: [HACKERS] Connection Pooling, a year later -In-Reply-To: <3C1F6DBF.2040000@pacifier.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: OR - -Does schema support will resolve this discussion ? -If I understand correctly, initial arguments for connection pooling -was restriction in number of persistent connections. it's right in -current postgresql that if one wants keep connection for performance -reason to several databases the total number of connections will -doubled, trippled and so on. But if I understand schema support will -eventually put away these problem because we could keep only one -pool of connections to the *one* database. - - Oleg - -On Tue, 18 Dec 2001, Don Baccus wrote: - -> Bruce Momjian wrote: -> -> -> > Yes, that is assuming you are using PHP. If you are using something -> > else, you connection pooling in there too. All those client interfaces -> > reimplementing connection pooling seems like a waste to me. -> -> -> Effective pooling's pretty specific to your environment, though, so any -> general mechanism would have to provide a wide-ranging suite of -> parameters governing the number to pool, how long each handle should -> live, what to do if a handle's released by a client while in the midst -> of a transaction (AOLserver rolls back the transaction, other clients -> might want to do something else, i.e. fire a callback or the like), etc etc. -> -> I think it would be fairly complex and for those high-throughput -> applications already written with client-side pooling no improvement. -> -> And those are the only applications that need it. -> -> - - Regards, - Oleg -_____________________________________________________________ -Oleg Bartunov, sci.researcher, hostmaster of AstroNet, -Sternberg Astronomical Institute, Moscow University (Russia) -Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ -phone: +007(095)939-16-83, +007(095)939-23-83 - - -From pgsql-hackers-owner+M16748@postgresql.org Tue Dec 18 15:11:46 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIKBj405415 - for ; Tue, 18 Dec 2001 15:11:45 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBIKB0N47430; - Tue, 18 Dec 2001 14:11:00 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16748@postgresql.org) -Received: from candle.pha.pa.us (216-55-132-35.dialup.tnt01.san-diego.abac.net [216.55.132.35]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBIJulm05030 - for ; Tue, 18 Dec 2001 14:56:47 -0500 (EST) - (envelope-from pgman@candle.pha.pa.us) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id fBIJuVB04553; - Tue, 18 Dec 2001 14:56:31 -0500 (EST) -From: Bruce Momjian -Message-ID: <200112181956.fBIJuVB04553@candle.pha.pa.us> -Subject: Re: [HACKERS] Connection Pooling, a year later -In-Reply-To: <3C1F6ED6.6080107@pacifier.com> "from Don Baccus at Dec 18, 2001 - 08:29:10 am" -To: Don Baccus -Date: Tue, 18 Dec 2001 14:56:31 -0500 (EST) -cc: Christopher Kings-Lynne , - mlw , owensmk@earthlink.net, - pgsql-hackers@postgresql.org, Tom Lane -X-Mailer: ELM [version 2.4ME+ PL90 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -> Bruce Momjian wrote: -> -> -> > -> > The trick for that is to call COMMIT before you pass the backend to a -> > new person. -> -> -> The failure to COMMIT is a programmer error - ROLLBACK's much safer. At -> least that's what we decided in the AOLserver community, and that's -> what the drivers for Oracle and PG (the two I maintain) implement. - - -Then you can issue a "BEGIN;ROLLBACK;" when you pass the session to the -next user, and "RESET ALL;" of course. - -> > Now, if you want to abort a left-over transaction, you can -> > do an ABORT but that is going to show up in the server logs because an -> > ABORT without a transaction causes an error message. -> -> -> The connection pooling mechanism needs to track the transaction state -> and only ROLLBACK a handle that's not in autocommit state or in the -> midst of a BEGIN/END transaction (again, Oracle vs. PG).. - -Seems like a lot of work to keep track of transaction state in the -client; seems easier to just unconditionally issue the begin;rollback. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M16793@postgresql.org Wed Dec 19 00:46:50 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJ5kn426988 - for ; Wed, 19 Dec 2001 00:46:49 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJ5gnN63439; - Tue, 18 Dec 2001 23:42:49 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16793@postgresql.org) -Received: from deborah.paradise.net.nz (deborah.paradise.net.nz [203.96.152.32]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJ5Uvm41224 - for ; Wed, 19 Dec 2001 00:30:58 -0500 (EST) - (envelope-from andrew@catalyst.net.nz) -Received: from heidegger.catalyst.net.nz (203-96-145-108.adsl.paradise.net.nz [203.96.145.108]) - by deborah.paradise.net.nz (Postfix) with ESMTP - id D1C7CD194D; Wed, 19 Dec 2001 18:31:01 +1300 (NZDT) -Received: from 127.0.0.1 (ident=unknown) by heidegger.catalyst.net.nz - with esmtp (MasqMail 0.1.15) id 16GZJK-5NU-00; Wed, 19 Dec 2001 - 18:30:34 +1300 -Subject: Re: [HACKERS] Connection Pooling, a year later -From: Andrew McMillan -To: owensmk@earthlink.net -cc: pgsql-hackers@postgresql.org -In-Reply-To: <200112180028.fBI0Sum06915@postgresql.org> -References: <200112180028.fBI0Sum06915@postgresql.org> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Evolution/1.0 (Preview Release) -Date: 19 Dec 2001 18:30:34 +1300 -Message-ID: <1008739834.25608.33.camel@kant.mcmillan.net.nz> -MIME-Version: 1.0 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Tue, 2001-12-18 at 13:46, Michael Owens wrote: -> -> By having the postmaster map multiple clients to a fixed number of backends, -> you achieve the happy medium: You never exceed the ideal number of active -> backends, and at the same time you are not limited to only accepting a fixed -> number of connections. Accepting connections can now be based on load -> (however you wish to define it), not number. You now make decisions based on -> utlization. -> -> If it were shown that even half of a backend's life consisted of idle time, -> leasing out that idle time to another active connection would potentially -> double the average number of simultaneous requests without (theoretically) -> incurring any significant degradation in performance. -> - -Have you looked at the client-side connection pooling solutions out -there? - -DBBalancer ( http://dbbalancer.sourceforge.net/ ) tries to sit very -transparently between your application and PostgreSQL, letting you -implement connection pooling with almost no application changes. - -There was another one I came across too, but that one requires you to -make more wide-reaching changes to the application. - -In my applications I have found DBBalancer to be roughly the same level -of performance as PHP persistent connections, but a lot fewer -connections are needed in the pool because they are only needed when -Apache is delivering dynamic content - not the associated static -stylesheets and images. - -Regards, - Andrew. --- --------------------------------------------------------------------- -Andrew @ Catalyst .Net.NZ Ltd, PO Box 11-053, Manners St, Wellington -WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St -DDI: +64(4)916-7201 MOB: +64(21)635-694 OFFICE: +64(4)499-2267 - Are you enrolled at http://schoolreunions.co.nz/ yet? - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M16834@postgresql.org Wed Dec 19 14:17:47 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJJHk404096 - for ; Wed, 19 Dec 2001 14:17:46 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJJENN87550; - Wed, 19 Dec 2001 13:14:23 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16834@postgresql.org) -Received: from asteroid.pacifier.com ([199.2.117.154]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJJ55m16181 - for ; Wed, 19 Dec 2001 14:05:05 -0500 (EST) - (envelope-from dhogaza@pacifier.com) -Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) - by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBJJ3fe20585; - Wed, 19 Dec 2001 11:03:41 -0800 (PST) -Message-ID: <3C20E4B9.8090200@pacifier.com> -Date: Wed, 19 Dec 2001 11:04:25 -0800 -From: Don Baccus -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: owensmk@earthlink.net -cc: Andrew McMillan , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Connection Pooling, a year later -References: <200112180028.fBI0Sum06915@postgresql.org> <1008739834.25608.33.camel@kant.mcmillan.net.nz> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Michael Owens wrote: - -> As long as each client's call is composed of a standalone transaction, there -> is no problem with external connection pools. But what about when a client's -> transactions spans two or more calls, such as SELECT FOR UPDATE? Then pooling -> is not safe: it offers no assurance of what may be interjected into an open -> transaction between calls. For example, each is a separate call to a shared -> connection: -> -> Client A: BEGIN WORK; SELECT last_name from customer for update where ; -> -> Client B: BEGIN WORK; SELECT street from customer for update where ; -> -> Client A: update customer set lastname= where ; COMMIT -> WORK; -> -> -> Now, isn't Client B's write lock gone with Client A's commit? Yet Client A's -> lock is still hanging around. While Client B's commit will close it, Client B -> has lost the assurance of its lock, defeating the purpose of SELECT FOR -> UPDATE. -> -> If this is corrent, then external connection pools limit what you can do with -> the database to a single call. Any transaction spanning more than one call is -> unsafe, because it is not isolated from other clients sharing the same -> connection. - - -The general idea is that you grab a handle and hold onto it until you're -done. This makes the above scenario impossible. - -Forgetting to commit or rollback before relenquishing the handle is -another scenario that can lead to problems but that's already been -discussed in detail. - --- -Don Baccus -Portland, OR -http://donb.photo.net, http://birdnotes.net, http://openacs.org - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M16838@postgresql.org Wed Dec 19 15:17:32 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJKHV408663 - for ; Wed, 19 Dec 2001 15:17:32 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJKDNN89347; - Wed, 19 Dec 2001 14:13:23 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16838@postgresql.org) -Received: from gull.prod.itd.earthlink.net (gull.mail.pas.earthlink.net [207.217.120.84]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJKA2m62023 - for ; Wed, 19 Dec 2001 15:10:03 -0500 (EST) - (envelope-from owensmk@earthlink.net) -Received: from sdn-ar-004txfworp179.dialsprint.net ([158.252.142.219] helo=there) - by gull.prod.itd.earthlink.net with smtp (Exim 3.33 #1) - id 16Gn2K-0005YP-00; Wed, 19 Dec 2001 12:09:57 -0800 -Content-Type: text/plain; - charset="iso-8859-1" -From: Michael Owens -Reply-To: owensmk@earthlink.net -To: Don Baccus -Subject: Re: [HACKERS] Connection Pooling, a year later -Date: Wed, 19 Dec 2001 14:28:14 -0600 -X-Mailer: KMail [version 1.3.1] -cc: Andrew McMillan , pgsql-hackers@postgresql.org -References: <200112180028.fBI0Sum06915@postgresql.org> <3C20E4B9.8090200@pacifier.com> -In-Reply-To: <3C20E4B9.8090200@pacifier.com> -MIME-Version: 1.0 -Content-Transfer-Encoding: 8bit -Message-ID: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Wednesday 19 December 2001 01:04 pm, Don Baccus wrote: - - -> The general idea is that you grab a handle and hold onto it until you're -> done. This makes the above scenario impossible. -> -> Forgetting to commit or rollback before relenquishing the handle is -> another scenario that can lead to problems but that's already been -> discussed in detail. - -But then the shared connection is unshared, sitting idle while the client -works in between calls, thus introducing idle time among a fixed number of -connections. The server is doing less than it could. - -I agree that this connection pool has improved things in eliminating backend -startup time. But idle time still exists for the clients performing multiple -calls, proportional to the product of the number of multiple call clients and -the number of calls they make, plus the idle time between them. - -However this probably only ever happens on update. Inserts and selects can be -done in one call. And, I suppose updates comprise only a small fraction of -the requests sent to the database. Even then, you can probably eliminate some -multiple calls by using things such as procedures. - -Factoring all that in, you can probably do as well by optimizing your -particular database/application than by writing code. - -I relent. Thanks for your thoughts. - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M16855@postgresql.org Thu Dec 20 01:02:51 2001 -Return-path: -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBK62o404294 - for ; Thu, 20 Dec 2001 01:02:50 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBK5xnN05417; - Wed, 19 Dec 2001 23:59:49 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16855@postgresql.org) -Received: from deborah.paradise.net.nz (deborah.paradise.net.nz [203.96.152.32]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fBK5mvm47263 - for ; Thu, 20 Dec 2001 00:48:57 -0500 (EST) - (envelope-from andrew@catalyst.net.nz) -Received: from heidegger.catalyst.net.nz (203-96-145-94.adsl.paradise.net.nz [203.96.145.94]) - by deborah.paradise.net.nz (Postfix) with ESMTP - id 7407FD2B76; Thu, 20 Dec 2001 18:49:01 +1300 (NZDT) -Received: from 127.0.0.1 (ident=unknown) by heidegger.catalyst.net.nz - with esmtp (MasqMail 0.1.15) id 16GrRk-2Ry-00; Thu, 20 Dec 2001 - 13:52:28 +1300 -Subject: Re: [HACKERS] Connection Pooling, a year later -From: Andrew McMillan -To: owensmk@earthlink.net -cc: pgsql-hackers@postgresql.org -In-Reply-To: -References: <200112180028.fBI0Sum06915@postgresql.org> - <1008739834.25608.33.camel@kant.mcmillan.net.nz> - -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Evolution/1.0 (Preview Release) -Date: 20 Dec 2001 13:52:28 +1300 -Message-ID: <1008809548.24470.48.camel@kant.mcmillan.net.nz> -MIME-Version: 1.0 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Thu, 2001-12-20 at 07:22, Michael Owens wrote: -> As long as each client's call is composed of a standalone transaction, there -> is no problem with external connection pools. But what about when a client's -> transactions spans two or more calls, such as SELECT FOR UPDATE? Then pooling -> is not safe: it offers no assurance of what may be interjected into an open -> transaction between calls. For example, each is a separate call to a shared -> connection: -> -> Client A: BEGIN WORK; SELECT last_name from customer for update where ; -> -> Client B: BEGIN WORK; SELECT street from customer for update where ; -> -> Client A: update customer set lastname= where ; COMMIT -> WORK; -> -> -> Now, isn't Client B's write lock gone with Client A's commit? Yet Client A's -> lock is still hanging around. While Client B's commit will close it, Client B -> has lost the assurance of its lock, defeating the purpose of SELECT FOR -> UPDATE. -> -> If this is corrent, then external connection pools limit what you can do with -> the database to a single call. Any transaction spanning more than one call is -> unsafe, because it is not isolated from other clients sharing the same -> connection. - -Oh, I see. You are absolutely correct that client-side pooling wouldn't -work in that situation of course. - -As an application developer nobody has forced me into such a corner yet, -however. Long running transactions are something I avoid like the -plague. - -Cheers, - Andrew. --- --------------------------------------------------------------------- -Andrew @ Catalyst .Net.NZ Ltd, PO Box 11-053, Manners St, Wellington -WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St -DDI: +64(4)916-7201 MOB: +64(21)635-694 OFFICE: +64(4)499-2267 - Are you enrolled at http://schoolreunions.co.nz/ yet? - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From owner-pgsql-hackers@hub.org Mon May 11 11:31:09 1998 -Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) - by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA03006 - for ; Mon, 11 May 1998 11:31:07 -0400 (EDT) -Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.17 $) with ESMTP id LAA01663 for ; Mon, 11 May 1998 11:24:42 -0400 (EDT) -Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id LAA21841; Mon, 11 May 1998 11:15:25 -0400 (EDT) -Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 11 May 1998 11:15:12 +0000 (EDT) -Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id LAA21683 for pgsql-hackers-outgoing; Mon, 11 May 1998 11:15:09 -0400 (EDT) -Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id LAA21451 for ; Mon, 11 May 1998 11:15:03 -0400 (EDT) -Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) - by sss.sss.pgh.pa.us (8.8.5/8.8.5) with ESMTP id LAA24915; - Mon, 11 May 1998 11:14:43 -0400 (EDT) -To: Brett McCormick -cc: hackers@postgreSQL.org -Subject: Re: [HACKERS] Re: [PATCHES] Try again: S_LOCK reduced contentionh] -In-reply-to: Your message of Mon, 11 May 1998 07:57:23 -0700 (PDT) - <13655.4384.345723.466046@abraxas.scene.com> -Date: Mon, 11 May 1998 11:14:43 -0400 -Message-ID: <24913.894899683@sss.pgh.pa.us> -From: Tom Lane -Sender: owner-pgsql-hackers@hub.org -Precedence: bulk -Status: RO - -Brett McCormick writes: -> same way that the current network socket is passed -- through an execv -> argument. hopefully, however, the non-execv()ing fork will be in 6.4. - -Um, you missed the point, Brett. David was hoping to transfer a client -connection from the postmaster to an *already existing* backend process. -Fork, with or without exec, solves the problem for a backend that's -started after the postmaster has accepted the client socket. - -This does lead to a different line of thought, however. Pre-started -backends would have access to the "master" connection socket on which -the postmaster listens for client connections, right? Suppose that we -fire the postmaster as postmaster, and demote it to being simply a -manufacturer of new backend processes as old ones get used up. Have -one of the idle backend processes be the one doing the accept() on the -master socket. Once it has a client connection, it performs the -authentication handshake and then starts serving the client (or just -quits if authentication fails). Meanwhile the next idle backend process -has executed accept() on the master socket and is waiting for the next -client; and shortly the postmaster/factory/whateverwecallitnow notices -that it needs to start another backend to add to the idle-backend pool. - -This'd probably need some interlocking among the backends. I have no -idea whether it'd be safe to have all the idle backends trying to -do accept() on the master socket simultaneously, but it sounds risky. -Better to use a mutex so that only one gets to do it while the others -sleep. - - regards, tom lane - - -From owner-pgsql-hackers@hub.org Mon May 11 11:35:55 1998 -Received: from hub.org (hub.org [209.47.148.200]) - by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA03043 - for ; Mon, 11 May 1998 11:35:53 -0400 (EDT) -Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id LAA23494; Mon, 11 May 1998 11:27:10 -0400 (EDT) -Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 11 May 1998 11:27:02 +0000 (EDT) -Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id LAA23473 for pgsql-hackers-outgoing; Mon, 11 May 1998 11:27:01 -0400 (EDT) -Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id LAA23462 for ; Mon, 11 May 1998 11:26:56 -0400 (EDT) -Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) - by sss.sss.pgh.pa.us (8.8.5/8.8.5) with ESMTP id LAA25006; - Mon, 11 May 1998 11:26:44 -0400 (EDT) -To: Brett McCormick -cc: hackers@postgreSQL.org -Subject: Re: [HACKERS] Re: [PATCHES] Try again: S_LOCK reduced contentionh] -In-reply-to: Your message of Mon, 11 May 1998 07:57:23 -0700 (PDT) - <13655.4384.345723.466046@abraxas.scene.com> -Date: Mon, 11 May 1998 11:26:44 -0400 -Message-ID: <25004.894900404@sss.pgh.pa.us> -From: Tom Lane -Sender: owner-pgsql-hackers@hub.org -Precedence: bulk -Status: RO - -Meanwhile, *I* missed the point about Brett's second comment :-( - -Brett McCormick writes: -> There will have to be some sort of arg parsing in any case, -> considering that you can pass configurable arguments to the backend.. - -If we do the sort of change David and I were just discussing, then the -pre-spawned backend would become responsible for parsing and dealing -with the PGOPTIONS portion of the client's connection request message. -That's just part of shifting the authentication handshake code from -postmaster to backend, so it shouldn't be too hard. - -BUT: the whole point is to be able to initialize the backend before it -is connected to a client. How much of the expensive backend startup -work depends on having the client connection options available? -Any work that needs to know the options will have to wait until after -the client connects. If that means most of the startup work can't -happen in advance anyway, then we're out of luck; a pre-started backend -won't save enough time to be worth the effort. (Unless we are willing -to eliminate or redefine the troublesome options...) - - regards, tom lane - -