diff --git a/doc/TODO.detail/crossdb b/doc/TODO.detail/crossdb new file mode 100644 index 0000000000..a68fb19828 --- /dev/null +++ b/doc/TODO.detail/crossdb @@ -0,0 +1,771 @@ +From pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 13:31:28 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6IVRZ13376 + for ; Thu, 6 Dec 2001 13:31:27 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6IVQa26949 + for ; Thu, 6 Dec 2001 13:31:26 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6IRvR61959 + for ; Thu, 6 Dec 2001 12:29:06 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org) +Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6IQ2m78192 + for ; Thu, 6 Dec 2001 13:26:05 -0500 (EST) + (envelope-from markw@mohawksoft.com) +Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) + by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id NAA10076 + for ; Thu, 6 Dec 2001 13:28:04 -0500 +Message-ID: <3C0FB8B4.382C7736@mohawksoft.com> +Date: Thu, 06 Dec 2001 13:28:04 -0500 +From: mlw +X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) +X-Accept-Language: en +MIME-Version: 1.0 +To: "pgsql-hackers@postgresql.org" +Subject: [HACKERS] Remote connections? +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +I just found out something about Oracle which that looks like something +that could be doable in PostgreSQL. + +What do you all think: + +Oracle's version is something like this: + +create [public] database link using [...] + +select * from sometable@remotelink + + +I was thinking how this could be done with postgreSQL. How hard would it +be to make something that is similar to a view, but executes a query +remotely? I envision something like this: + +create [public] link name query using [...] + +The table link will be similar to a view. It could be used like this: + +CREATE LINK test as select * from test WITH 'user=postgres host=remote +db=data'; + +SELECT * from test; + +or + +SELECT * from fubar join test on (fubar.id = test.id) ; + +So, what do you think? Impossible, possible, too hard? too easy? + +---------------------------(end of broadcast)--------------------------- +TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org + +From pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:12:28 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KCQZ19987 + for ; Thu, 6 Dec 2001 15:12:27 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KCQa13967 + for ; Thu, 6 Dec 2001 15:12:26 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6K6nR65020 + for ; Thu, 6 Dec 2001 14:07:54 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org) +Received: from ece.rice.edu (ece.rice.edu [128.42.4.34]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6K6Im96910 + for ; Thu, 6 Dec 2001 15:06:18 -0500 (EST) + (envelope-from reedstrm@rice.edu) +Received: from localhost (localhost [127.0.0.1]) + by ece.rice.edu (Postfix) with ESMTP + id A9E4E68A1D; Thu, 6 Dec 2001 14:06:17 -0600 (CST) +Received: from wallace.ece.rice.edu (wallace.ece.rice.edu [128.42.12.154]) + by ece.rice.edu (Postfix) with ESMTP + id EA06668A17; Thu, 6 Dec 2001 14:06:16 -0600 (CST) +Received: from reedstrm by wallace.ece.rice.edu with local (Exim 3.22 #1 (Debian)) + id 16C4me-0002uX-00; Thu, 06 Dec 2001 14:06:16 -0600 +Date: Thu, 6 Dec 2001 14:06:16 -0600 +From: "Ross J. Reedstrom" +To: mlw +cc: "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +Message-ID: <20011206140616.C10995@rice.edu> +Mail-Followup-To: "Ross J. Reedstrom" , + mlw , + "pgsql-hackers@postgresql.org" +References: <3C0FB8B4.382C7736@mohawksoft.com> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <3C0FB8B4.382C7736@mohawksoft.com> +User-Agent: Mutt/1.3.18i +X-Virus-Scanned: by AMaViS snapshot-20010714 +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: +> I just found out something about Oracle which that looks like something +> that could be doable in PostgreSQL. +> +> What do you all think: +> +> Oracle's version is something like this: +> +> create [public] database link using [...] +> +> select * from sometable@remotelink +> +> +> I was thinking how this could be done with postgreSQL. How hard would it +> be to make something that is similar to a view, but executes a query +> remotely? I envision something like this: +> +> create [public] link name query using [...] +> +> The table link will be similar to a view. It could be used like this: +> +> CREATE LINK test as select * from test WITH 'user=postgres host=remote +> db=data'; +> +> SELECT * from test; +> +> or +> +> SELECT * from fubar join test on (fubar.id = test.id) ; +> +> So, what do you think? Impossible, possible, too hard? too easy? + +Here we come, full circle. This is just about where I came on board. +Many moons ago, I started looking at Mariposa, in the hopes of forward +patching it into PostgreSQL, and generalizing the 'remote' part to allow +exactly the sort of access you described above. + +The biggest problem with this is transactional semantics: you need +two-stage commits to get this right, and we don't hav'em. (Has there +been an indepth discussion concerning what how hard it would be to do +that with postgresql?) + +The _actual_ biggest problem was my lack of knowledge of the PostgreSQL +codebase ;-) + +Ross +-- +Ross Reedstrom, Ph.D. reedstrm@rice.edu +Executive Director phone: 713-348-6166 +Gulf Coast Consortium for Bioinformatics fax: 713-348-6182 +Rice University MS-39 +Houston, TX 77005 + +---------------------------(end of broadcast)--------------------------- +TIP 4: Don't 'kill -9' the postmaster + +From pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:31:27 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KVQZ21158 + for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KVQa22900 + for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6KRrR65596 + for ; Thu, 6 Dec 2001 14:28:55 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org) +Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6KJXm97564 + for ; Thu, 6 Dec 2001 15:19:33 -0500 (EST) + (envelope-from markw@mohawksoft.com) +Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) + by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id PAA10771; + Thu, 6 Dec 2001 15:21:13 -0500 +Message-ID: <3C0FD339.6F663329@mohawksoft.com> +Date: Thu, 06 Dec 2001 15:21:13 -0500 +From: mlw +X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) +X-Accept-Language: en +MIME-Version: 1.0 +To: "Ross J. Reedstrom" +cc: "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +"Ross J. Reedstrom" wrote: +> +> On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: +> > I just found out something about Oracle which that looks like something +> > that could be doable in PostgreSQL. +> > +> > What do you all think: +> > +> > Oracle's version is something like this: +> > +> > create [public] database link using [...] +> > +> > select * from sometable@remotelink +> > +> > +> > I was thinking how this could be done with postgreSQL. How hard would it +> > be to make something that is similar to a view, but executes a query +> > remotely? I envision something like this: +> > +> > create [public] link name query using [...] +> > +> > The table link will be similar to a view. It could be used like this: +> > +> > CREATE LINK test as select * from test WITH 'user=postgres host=remote +> > db=data'; +> > +> > SELECT * from test; +> > +> > or +> > +> > SELECT * from fubar join test on (fubar.id = test.id) ; +> > +> > So, what do you think? Impossible, possible, too hard? too easy? +> +> Here we come, full circle. This is just about where I came on board. +> Many moons ago, I started looking at Mariposa, in the hopes of forward +> patching it into PostgreSQL, and generalizing the 'remote' part to allow +> exactly the sort of access you described above. +> +> The biggest problem with this is transactional semantics: you need +> two-stage commits to get this right, and we don't hav'em. (Has there +> been an indepth discussion concerning what how hard it would be to do +> that with postgresql?) +> +> The _actual_ biggest problem was my lack of knowledge of the PostgreSQL +> codebase ;-) + +I think we can we can dispense worrying about two stage commits, if we +assume that remote connections are treated as views with no rules. As +long as remote tables are "read only" then the implementation is much +easier. + +I too find the internals of PostgreSQL virtually incomprehensible at the +internal level. If there were a document somewhere which published how a +function could return multiple tuples, remote views would be a trivial +undertaking. It could look like: + +select * from remote('select *from table', 'user=postgres host=outland +db=remote'); + +---------------------------(end of broadcast)--------------------------- +TIP 4: Don't 'kill -9' the postmaster + +From pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 17:11:29 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6MBSZ06676 + for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6MBSq07059 + for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6M6sR68489 + for ; Thu, 6 Dec 2001 16:08:16 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org) +Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6M3Im04451 + for ; Thu, 6 Dec 2001 17:03:18 -0500 (EST) + (envelope-from joseph.conway@home.com) +Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) + by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) + with ESMTP id 1707032; Thu, 06 Dec 2001 14:03:14 -0800 +Message-ID: <3C0FEB2C.70000@home.com> +Date: Thu, 06 Dec 2001 14:03:24 -0800 +From: Joe Conway +User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 +X-Accept-Language: en-us +MIME-Version: 1.0 +To: mlw +cc: "Ross J. Reedstrom" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> +Content-Type: text/plain; charset=us-ascii; format=flowed +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +mlw wrote: + +> I too find the internals of PostgreSQL virtually incomprehensible at the +> internal level. If there were a document somewhere which published how a +> function could return multiple tuples, remote views would be a trivial +> undertaking. It could look like: +> +> select * from remote('select *from table', 'user=postgres host=outland +> db=remote'); +> + +See contrib/dblink in the 7.2 beta. It was my attempt inspired by +Oracle's dblink and some code that you (mlw) posted a while back. Based +on the limitations wrt returning muliple tuples, I wound up with +something like: + +lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select +dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres +password=postgres','select proname from pg_proc') as dblink_p) as t1; + +Which, as has been pointed out more than once, is pretty ugly, but at +least it's a start ;-) + + +Joe + + +---------------------------(end of broadcast)--------------------------- +TIP 4: Don't 'kill -9' the postmaster + +From pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 18:41:31 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6NfPZ12249 + for ; Thu, 6 Dec 2001 18:41:25 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6NfQq03244 + for ; Thu, 6 Dec 2001 18:41:26 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6NbOR70960 + for ; Thu, 6 Dec 2001 17:38:19 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org) +Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6NIgm07232 + for ; Thu, 6 Dec 2001 18:18:43 -0500 (EST) + (envelope-from alex@pilosoft.com) +Received: from localhost (alexmail@localhost) + by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id SAA23999; + Thu, 6 Dec 2001 18:23:22 -0500 (EST) +Date: Thu, 6 Dec 2001 18:23:22 -0500 (EST) +From: Alex Pilosov +To: mlw +cc: "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +In-Reply-To: <3C0FD339.6F663329@mohawksoft.com> +Message-ID: +MIME-Version: 1.0 +Content-Type: TEXT/PLAIN; charset=US-ASCII +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +On Thu, 6 Dec 2001, mlw wrote: + +> I too find the internals of PostgreSQL virtually incomprehensible at the +> internal level. If there were a document somewhere which published how a +> function could return multiple tuples, remote views would be a trivial +> undertaking. It could look like: +> +> select * from remote('select *from table', 'user=postgres host=outland +> db=remote'); +This isn't possible yet. I was working on implementation of this, about +80% done, but never finished. Now I'm out of time to work more on this for +a while. :( + +Let me know if you want my code. + +-alex + + +---------------------------(end of broadcast)--------------------------- +TIP 6: Have you searched our list archives? + +http://archives.postgresql.org + +From pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 00:32:59 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB75WsZ06911 + for ; Fri, 7 Dec 2001 00:32:54 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB75Wtq16801 + for ; Fri, 7 Dec 2001 00:32:55 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB75SCR80525 + for ; Thu, 6 Dec 2001 23:29:17 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org) +Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7593m27913 + for ; Fri, 7 Dec 2001 00:09:03 -0500 (EST) + (envelope-from markw@mohawksoft.com) +Received: from mohawksoft.com (localhost [127.0.0.1]) + by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7561030613; + Fri, 7 Dec 2001 00:06:01 -0500 +Message-ID: <3C104E38.DA19C867@mohawksoft.com> +Date: Fri, 07 Dec 2001 00:06:01 -0500 +From: mlw +X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) +X-Accept-Language: en +MIME-Version: 1.0 +To: Joe Conway +cc: "Ross J. Reedstrom" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +Hey this looks really cool. It looks like something I was thinking about doing. +I have a couple suggestions that could make it a little better, I hope you will +not be offended. (If you want my help, I'll chip in!) + +Why not use a binary cursor? That way native types can slip through without the +overhead of conversion. + +Right now you get all rows up front, you may be able to increase overall +performance by fetching only a few rows at a time, rather than get everything +all at once. (Think on the order of 4 million rows from your remote query!) +Execute the commit at the end of processing. There are even some asynchronous +functions you may be able to utilize to reduce the I/O bottleneck. Use the +synchronous function first, then before you return initiate an asynchronous +read. Every successive pass through the function, read the newly arrived tuple, +and initiate the next asynchronous read. (The two machine could be processing +the query simultaneously, and this could even IMPROVE performance over a single +system for heavy duty queries.) + +Setup a hash table for field names, rather than requiring field numbers. (Keep +field number for efficiency, of course.) + +You could eliminate having to pass the result pointer around by keeping a +static array in your extension. Use something like Oracle's "contains" notation +of result number. Where each instantiation of "contains()" and "score()" +require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have +some code that does this for a PostgreSQL extension (it implements contains) on +my website (pgcontains, under download). It is ugly but works for the most +part. + +Seriously, your stuff looks great. I think it could be the beginning of a +fairly usable system for my company. Any help you need/want, just let me know. + + +Joe Conway wrote: +> +> mlw wrote: +> +> > I too find the internals of PostgreSQL virtually incomprehensible at the +> > internal level. If there were a document somewhere which published how a +> > function could return multiple tuples, remote views would be a trivial +> > undertaking. It could look like: +> > +> > select * from remote('select *from table', 'user=postgres host=outland +> > db=remote'); +> > +> +> See contrib/dblink in the 7.2 beta. It was my attempt inspired by +> Oracle's dblink and some code that you (mlw) posted a while back. Based +> on the limitations wrt returning muliple tuples, I wound up with +> something like: +> +> lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select +> dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres +> password=postgres','select proname from pg_proc') as dblink_p) as t1; +> +> Which, as has been pointed out more than once, is pretty ugly, but at +> least it's a start ;-) +> +> Joe +> +> ---------------------------(end of broadcast)--------------------------- +> TIP 4: Don't 'kill -9' the postmaster + +---------------------------(end of broadcast)--------------------------- +TIP 3: if posting/reading through Usenet, please send an appropriate +subscribe-nomail command to majordomo@postgresql.org so that your +message can get through to the mailing list cleanly + +From pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 02:51:51 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB77poZ14221 + for ; Fri, 7 Dec 2001 02:51:50 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB77pqq08152 + for ; Fri, 7 Dec 2001 02:51:52 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB77lwR84105 + for ; Fri, 7 Dec 2001 01:49:04 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org) +Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB77bmm32783 + for ; Fri, 7 Dec 2001 02:37:49 -0500 (EST) + (envelope-from oleg@sai.msu.su) +Received: from ra (ra [158.250.29.2]) + by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id KAA04160; + Fri, 7 Dec 2001 10:37:04 +0300 (GMT) +Date: Fri, 7 Dec 2001 10:37:04 +0300 (GMT) +From: Oleg Bartunov +X-X-Sender: +To: mlw +cc: Joe Conway , + "Ross J. Reedstrom" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +In-Reply-To: <3C104E38.DA19C867@mohawksoft.com> +Message-ID: +MIME-Version: 1.0 +Content-Type: TEXT/PLAIN; charset=US-ASCII +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +On Fri, 7 Dec 2001, mlw wrote: + +> +> You could eliminate having to pass the result pointer around by keeping a +> static array in your extension. Use something like Oracle's "contains" notation +> of result number. Where each instantiation of "contains()" and "score()" +> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have +> some code that does this for a PostgreSQL extension (it implements contains) on +> my website (pgcontains, under download). It is ugly but works for the most +> part. + +contrib/intarray does this job very well + +> +> Seriously, your stuff looks great. I think it could be the beginning of a +> fairly usable system for my company. Any help you need/want, just let me know. +> +> +> Joe Conway wrote: +> > +> > mlw wrote: +> > +> > > I too find the internals of PostgreSQL virtually incomprehensible at the +> > > internal level. If there were a document somewhere which published how a +> > > function could return multiple tuples, remote views would be a trivial +> > > undertaking. It could look like: +> > > +> > > select * from remote('select *from table', 'user=postgres host=outland +> > > db=remote'); +> > > +> > +> > See contrib/dblink in the 7.2 beta. It was my attempt inspired by +> > Oracle's dblink and some code that you (mlw) posted a while back. Based +> > on the limitations wrt returning muliple tuples, I wound up with +> > something like: +> > +> > lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select +> > dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres +> > password=postgres','select proname from pg_proc') as dblink_p) as t1; +> > +> > Which, as has been pointed out more than once, is pretty ugly, but at +> > least it's a start ;-) +> > +> > Joe +> > +> > ---------------------------(end of broadcast)--------------------------- +> > TIP 4: Don't 'kill -9' the postmaster +> +> ---------------------------(end of broadcast)--------------------------- +> TIP 3: if posting/reading through Usenet, please send an appropriate +> subscribe-nomail command to majordomo@postgresql.org so that your +> message can get through to the mailing list cleanly +> + + Regards, + Oleg +_____________________________________________________________ +Oleg Bartunov, sci.researcher, hostmaster of AstroNet, +Sternberg Astronomical Institute, Moscow University (Russia) +Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ +phone: +007(095)939-16-83, +007(095)939-23-83 + + +---------------------------(end of broadcast)--------------------------- +TIP 3: if posting/reading through Usenet, please send an appropriate +subscribe-nomail command to majordomo@postgresql.org so that your +message can get through to the mailing list cleanly + +From pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org Mon Dec 10 12:35:01 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBAHZ0Z09772 + for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fBAHZ0a11739 + for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBAHRAR20284 + for ; Mon, 10 Dec 2001 11:32:00 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org) +Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7DhOm75114 + for ; Fri, 7 Dec 2001 08:43:24 -0500 (EST) + (envelope-from markw@mohawksoft.com) +Received: from mohawksoft.com (localhost [127.0.0.1]) + by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7De0000437; + Fri, 7 Dec 2001 08:40:01 -0500 +Message-ID: <3C10C6B0.865669C1@mohawksoft.com> +Date: Fri, 07 Dec 2001 08:40:00 -0500 +From: mlw +X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) +X-Accept-Language: en +MIME-Version: 1.0 +To: Oleg Bartunov +cc: Joe Conway , + "Ross J. Reedstrom" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +References: +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +The dblink code is a very cool idea. + +It got me thinking, what if, just thinking out load here, it was redesigned as +something a little more grandeous. + +Imagine this: + + +select dblink('select * from table', 'table_name', 'db=oracle.test user=chris +passwd=knight', 1) as t1, dblink('table2_name', 1) as t2 + + +Just something to think about. + +The first instance of dblink would take 4 parameters: query, table which it +returns, connect string, and link token. + +The second instance of dblink would just take the name of the table which it +returns and a link token. + +The cool bit is the notion that the query string could specify different +databases or even .DBF libraries. + +Just something to think about. + +It would REALLY be great if functions could return multiple tuples! + +---------------------------(end of broadcast)--------------------------- +TIP 5: Have you checked our extensive FAQ? + +http://www.postgresql.org/users-lounge/docs/faq.html + +From pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 12:32:26 2001 +Return-path: +Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB7HWMZ26245 + for ; Fri, 7 Dec 2001 12:32:22 -0500 (EST) +Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB7HWLB14472 + for ; Fri, 7 Dec 2001 12:32:21 -0500 (EST) +Received: from postgresql.org (postgresql.org [64.49.215.8]) + by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB7HSHR01506 + for ; Fri, 7 Dec 2001 11:29:07 -0600 (CST) + (envelope-from pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org) +Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7HQfm90424 + for ; Fri, 7 Dec 2001 12:26:42 -0500 (EST) + (envelope-from joseph.conway@home.com) +Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) + by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) + with ESMTP id 1722339; Fri, 07 Dec 2001 09:26:46 -0800 +Message-ID: <3C10FBD7.4070602@home.com> +Date: Fri, 07 Dec 2001 09:26:47 -0800 +From: Joe Conway +User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 +X-Accept-Language: en-us +MIME-Version: 1.0 +To: mlw +cc: "Ross J. Reedstrom" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Remote connections? +References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> <3C104E38.DA19C867@mohawksoft.com> +Content-Type: text/plain; charset=us-ascii; format=flowed +Content-Transfer-Encoding: 7bit +Precedence: bulk +Sender: pgsql-hackers-owner@postgresql.org +Status: OR + +mlw wrote: + +> Hey this looks really cool. It looks like something I was thinking about doing. +> I have a couple suggestions that could make it a little better, I hope you will +> not be offended. (If you want my help, I'll chip in!) +> + + +Thanks! Suggestions welcomed. + +> Why not use a binary cursor? That way native types can slip through without the +> overhead of conversion. +> + + +I wasn't sure that would work. Would you create dblink_tok as returning +opaque then? + + +> Right now you get all rows up front, you may be able to increase overall +> performance by fetching only a few rows at a time, rather than get everything +> all at once. (Think on the order of 4 million rows from your remote query!) +> Execute the commit at the end of processing. There are even some asynchronous +> functions you may be able to utilize to reduce the I/O bottleneck. Use the +> synchronous function first, then before you return initiate an asynchronous +> read. Every successive pass through the function, read the newly arrived tuple, +> and initiate the next asynchronous read. (The two machine could be processing +> the query simultaneously, and this could even IMPROVE performance over a single +> system for heavy duty queries.) + + +Interesting . . . but aren't there some issues with the asynch functions? + +> +> Setup a hash table for field names, rather than requiring field numbers. (Keep +> field number for efficiency, of course.) +> +> You could eliminate having to pass the result pointer around by keeping a +> static array in your extension. Use something like Oracle's "contains" notation +> of result number. Where each instantiation of "contains()" and "score()" +> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have +> some code that does this for a PostgreSQL extension (it implements contains) on +> my website (pgcontains, under download). It is ugly but works for the most +> part. +> + + +I thought about the static array, but I'm not familiar with Oracle +contains() and score() -- I'm only fluent enough with Oracle to be +dangerous. Guess I'll have to dig out the books . . . + + +> Seriously, your stuff looks great. I think it could be the beginning of a +> fairly usable system for my company. Any help you need/want, just let me know. +> + +I am planning to improve dblink during the next release cycle, so I'll +keep all this in mind (and might take you up on the help offer too!). I +was hoping we'd have functions returning tuples by now, which would +improve this extension dramatically. Unfortunately, it sounds like Alex +won't have time to finish that even for 7.3 :( + +Alex, can we get a look at your latest code? Is it any different the +your last submission to PATCHES? + +Joe + + +---------------------------(end of broadcast)--------------------------- +TIP 3: if posting/reading through Usenet, please send an appropriate +subscribe-nomail command to majordomo@postgresql.org so that your +message can get through to the mailing list cleanly +