From pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 13:31:28 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6IVRZ13376 for ; Thu, 6 Dec 2001 13:31:27 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6IVQa26949 for ; Thu, 6 Dec 2001 13:31:26 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6IRvR61959 for ; Thu, 6 Dec 2001 12:29:06 -0600 (CST) (envelope-from pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org) Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6IQ2m78192 for ; Thu, 6 Dec 2001 13:26:05 -0500 (EST) (envelope-from markw@mohawksoft.com) Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id NAA10076 for ; Thu, 6 Dec 2001 13:28:04 -0500 Message-ID: <3C0FB8B4.382C7736@mohawksoft.com> Date: Thu, 06 Dec 2001 13:28:04 -0500 From: mlw X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) X-Accept-Language: en MIME-Version: 1.0 To: "pgsql-hackers@postgresql.org" Subject: [HACKERS] Remote connections? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR I just found out something about Oracle which that looks like something that could be doable in PostgreSQL. What do you all think: Oracle's version is something like this: create [public] database link using [...] select * from sometable@remotelink I was thinking how this could be done with postgreSQL. How hard would it be to make something that is similar to a view, but executes a query remotely? I envision something like this: create [public] link name query using [...] The table link will be similar to a view. It could be used like this: CREATE LINK test as select * from test WITH 'user=postgres host=remote db=data'; SELECT * from test; or SELECT * from fubar join test on (fubar.id = test.id) ; So, what do you think? Impossible, possible, too hard? too easy? ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org From pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:12:28 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KCQZ19987 for ; Thu, 6 Dec 2001 15:12:27 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KCQa13967 for ; Thu, 6 Dec 2001 15:12:26 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6K6nR65020 for ; Thu, 6 Dec 2001 14:07:54 -0600 (CST) (envelope-from pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org) Received: from ece.rice.edu (ece.rice.edu [128.42.4.34]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6K6Im96910 for ; Thu, 6 Dec 2001 15:06:18 -0500 (EST) (envelope-from reedstrm@rice.edu) Received: from localhost (localhost [127.0.0.1]) by ece.rice.edu (Postfix) with ESMTP id A9E4E68A1D; Thu, 6 Dec 2001 14:06:17 -0600 (CST) Received: from wallace.ece.rice.edu (wallace.ece.rice.edu [128.42.12.154]) by ece.rice.edu (Postfix) with ESMTP id EA06668A17; Thu, 6 Dec 2001 14:06:16 -0600 (CST) Received: from reedstrm by wallace.ece.rice.edu with local (Exim 3.22 #1 (Debian)) id 16C4me-0002uX-00; Thu, 06 Dec 2001 14:06:16 -0600 Date: Thu, 6 Dec 2001 14:06:16 -0600 From: "Ross J. Reedstrom" To: mlw cc: "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? Message-ID: <20011206140616.C10995@rice.edu> Mail-Followup-To: "Ross J. Reedstrom" , mlw , "pgsql-hackers@postgresql.org" References: <3C0FB8B4.382C7736@mohawksoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3C0FB8B4.382C7736@mohawksoft.com> User-Agent: Mutt/1.3.18i X-Virus-Scanned: by AMaViS snapshot-20010714 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: > I just found out something about Oracle which that looks like something > that could be doable in PostgreSQL. > > What do you all think: > > Oracle's version is something like this: > > create [public] database link using [...] > > select * from sometable@remotelink > > > I was thinking how this could be done with postgreSQL. How hard would it > be to make something that is similar to a view, but executes a query > remotely? I envision something like this: > > create [public] link name query using [...] > > The table link will be similar to a view. It could be used like this: > > CREATE LINK test as select * from test WITH 'user=postgres host=remote > db=data'; > > SELECT * from test; > > or > > SELECT * from fubar join test on (fubar.id = test.id) ; > > So, what do you think? Impossible, possible, too hard? too easy? Here we come, full circle. This is just about where I came on board. Many moons ago, I started looking at Mariposa, in the hopes of forward patching it into PostgreSQL, and generalizing the 'remote' part to allow exactly the sort of access you described above. The biggest problem with this is transactional semantics: you need two-stage commits to get this right, and we don't hav'em. (Has there been an indepth discussion concerning what how hard it would be to do that with postgresql?) The _actual_ biggest problem was my lack of knowledge of the PostgreSQL codebase ;-) Ross -- Ross Reedstrom, Ph.D. reedstrm@rice.edu Executive Director phone: 713-348-6166 Gulf Coast Consortium for Bioinformatics fax: 713-348-6182 Rice University MS-39 Houston, TX 77005 ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:31:27 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KVQZ21158 for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KVQa22900 for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6KRrR65596 for ; Thu, 6 Dec 2001 14:28:55 -0600 (CST) (envelope-from pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org) Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6KJXm97564 for ; Thu, 6 Dec 2001 15:19:33 -0500 (EST) (envelope-from markw@mohawksoft.com) Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id PAA10771; Thu, 6 Dec 2001 15:21:13 -0500 Message-ID: <3C0FD339.6F663329@mohawksoft.com> Date: Thu, 06 Dec 2001 15:21:13 -0500 From: mlw X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) X-Accept-Language: en MIME-Version: 1.0 To: "Ross J. Reedstrom" cc: "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR "Ross J. Reedstrom" wrote: > > On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: > > I just found out something about Oracle which that looks like something > > that could be doable in PostgreSQL. > > > > What do you all think: > > > > Oracle's version is something like this: > > > > create [public] database link using [...] > > > > select * from sometable@remotelink > > > > > > I was thinking how this could be done with postgreSQL. How hard would it > > be to make something that is similar to a view, but executes a query > > remotely? I envision something like this: > > > > create [public] link name query using [...] > > > > The table link will be similar to a view. It could be used like this: > > > > CREATE LINK test as select * from test WITH 'user=postgres host=remote > > db=data'; > > > > SELECT * from test; > > > > or > > > > SELECT * from fubar join test on (fubar.id = test.id) ; > > > > So, what do you think? Impossible, possible, too hard? too easy? > > Here we come, full circle. This is just about where I came on board. > Many moons ago, I started looking at Mariposa, in the hopes of forward > patching it into PostgreSQL, and generalizing the 'remote' part to allow > exactly the sort of access you described above. > > The biggest problem with this is transactional semantics: you need > two-stage commits to get this right, and we don't hav'em. (Has there > been an indepth discussion concerning what how hard it would be to do > that with postgresql?) > > The _actual_ biggest problem was my lack of knowledge of the PostgreSQL > codebase ;-) I think we can we can dispense worrying about two stage commits, if we assume that remote connections are treated as views with no rules. As long as remote tables are "read only" then the implementation is much easier. I too find the internals of PostgreSQL virtually incomprehensible at the internal level. If there were a document somewhere which published how a function could return multiple tuples, remote views would be a trivial undertaking. It could look like: select * from remote('select *from table', 'user=postgres host=outland db=remote'); ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 17:11:29 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6MBSZ06676 for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6MBSq07059 for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6M6sR68489 for ; Thu, 6 Dec 2001 16:08:16 -0600 (CST) (envelope-from pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org) Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6M3Im04451 for ; Thu, 6 Dec 2001 17:03:18 -0500 (EST) (envelope-from joseph.conway@home.com) Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) with ESMTP id 1707032; Thu, 06 Dec 2001 14:03:14 -0800 Message-ID: <3C0FEB2C.70000@home.com> Date: Thu, 06 Dec 2001 14:03:24 -0800 From: Joe Conway User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 X-Accept-Language: en-us MIME-Version: 1.0 To: mlw cc: "Ross J. Reedstrom" , "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR mlw wrote: > I too find the internals of PostgreSQL virtually incomprehensible at the > internal level. If there were a document somewhere which published how a > function could return multiple tuples, remote views would be a trivial > undertaking. It could look like: > > select * from remote('select *from table', 'user=postgres host=outland > db=remote'); > See contrib/dblink in the 7.2 beta. It was my attempt inspired by Oracle's dblink and some code that you (mlw) posted a while back. Based on the limitations wrt returning muliple tuples, I wound up with something like: lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres password=postgres','select proname from pg_proc') as dblink_p) as t1; Which, as has been pointed out more than once, is pretty ugly, but at least it's a start ;-) Joe ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 18:41:31 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6NfPZ12249 for ; Thu, 6 Dec 2001 18:41:25 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6NfQq03244 for ; Thu, 6 Dec 2001 18:41:26 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6NbOR70960 for ; Thu, 6 Dec 2001 17:38:19 -0600 (CST) (envelope-from pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org) Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6NIgm07232 for ; Thu, 6 Dec 2001 18:18:43 -0500 (EST) (envelope-from alex@pilosoft.com) Received: from localhost (alexmail@localhost) by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id SAA23999; Thu, 6 Dec 2001 18:23:22 -0500 (EST) Date: Thu, 6 Dec 2001 18:23:22 -0500 (EST) From: Alex Pilosov To: mlw cc: "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? In-Reply-To: <3C0FD339.6F663329@mohawksoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Thu, 6 Dec 2001, mlw wrote: > I too find the internals of PostgreSQL virtually incomprehensible at the > internal level. If there were a document somewhere which published how a > function could return multiple tuples, remote views would be a trivial > undertaking. It could look like: > > select * from remote('select *from table', 'user=postgres host=outland > db=remote'); This isn't possible yet. I was working on implementation of this, about 80% done, but never finished. Now I'm out of time to work more on this for a while. :( Let me know if you want my code. -alex ---------------------------(end of broadcast)--------------------------- TIP 6: Have you searched our list archives? http://archives.postgresql.org From pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 00:32:59 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB75WsZ06911 for ; Fri, 7 Dec 2001 00:32:54 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB75Wtq16801 for ; Fri, 7 Dec 2001 00:32:55 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB75SCR80525 for ; Thu, 6 Dec 2001 23:29:17 -0600 (CST) (envelope-from pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org) Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7593m27913 for ; Fri, 7 Dec 2001 00:09:03 -0500 (EST) (envelope-from markw@mohawksoft.com) Received: from mohawksoft.com (localhost [127.0.0.1]) by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7561030613; Fri, 7 Dec 2001 00:06:01 -0500 Message-ID: <3C104E38.DA19C867@mohawksoft.com> Date: Fri, 07 Dec 2001 00:06:01 -0500 From: mlw X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) X-Accept-Language: en MIME-Version: 1.0 To: Joe Conway cc: "Ross J. Reedstrom" , "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Hey this looks really cool. It looks like something I was thinking about doing. I have a couple suggestions that could make it a little better, I hope you will not be offended. (If you want my help, I'll chip in!) Why not use a binary cursor? That way native types can slip through without the overhead of conversion. Right now you get all rows up front, you may be able to increase overall performance by fetching only a few rows at a time, rather than get everything all at once. (Think on the order of 4 million rows from your remote query!) Execute the commit at the end of processing. There are even some asynchronous functions you may be able to utilize to reduce the I/O bottleneck. Use the synchronous function first, then before you return initiate an asynchronous read. Every successive pass through the function, read the newly arrived tuple, and initiate the next asynchronous read. (The two machine could be processing the query simultaneously, and this could even IMPROVE performance over a single system for heavy duty queries.) Setup a hash table for field names, rather than requiring field numbers. (Keep field number for efficiency, of course.) You could eliminate having to pass the result pointer around by keeping a static array in your extension. Use something like Oracle's "contains" notation of result number. Where each instantiation of "contains()" and "score()" require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have some code that does this for a PostgreSQL extension (it implements contains) on my website (pgcontains, under download). It is ugly but works for the most part. Seriously, your stuff looks great. I think it could be the beginning of a fairly usable system for my company. Any help you need/want, just let me know. Joe Conway wrote: > > mlw wrote: > > > I too find the internals of PostgreSQL virtually incomprehensible at the > > internal level. If there were a document somewhere which published how a > > function could return multiple tuples, remote views would be a trivial > > undertaking. It could look like: > > > > select * from remote('select *from table', 'user=postgres host=outland > > db=remote'); > > > > See contrib/dblink in the 7.2 beta. It was my attempt inspired by > Oracle's dblink and some code that you (mlw) posted a while back. Based > on the limitations wrt returning muliple tuples, I wound up with > something like: > > lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select > dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres > password=postgres','select proname from pg_proc') as dblink_p) as t1; > > Which, as has been pointed out more than once, is pretty ugly, but at > least it's a start ;-) > > Joe > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster ---------------------------(end of broadcast)--------------------------- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly From pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 02:51:51 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB77poZ14221 for ; Fri, 7 Dec 2001 02:51:50 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB77pqq08152 for ; Fri, 7 Dec 2001 02:51:52 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB77lwR84105 for ; Fri, 7 Dec 2001 01:49:04 -0600 (CST) (envelope-from pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org) Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB77bmm32783 for ; Fri, 7 Dec 2001 02:37:49 -0500 (EST) (envelope-from oleg@sai.msu.su) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id KAA04160; Fri, 7 Dec 2001 10:37:04 +0300 (GMT) Date: Fri, 7 Dec 2001 10:37:04 +0300 (GMT) From: Oleg Bartunov X-X-Sender: To: mlw cc: Joe Conway , "Ross J. Reedstrom" , "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? In-Reply-To: <3C104E38.DA19C867@mohawksoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Fri, 7 Dec 2001, mlw wrote: > > You could eliminate having to pass the result pointer around by keeping a > static array in your extension. Use something like Oracle's "contains" notation > of result number. Where each instantiation of "contains()" and "score()" > require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have > some code that does this for a PostgreSQL extension (it implements contains) on > my website (pgcontains, under download). It is ugly but works for the most > part. contrib/intarray does this job very well > > Seriously, your stuff looks great. I think it could be the beginning of a > fairly usable system for my company. Any help you need/want, just let me know. > > > Joe Conway wrote: > > > > mlw wrote: > > > > > I too find the internals of PostgreSQL virtually incomprehensible at the > > > internal level. If there were a document somewhere which published how a > > > function could return multiple tuples, remote views would be a trivial > > > undertaking. It could look like: > > > > > > select * from remote('select *from table', 'user=postgres host=outland > > > db=remote'); > > > > > > > See contrib/dblink in the 7.2 beta. It was my attempt inspired by > > Oracle's dblink and some code that you (mlw) posted a while back. Based > > on the limitations wrt returning muliple tuples, I wound up with > > something like: > > > > lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select > > dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres > > password=postgres','select proname from pg_proc') as dblink_p) as t1; > > > > Which, as has been pointed out more than once, is pretty ugly, but at > > least it's a start ;-) > > > > Joe > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 4: Don't 'kill -9' the postmaster > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 ---------------------------(end of broadcast)--------------------------- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly From pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org Mon Dec 10 12:35:01 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBAHZ0Z09772 for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fBAHZ0a11739 for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBAHRAR20284 for ; Mon, 10 Dec 2001 11:32:00 -0600 (CST) (envelope-from pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org) Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7DhOm75114 for ; Fri, 7 Dec 2001 08:43:24 -0500 (EST) (envelope-from markw@mohawksoft.com) Received: from mohawksoft.com (localhost [127.0.0.1]) by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7De0000437; Fri, 7 Dec 2001 08:40:01 -0500 Message-ID: <3C10C6B0.865669C1@mohawksoft.com> Date: Fri, 07 Dec 2001 08:40:00 -0500 From: mlw X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) X-Accept-Language: en MIME-Version: 1.0 To: Oleg Bartunov cc: Joe Conway , "Ross J. Reedstrom" , "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR The dblink code is a very cool idea. It got me thinking, what if, just thinking out load here, it was redesigned as something a little more grandeous. Imagine this: select dblink('select * from table', 'table_name', 'db=oracle.test user=chris passwd=knight', 1) as t1, dblink('table2_name', 1) as t2 Just something to think about. The first instance of dblink would take 4 parameters: query, table which it returns, connect string, and link token. The second instance of dblink would just take the name of the table which it returns and a link token. The cool bit is the notion that the query string could specify different databases or even .DBF libraries. Just something to think about. It would REALLY be great if functions could return multiple tuples! ---------------------------(end of broadcast)--------------------------- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html From pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 12:32:26 2001 Return-path: Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB7HWMZ26245 for ; Fri, 7 Dec 2001 12:32:22 -0500 (EST) Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB7HWLB14472 for ; Fri, 7 Dec 2001 12:32:21 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB7HSHR01506 for ; Fri, 7 Dec 2001 11:29:07 -0600 (CST) (envelope-from pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org) Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7HQfm90424 for ; Fri, 7 Dec 2001 12:26:42 -0500 (EST) (envelope-from joseph.conway@home.com) Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) with ESMTP id 1722339; Fri, 07 Dec 2001 09:26:46 -0800 Message-ID: <3C10FBD7.4070602@home.com> Date: Fri, 07 Dec 2001 09:26:47 -0800 From: Joe Conway User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 X-Accept-Language: en-us MIME-Version: 1.0 To: mlw cc: "Ross J. Reedstrom" , "pgsql-hackers@postgresql.org" Subject: Re: [HACKERS] Remote connections? References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> <3C104E38.DA19C867@mohawksoft.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR mlw wrote: > Hey this looks really cool. It looks like something I was thinking about doing. > I have a couple suggestions that could make it a little better, I hope you will > not be offended. (If you want my help, I'll chip in!) > Thanks! Suggestions welcomed. > Why not use a binary cursor? That way native types can slip through without the > overhead of conversion. > I wasn't sure that would work. Would you create dblink_tok as returning opaque then? > Right now you get all rows up front, you may be able to increase overall > performance by fetching only a few rows at a time, rather than get everything > all at once. (Think on the order of 4 million rows from your remote query!) > Execute the commit at the end of processing. There are even some asynchronous > functions you may be able to utilize to reduce the I/O bottleneck. Use the > synchronous function first, then before you return initiate an asynchronous > read. Every successive pass through the function, read the newly arrived tuple, > and initiate the next asynchronous read. (The two machine could be processing > the query simultaneously, and this could even IMPROVE performance over a single > system for heavy duty queries.) Interesting . . . but aren't there some issues with the asynch functions? > > Setup a hash table for field names, rather than requiring field numbers. (Keep > field number for efficiency, of course.) > > You could eliminate having to pass the result pointer around by keeping a > static array in your extension. Use something like Oracle's "contains" notation > of result number. Where each instantiation of "contains()" and "score()" > require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have > some code that does this for a PostgreSQL extension (it implements contains) on > my website (pgcontains, under download). It is ugly but works for the most > part. > I thought about the static array, but I'm not familiar with Oracle contains() and score() -- I'm only fluent enough with Oracle to be dangerous. Guess I'll have to dig out the books . . . > Seriously, your stuff looks great. I think it could be the beginning of a > fairly usable system for my company. Any help you need/want, just let me know. > I am planning to improve dblink during the next release cycle, so I'll keep all this in mind (and might take you up on the help offer too!). I was hoping we'd have functions returning tuples by now, which would improve this extension dramatically. Unfortunately, it sounds like Alex won't have time to finish that even for 7.3 :( Alex, can we get a look at your latest code? Is it any different the your last submission to PATCHES? Joe ---------------------------(end of broadcast)--------------------------- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly