From pgsql-hackers-owner+M3040@hub.org Thu Jun 8 00:31:01 2000 Received: from renoir.op.net (root@renoir.op.net [207.29.195.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA13157 for ; Thu, 8 Jun 2000 00:31:00 -0400 (EDT) Received: from hub.org (root@hub.org [216.126.84.1]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id AAA01089 for ; Thu, 8 Jun 2000 00:17:19 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e5846ib99782; Thu, 8 Jun 2000 00:06:44 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by hub.org (8.10.1/8.10.1) with ESMTP id e5846Xb99707 for ; Thu, 8 Jun 2000 00:06:33 -0400 (EDT) Received: from cadzone ([126.0.1.40] (may be forged)) by sd.tpf.co.jp (2.5 Build 2640 (Berkeley 8.8.6)/8.8.4) with SMTP id NAA01145; Thu, 08 Jun 2000 13:05:42 +0900 From: "Hiroshi Inoue" To: "Bruce Momjian" Cc: "PostgreSQL-development" Subject: RE: [HACKERS] DROP COLUMN status Date: Thu, 8 Jun 2000 13:07:44 +0900 Message-ID: <000d01bfd0ff$194d56c0$2801007e@tpf.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: <200006080309.XAA10305@candle.pha.pa.us> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR > -----Original Message----- > From: pgsql-hackers-owner@hub.org [mailto:pgsql-hackers-owner@hub.org]On > Behalf Of Bruce Momjian > > Can someone comment on where we are with DROP COLUMN? > I've already committed my trial implementation 3 months ago. They are $ifdef'd by _DROP_COLUMN_HACK__. Please enable the feature and evaluate it. You could enable the feature without initdb. Regards. Hiroshi Inoue Inoue@tpf.co.jp From Inoue@tpf.co.jp Thu Jun 8 02:03:27 2000 Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA14243 for ; Thu, 8 Jun 2000 02:03:25 -0400 (EDT) Received: from cadzone ([126.0.1.40] (may be forged)) by sd.tpf.co.jp (2.5 Build 2640 (Berkeley 8.8.6)/8.8.4) with SMTP id PAA01221; Thu, 08 Jun 2000 15:03:23 +0900 From: "Hiroshi Inoue" To: "Bruce Momjian" Cc: "PostgreSQL-development" Subject: RE: [HACKERS] DROP COLUMN status Date: Thu, 8 Jun 2000 15:05:24 +0900 Message-ID: <000f01bfd10f$893798a0$2801007e@tpf.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: <200006080457.AAA13430@candle.pha.pa.us> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal Status: OR > -----Original Message----- > From: Bruce Momjian [mailto:pgman@candle.pha.pa.us] > Sent: Thursday, June 08, 2000 1:58 PM > > [ Charset ISO-8859-1 unsupported, converting... ] > > > -----Original Message----- > > > From: pgsql-hackers-owner@hub.org > [mailto:pgsql-hackers-owner@hub.org]On > > > Behalf Of Bruce Momjian > > > > > > Can someone comment on where we are with DROP COLUMN? > > > > > > > I've already committed my trial implementation 3 months ago. > > They are $ifdef'd by _DROP_COLUMN_HACK__. > > Please enable the feature and evaluate it. > > You could enable the feature without initdb. > > OK, can you explain how it works, and add any needed documentation so we > can enable it. > First it's only a trial so I don't implement it completely. Especially I don't completely drop related objects (FK_constraint,triggers,views etc). I don't know whether we could drop them properly or not. The implementation makes the dropped column invisible by changing its attnum to -attnum - offset(currently 20) and attnam to ("*already Dropped%d",attnum). It doesn't touch the table at all. After dropping a column insert/update operation regards the column as NULL and other related stuff simply ignores the column. Regards. Hiroshi Inoue Inoue@tpf.co.jp From tgl@sss.pgh.pa.us Thu Jun 8 10:20:34 2000 Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA29148 for ; Thu, 8 Jun 2000 10:20:33 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id KAA15725; Thu, 8 Jun 2000 10:20:11 -0400 (EDT) To: "Hiroshi Inoue" cc: "Bruce Momjian" , "PostgreSQL-development" Subject: Re: [HACKERS] DROP COLUMN status In-reply-to: <000f01bfd10f$893798a0$2801007e@tpf.co.jp> References: <000f01bfd10f$893798a0$2801007e@tpf.co.jp> Comments: In-reply-to "Hiroshi Inoue" message dated "Thu, 08 Jun 2000 15:05:24 +0900" Date: Thu, 08 Jun 2000 10:20:11 -0400 Message-ID: <15722.960474011@sss.pgh.pa.us> From: Tom Lane Status: ORr "Hiroshi Inoue" writes: > The implementation makes the dropped column invisible by > changing its attnum to -attnum - offset(currently 20) and > attnam to ("*already Dropped%d",attnum). Ugh. No wonder you had to hack so many places in such an ugly fashion. Why not leave the attnum as-is, and just add a bool saying "column is dropped" to pg_attribute? As long as the parser ignores columns marked that way for field lookup and expansion of *, it seems the rest of the system wouldn't need to treat dropped columns specially in any way. regards, tom lane From pgsql-hackers-owner+M3094@hub.org Thu Jun 8 15:58:30 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id PAA25109 for ; Thu, 8 Jun 2000 15:58:28 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e58JrqT91713; Thu, 8 Jun 2000 15:53:52 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by hub.org (8.10.1/8.10.1) with ESMTP id e58JqpT91436 for ; Thu, 8 Jun 2000 15:52:51 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id PAA19690; Thu, 8 Jun 2000 15:52:43 -0400 (EDT) To: Bruce Momjian cc: Hiroshi Inoue , PostgreSQL-development Subject: Re: [HACKERS] DROP COLUMN status In-reply-to: <200006081541.LAA01566@candle.pha.pa.us> References: <200006081541.LAA01566@candle.pha.pa.us> Comments: In-reply-to Bruce Momjian message dated "Thu, 08 Jun 2000 11:41:43 -0400" Date: Thu, 08 Jun 2000 15:52:43 -0400 Message-ID: <19687.960493963@sss.pgh.pa.us> From: Tom Lane X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR >>>> The implementation makes the dropped column invisible by >>>> changing its attnum to -attnum - offset(currently 20) and >>>> attnam to ("*already Dropped%d",attnum). >> >> Ugh. No wonder you had to hack so many places in such an ugly fashion. >> Why not leave the attnum as-is, and just add a bool saying "column is >> dropped" to pg_attribute? As long as the parser ignores columns marked >> that way for field lookup and expansion of *, it seems the rest of the >> system wouldn't need to treat dropped columns specially in any way. > If we leave it as positive, don't we have to change user applications > that query pg_attribute so they also know to skip it? Good point, but I think user applications that query pg_attribute are likely to have trouble anyway: if they're expecting a consecutive series of attnums then they're going to lose no matter what. regards, tom lane From hannu@tm.ee Sat Jun 10 01:02:57 2000 Received: from me.tm.ee (ppp15.tele2.ee [212.107.33.15]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA10377 for ; Sat, 10 Jun 2000 01:02:55 -0400 (EDT) Received: from tm.ee (IDENT:hannu@localhost.localdomain [127.0.0.1]) by me.tm.ee (8.9.3/8.9.3) with ESMTP id GAA00940; Sat, 10 Jun 2000 06:59:33 +0300 Sender: hannu@me.tm.ee Message-ID: <3941BD25.96760D2E@tm.ee> Date: Sat, 10 Jun 2000 06:59:33 +0300 From: Hannu Krosing X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bruce Momjian CC: Tom Lane , Peter Eisentraut , PostgreSQL Development Subject: Re: [HACKERS] ALTER TABLE DROP COLUMN References: <200006091249.IAA18730@candle.pha.pa.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: OR Bruce Momjian wrote: > > Seems we have 4 DROP COLUMN ideas: > > Method Advantage > ----------------------------------------------------------------- > 1 invisible column marked by negative attnum fast > 2 invisible column marked by is_dropped column fast > 3 make copy of table without column col removed > 4 make new tuples in existing table without column col removed IIRC there was a fifth idea, a variation of 2 that would work better with inheritance - 5 all columns have is_real_column attribute that is true for all coluns present in that relation, so situations like create table tab_a(a_i int); create table tab_b(b_i int) inherits(tab_a); alter table tab_a add column c_i int; can be made to work. It would also require clients to ignore all missing columns that backend can pass to them as nulls (which is usually quite cheap in bandwith usage) in case of "SELECT **" queries. We could even rename attno to attid to make folks aware that it is not be assumed to be continuous. > Folks, we had better choose one and get started. > > Number 1 Hiroshi has ifdef'ed out in the code. Items 1 and 2 have > problems with backend code and 3rd party code not seeing the dropped > columns, or having gaps in the attno numbering. If we want to make ADD COLUMN to work with inheritance wihout having to rewrite every single tuple in both parent and inherited tables, we will have to accept the fact that there are caps in in attno numbering. > Number 3 has problems > with making it an atomic operation, and number 4 is described below. Nr 4 has still problems with attno numbering _changing_ during drop which could either be better or worse for client software than having gaps - in both cases client must be prepared to deal with runtime changes in attribute definition. -------------- Hannu From Inoue@tpf.co.jp Sat Jun 10 01:01:01 2000 Received: from renoir.op.net (root@renoir.op.net [207.29.195.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA10355 for ; Sat, 10 Jun 2000 01:01:00 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id AAA25467 for ; Sat, 10 Jun 2000 00:41:32 -0400 (EDT) Received: from mcadnote1 (ppm110.noc.fukui.nsk.ne.jp [210.161.188.29] (may be forged)) by sd.tpf.co.jp (2.5 Build 2640 (Berkeley 8.8.6)/8.8.4) with SMTP id NAA03125; Sat, 10 Jun 2000 13:40:40 +0900 From: "Hiroshi Inoue" To: "Bruce Momjian" , "Tom Lane" Cc: "Peter Eisentraut" , "PostgreSQL Development" Subject: RE: [HACKERS] ALTER TABLE DROP COLUMN Date: Sat, 10 Jun 2000 13:43:26 +0900 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <200006091249.IAA18730@candle.pha.pa.us> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Status: ORr > -----Original Message----- > From: pgsql-hackers-owner@hub.org > [mailto:pgsql-hackers-owner@hub.org]On Behalf Of Bruce Momjian > > Seems we have 4 DROP COLUMN ideas: > > Method Advantage > ----------------------------------------------------------------- > 1 invisible column marked by negative attnum fast > 2 invisible column marked by is_dropped column fast > 3 make copy of table without column col removed > 4 make new tuples in existing table without column col removed > > Folks, we had better choose one and get started. > > Number 1 Hiroshi has ifdef'ed out in the code. Items 1 and 2 have > problems with backend code and 3rd party code not seeing the dropped > columns, Hmm,doesn't *not seeing* mean the column is dropped ? > or having gaps in the attno numbering. Number 3 has problems > with making it an atomic operation, and number 4 is described below. > Don't forget another important point. Currently even DROP TABLE doesn't remove related objects completely. And I don't think I could remove objects related to the dropping column completely using 1)2) in ALTER TABLE DROP COLUMN implementation. Using 3)4) we should not only remove objects as 1)2) but also change attnum-s in all objects related to the relation. Otherwise PostgreSQL would do the wrong thing silently. Regards. Hiroshi Inoue Inoue@tpf.co.jp From dhogaza@pacifier.com Sat Jun 10 01:01:06 2000 Received: from smtp.pacifier.com (comet.pacifier.com [199.2.117.155]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA10370 for ; Sat, 10 Jun 2000 01:01:04 -0400 (EDT) Received: from desktop (dsl-dhogaza.pacifier.net [207.202.226.68]) by smtp.pacifier.com (8.9.3/8.9.3pop) with SMTP id WAA08521; Fri, 9 Jun 2000 22:01:00 -0700 (PDT) Message-Id: <3.0.1.32.20000609215758.0116d850@mail.pacifier.com> X-Sender: dhogaza@mail.pacifier.com X-Mailer: Windows Eudora Pro Version 3.0.1 (32) Date: Fri, 09 Jun 2000 21:57:58 -0700 To: "Hiroshi Inoue" , "Bruce Momjian" , "Tom Lane" From: Don Baccus Subject: RE: [HACKERS] ALTER TABLE DROP COLUMN Cc: "Peter Eisentraut" , "PostgreSQL Development" In-Reply-To: References: <200006091249.IAA18730@candle.pha.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Status: OR At 01:43 PM 6/10/00 +0900, Hiroshi Inoue wrote: >> -----Original Message----- >> From: pgsql-hackers-owner@hub.org >> [mailto:pgsql-hackers-owner@hub.org]On Behalf Of Bruce Momjian >> >> Seems we have 4 DROP COLUMN ideas: >> >> Method Advantage >> ----------------------------------------------------------------- >> 1 invisible column marked by negative attnum fast >> 2 invisible column marked by is_dropped column fast >> 3 make copy of table without column col removed >> 4 make new tuples in existing table without column col removed >> >> Folks, we had better choose one and get started. Oracle gives you the choice between the "cheating" fast method(s) and the "real" slow (really slow?) real method. So there's at least real world experience by virtue of example by the world's most successful database supplier that user control over "hide the column" and "really delete the column" is valuable. It really makes a lot of sense to give such a choice. If one does so by "hiding", at a later date one would think the choice of "really deleting" would be a possibility. I don't know if Oracle does this... If not, they might not care. In today's world, there are bazillions of dollars for Oracle to scoop up from users who could just as easily be PG users - all those "we'll fail if don't IPO 'cause we'll never have any customers" database-backed websites :) - Don Baccus, Portland OR Nature photos, on-line guides, Pacific Northwest Rare Bird Alert Service and other goodies at http://donb.photo.net. From tgl@sss.pgh.pa.us Sat Jun 10 01:31:04 2000 Received: from renoir.op.net (root@renoir.op.net [207.29.195.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA10922 for ; Sat, 10 Jun 2000 01:31:03 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id BAA27265 for ; Sat, 10 Jun 2000 01:16:07 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id BAA06206; Sat, 10 Jun 2000 01:14:37 -0400 (EDT) To: Don Baccus cc: "Hiroshi Inoue" , "Bruce Momjian" , "Peter Eisentraut" , "PostgreSQL Development" Subject: Re: [HACKERS] ALTER TABLE DROP COLUMN In-reply-to: <3.0.1.32.20000609215758.0116d850@mail.pacifier.com> References: <200006091249.IAA18730@candle.pha.pa.us> <3.0.1.32.20000609215758.0116d850@mail.pacifier.com> Comments: In-reply-to Don Baccus message dated "Fri, 09 Jun 2000 21:57:58 -0700" Date: Sat, 10 Jun 2000 01:14:37 -0400 Message-ID: <6203.960614077@sss.pgh.pa.us> From: Tom Lane Status: OR Don Baccus writes: > Oracle gives you the choice between the "cheating" fast method(s) and > the "real" slow (really slow?) real method. > So there's at least real world experience by virtue of example by > the world's most successful database supplier that user control > over "hide the column" and "really delete the column" is valuable. Sure, but you don't need any help from the database to do "really delete the column". SELECT INTO... is enough, and it's not even any slower than the implementations under discussion. So I'm satisfied if we offer the "hide the column" approach. Has anyone thought about what happens to table constraints that use the doomed column? Triggers, RI rules, yadda yadda? Has anyone thought about undoing a DELETE COLUMN? The data is still there, at least in tuples that have not been updated, so it's not totally unreasonable. regards, tom lane From dhogaza@pacifier.com Sat Jun 10 09:30:59 2000 Received: from renoir.op.net (root@renoir.op.net [207.29.195.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id JAA25987 for ; Sat, 10 Jun 2000 09:30:58 -0400 (EDT) Received: from smtp.pacifier.com (comet.pacifier.com [199.2.117.155]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id JAA18716 for ; Sat, 10 Jun 2000 09:15:08 -0400 (EDT) Received: from desktop (dsl-dhogaza.pacifier.net [207.202.226.68]) by smtp.pacifier.com (8.9.3/8.9.3pop) with SMTP id GAA15799; Sat, 10 Jun 2000 06:14:28 -0700 (PDT) Message-Id: <3.0.1.32.20000610054306.0115f020@mail.pacifier.com> X-Sender: dhogaza@mail.pacifier.com X-Mailer: Windows Eudora Pro Version 3.0.1 (32) Date: Sat, 10 Jun 2000 05:43:06 -0700 To: Tom Lane From: Don Baccus Subject: Re: [HACKERS] ALTER TABLE DROP COLUMN Cc: "Hiroshi Inoue" , "Bruce Momjian" , "Peter Eisentraut" , "PostgreSQL Development" In-Reply-To: <6203.960614077@sss.pgh.pa.us> References: <3.0.1.32.20000609215758.0116d850@mail.pacifier.com> <200006091249.IAA18730@candle.pha.pa.us> <3.0.1.32.20000609215758.0116d850@mail.pacifier.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Status: OR At 01:14 AM 6/10/00 -0400, Tom Lane wrote: >Don Baccus writes: >> Oracle gives you the choice between the "cheating" fast method(s) and >> the "real" slow (really slow?) real method. > >> So there's at least real world experience by virtue of example by >> the world's most successful database supplier that user control >> over "hide the column" and "really delete the column" is valuable. > >Sure, but you don't need any help from the database to do "really delete >the column". SELECT INTO... is enough, and it's not even any slower >than the implementations under discussion. > >So I'm satisfied if we offer the "hide the column" approach. I wouldn't put a "real" drop column at the top of my list of priorities, but there is something to be said for user convenience. - Don Baccus, Portland OR Nature photos, on-line guides, Pacific Northwest Rare Bird Alert Service and other goodies at http://donb.photo.net. From tgl@sss.pgh.pa.us Sun Jun 11 12:31:03 2000 Received: from renoir.op.net (root@renoir.op.net [207.29.195.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA05771 for ; Sun, 11 Jun 2000 12:31:01 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id MAA19315 for ; Sun, 11 Jun 2000 12:24:06 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id MAA09503; Sun, 11 Jun 2000 12:22:42 -0400 (EDT) To: "Hiroshi Inoue" cc: "Bruce Momjian" , "Peter Eisentraut" , "PostgreSQL Development" Subject: Re: [HACKERS] ALTER TABLE DROP COLUMN In-reply-to: References: Comments: In-reply-to "Hiroshi Inoue" message dated "Sat, 10 Jun 2000 13:43:26 +0900" Date: Sun, 11 Jun 2000 12:22:42 -0400 Message-ID: <9500.960740562@sss.pgh.pa.us> From: Tom Lane Status: ORr >> Seems we have 4 DROP COLUMN ideas: >> Method Advantage >> ----------------------------------------------------------------- >> 1 invisible column marked by negative attnum fast >> 2 invisible column marked by is_dropped column fast >> 3 make copy of table without column col removed >> 4 make new tuples in existing table without column col removed Bruce and I talked about this by phone yesterday, and we realized that none of these are very satisfactory. #1 and #2 both have the flaw that applications that examine pg_attribute will probably break: they will see a sequence of attnum values with gaps in it. And what should the rel's relnatts field be set to? #3 and #4 are better on that point, but they leave us with the problem of renumbering references to columns after the dropped one in constraints, rules, PL functions, etc. Furthermore, there is a closely related problem that none of these approaches give us much help on: recursive ALTER TABLE ADD COLUMN. Right now, ADD puts the new column at the end of each table it's added to, which often means that it gets a different column number in child tables than in parent tables. That leads to havoc for pg_dump. I think the only clean solution is to create a clear distinction between physical and logical column numbers. Each pg_attribute tuple would need two attnum fields, and pg_class would need two relnatts fields as well. A column once created would never change its physical column number, but its logical column number might change as a consequence of adding or dropping columns before it. ADD COLUMN would ensure that a column added to child tables receives the same logical column number as it has in the parent table, thus solving the dump/reload problem. DROP COLUMN would assign an invalid logical column number to dropped columns. They could be numbered zero except that we'd probably still want a unique index on attrelid+attnum, and the index would complain. I'd suggest using Hiroshi's idea: give a dropped column a logical attnum equal to -(physical_attnum + offset). With this approach, internal operations on tuples would all use physical column numbers, but operations that interface to the outside world would present a view of only the valid logical columns. For example, the parser would only allow logical columns to be referenced by name; "SELECT *" would expand to valid logical columns in logical- column-number order; COPY would send or receive valid logical columns in logical-column-number order; etc. Stored rules and so forth probably should store physical column numbers so that they need not be modified during column add/drop. This would require looking at all the places in the backend to determine whether they should be working with logical or physical column numbers, but the design is such that most all places would want to be using physical numbers, so I don't think it'd be too painful. Although I'd prefer to give the replacement columns two new names (eg, "attlnum" and "attpnum") to ensure we find all uses, this would surely break applications that examine pg_attribute. For compatibility we'd have to recycle "attnum" and "relnatts" to indicate logical column number and logical column count, while adding new fields (say "attpnum" and "relnpatts") for the physical number and count. Comments? regards, tom lane From pgsql-hackers-owner+M3184@hub.org Mon Jun 12 09:29:17 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id JAA16538 for ; Mon, 12 Jun 2000 09:29:15 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e5C9RxT92685; Mon, 12 Jun 2000 05:27:59 -0400 (EDT) Received: from clio.trends.ca (root@clio.trends.ca [209.47.148.2]) by hub.org (8.10.1/8.10.1) with ESMTP id e5C8YWT89945 for ; Mon, 12 Jun 2000 04:34:32 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by clio.trends.ca (8.9.3+Sun/8.9.3) with ESMTP id VAA17711 for ; Sun, 11 Jun 2000 21:40:28 -0400 (EDT) Received: from cadzone ([126.0.1.40] (may be forged)) by sd.tpf.co.jp (2.5 Build 2640 (Berkeley 8.8.6)/8.8.4) with SMTP id KAA03734; Mon, 12 Jun 2000 10:38:42 +0900 From: "Hiroshi Inoue" To: "Tom Lane" Cc: "Bruce Momjian" , "Peter Eisentraut" , "PostgreSQL Development" Subject: RE: [HACKERS] ALTER TABLE DROP COLUMN Date: Mon, 12 Jun 2000 10:40:47 +0900 Message-ID: <000b01bfd40f$3b3091e0$2801007e@tpf.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: <9500.960740562@sss.pgh.pa.us> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR > -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > > >> Seems we have 4 DROP COLUMN ideas: > >> Method Advantage > >> ----------------------------------------------------------------- > >> 1 invisible column marked by negative attnum fast > >> 2 invisible column marked by is_dropped column fast > >> 3 make copy of table without column col removed > >> 4 make new tuples in existing table without column col removed > Hmm,I've received no pg-ML mails for more than 1 day. What's happened with pgsql ML ? > Bruce and I talked about this by phone yesterday, and we realized that > none of these are very satisfactory. #1 and #2 both have the flaw that > applications that examine pg_attribute will probably break: they will > see a sequence of attnum values with gaps in it. And what should the > rel's relnatts field be set to? #3 and #4 are better on that point, > but they leave us with the problem of renumbering references to columns > after the dropped one in constraints, rules, PL functions, etc. > > Furthermore, there is a closely related problem that none of these > approaches give us much help on: recursive ALTER TABLE ADD COLUMN. > Right now, ADD puts the new column at the end of each table it's added > to, which often means that it gets a different column number in child > tables than in parent tables. That leads to havoc for pg_dump. > Inheritance is one of the reason why I didn't take #2. I don't understand marking is_dropped is needed or not when pg_attribute is overhauled for inheritance. I myself have never wanted to use current inheritance functionality mainly because of this big flaw. Judging from the recent discussion about oo(though I don't understand details),the change seems to be needed in order to make inheritance functionality really useful. > I think the only clean solution is to create a clear distinction between > physical and logical column numbers. Each pg_attribute tuple would need > two attnum fields, and pg_class would need two relnatts fields as well. > A column once created would never change its physical column number, but I don't understand inheritance well. In the near future wouldn't the implementation require e.g. attid which is common to all children of a parent and is never changed ? If so,we would need the third attid field which is irrevalent to physical/logical position. If not, physical column number would be sufficient . > its logical column number might change as a consequence of adding or > dropping columns before it. ADD COLUMN would ensure that a column added > to child tables receives the same logical column number as it has in the > parent table, thus solving the dump/reload problem. DROP COLUMN would > assign an invalid logical column number to dropped columns. They could > be numbered zero except that we'd probably still want a unique index on > attrelid+attnum, and the index would complain. I'd suggest using > Hiroshi's idea: give a dropped column a logical attnum equal to > -(physical_attnum + offset). > > With this approach, internal operations on tuples would all use > physical column numbers, but operations that interface to the outside > world would present a view of only the valid logical columns. For > example, the parser would only allow logical columns to be referenced > by name; "SELECT *" would expand to valid logical columns in logical- > column-number order; COPY would send or receive valid logical columns > in logical-column-number order; etc. > > Stored rules and so forth probably should store physical column numbers > so that they need not be modified during column add/drop. > > This would require looking at all the places in the backend to determine > whether they should be working with logical or physical column numbers, > but the design is such that most all places would want to be using > physical numbers, so I don't think it'd be too painful. > > Although I'd prefer to give the replacement columns two new names > (eg, "attlnum" and "attpnum") to ensure we find all uses, this would > surely break applications that examine pg_attribute. For compatibility > we'd have to recycle "attnum" and "relnatts" to indicate logical column > number and logical column count, while adding new fields (say "attpnum" > and "relnpatts") for the physical number and count. > I agree with you that we would add attpnum and change the meaing of attnum as logical column number for backward compatibility. Regards. Hiroshi Inoue Inoue@tpf.co.jp From pgsql-hackers-owner+M3050@postgresql.org Thu Jan 11 21:49:43 2001 Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA20277 for ; Thu, 11 Jan 2001 21:49:42 -0500 (EST) Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f0C2lhp74989; Thu, 11 Jan 2001 21:47:43 -0500 (EST) (envelope-from pgsql-hackers-owner+M3050@postgresql.org) Received: from dynworks.com (adsl-63-206-168-198.dsl.sktn01.pacbell.net [63.206.168.198]) by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f0C2lNp74855 for ; Thu, 11 Jan 2001 21:47:23 -0500 (EST) (envelope-from jdavis@dynworks.com) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by dynworks.com (Postfix) with ESMTP id CC44F31FAB for ; Thu, 11 Jan 2001 18:48:36 -0800 (PST) Date: Thu, 11 Jan 2001 18:48:36 PST From: Jeff Davis To: pgsql-hackers@postgresql.org Subject: [HACKERS] alter table drop column Reply-To: jdavis@dynworks.com X-Mailer: Spruce 0.6.5 for X11 w/smtpio 0.7.9 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit Message-Id: <20010112024836.CC44F31FAB@dynworks.com> Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR I read the transcript of the alter table drop column discussion (old discussion) at http://www.postgresql.org/docs/pgsql/doc/TODO.detail/drop, and I have something to add: People mentioned such ideas as a hidden column and a really deleted column, and it occurred to me that perhaps "vacuum" would be a good option to use. When a delete was issued, the column would be hidden (by a negative/invalid logical column number, it appears was the consensus). Upon issuing a vacuum, it could perform a complete deletion. This method would allow users to know that the process may take a while (I think the agreed method for a complete delete was to "select into..." the right columns and leave out the deleted ones, then delete the old table). Furthermore, I liked the idea of some kind of "undelete", as long as it was just hidden. This could apply to anything that is cleaned out with a vacuum (before it is cleaned out), although I am not sure how feasible this is, and it isn't particularly important to me. Regards, Jeff -- Jeff Davis Dynamic Works jdavis@dynworks.com http://dynworks.com From owner-pgsql-hackers@hub.org Sat Feb 26 01:07:45 2000 Received: from hub.org (hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA17776 for ; Sat, 26 Feb 2000 01:07:43 -0500 (EST) Received: from localhost (majordom@localhost) by hub.org (8.9.3/8.9.3) with SMTP id BAA06232; Sat, 26 Feb 2000 01:03:53 -0500 (EST) (envelope-from owner-pgsql-hackers) Received: by hub.org (bulk_mailer v1.5); Sat, 26 Feb 2000 01:03:26 -0500 Received: (from majordom@localhost) by hub.org (8.9.3/8.9.3) id BAA05808 for pgsql-hackers-outgoing; Sat, 26 Feb 2000 01:02:28 -0500 (EST) (envelope-from owner-pgsql-hackers@postgreSQL.org) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by hub.org (8.9.3/8.9.3) with ESMTP id BAA05426 for ; Sat, 26 Feb 2000 01:01:46 -0500 (EST) (envelope-from tgl@sss.pgh.pa.us) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id BAA14228; Sat, 26 Feb 2000 01:01:34 -0500 (EST) To: Bruce Momjian cc: Peter Eisentraut , PostgreSQL Development Subject: Re: [HACKERS] ALTER TABLE DROP COLUMN In-reply-to: <200002260412.XAA14752@candle.pha.pa.us> References: <200002260412.XAA14752@candle.pha.pa.us> Comments: In-reply-to Bruce Momjian message dated "Fri, 25 Feb 2000 23:12:26 -0500" Date: Sat, 26 Feb 2000 01:01:33 -0500 Message-ID: <14225.951544893@sss.pgh.pa.us> From: Tom Lane Sender: owner-pgsql-hackers@postgreSQL.org Status: ORr Bruce Momjian writes: > You can exclusively lock the table, then do a heap_getnext() scan over > the entire table, remove the dropped column, do a heap_insert(), then a > heap_delete() on the current tuple, making sure to skip over the tuples > inserted by the current transaction. When completed, remove the column > from pg_attribute, mark the transaction as committed (if desired), and > run vacuum over the table to remove the deleted rows. Hmm, that would work --- the new tuples commit at the same instant that the schema updates commit, so it should be correct. You have the 2x disk usage problem, but there's no way around that without losing rollback ability. A potentially tricky bit will be persuading the tuple-reading and tuple- writing subroutines to pay attention to different versions of the tuple structure for the same table. I haven't looked to see if this will be difficult or not. If you can pass the TupleDesc explicitly then it shouldn't be a problem. I'd suggest that the cleanup vacuum *not* be an automatic part of the operation; just recommend that people do it ASAP after dropping a column. Consider needing to drop several columns... regards, tom lane ************ From pgsql-hackers-owner+M18768=candle.pha.pa.us=pgman@postgresql.org Wed Feb 13 03:52:00 2002 Return-path: Received: from server1.pgsql.org (www.postgresql.org [64.49.215.9]) by candle.pha.pa.us (8.11.6/8.10.1) with SMTP id g1D8pxP21056 for ; Wed, 13 Feb 2002 03:52:00 -0500 (EST) Received: (qmail 97959 invoked by alias); 13 Feb 2002 08:51:46 -0000 Received: from unknown (HELO postgresql.org) (64.49.215.8) by www.postgresql.org with SMTP; 13 Feb 2002 08:51:46 -0000 Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by postgresql.org (8.11.3/8.11.4) with SMTP id g1D8mRE97432 for ; Wed, 13 Feb 2002 03:48:28 -0500 (EST) (envelope-from Inoue@tpf.co.jp) Received: (qmail 26891 invoked from network); 13 Feb 2002 08:48:27 -0000 Received: from unknown (HELO viscomail.tpf.co.jp) (100.0.0.108) by sd2.tpf-fw-c.co.jp with SMTP; 13 Feb 2002 08:48:27 -0000 Received: from tpf.co.jp (3dgateway1 [126.0.1.60]) by viscomail.tpf.co.jp (8.8.8+Sun/8.8.8) with ESMTP id RAA01019; Wed, 13 Feb 2002 17:48:20 +0900 (JST) Message-ID: <3C6A2861.6E71A124@tpf.co.jp> Date: Wed, 13 Feb 2002 17:48:33 +0900 From: Hiroshi Inoue X-Mailer: Mozilla 4.73 [ja] (Windows NT 5.0; U) X-Accept-Language: ja MIME-Version: 1.0 To: Christopher Kings-Lynne cc: Tom Lane , Kovacs Zoltan , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] alter table drop column status References: Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Christopher Kings-Lynne wrote: > > > No there was an unapplied hack which uses logical/physical > > attribute numbers. I have synchronized it with cvs for a > > year or so but stop it now. Though it had some flaws It > > solved the following TODOs. > > > > * Add ALTER TABLE DROP COLUMN feature > > * ALTER TABLE ADD COLUMN to inherited table put column in wrong place > > * Prevent column dropping if column is used by foreign key > > This seems fantastic - why can't this be committed? Surely if it's > committed then the flaws will fairly quickly be ironed out? Even if it has > flaws, then if we say 'this function is not yet stable' at least people can > start testing it and reporting the problems? > > > I gave up to apply the hack mainly because it may introduce > > the maintenance headache. > > Is it a maintenance headache just for you to keep it up to date, or how > would it be a maintenance headache if it were committed? Probably(oops I don't remember well now sorry) the main reason why I didn't insist to apply the patch was that it wasn't so clean as I had expected. My trial implementation uses logical(for clients) and physical (for backend internal) attribute numbers but there were many places where I wasn't able to judge which to use immediately. I'm pretty suspicious if a developer could be careful about the choise when he is implementing an irrevant feature. (Un)fortunately the numbers have the same values mostly and he could hardly notice the mistake even if he chose the wrong attribute numbers. I'm not sure if I myself chose the right attribute numbers everywhere in my implementation. In addtion (probably) there were some pretty essential flaws. I intended to manage the backend internal object references without the logical attribute numbers but I found it difficult in some cases (probably the handling of virtual(not existent in any real table) tuples). Sorry it was more than 1 year ago when I implemented it and I can't remember well what I'd thougth then. Though I'd kept my local branch up to date for about a year, it's about half a year since I touched the stuff last. regards, Hiroshi Inoue ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From chriskl@familyhealth.com.au Thu Apr 11 12:00:22 2002 Return-path: Received: from houston.familyhealth.com.au (root@i231-006.nv.iinet.net.au [203.59.231.6]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BG0KS02910 for ; Thu, 11 Apr 2002 12:00:20 -0400 (EDT) Received: from localhost (chriskl@localhost) by houston.familyhealth.com.au (8.11.6/8.11.6) with ESMTP id g3BG0GJ70765; Fri, 12 Apr 2002 00:00:16 +0800 (WST) (envelope-from chriskl@familyhealth.com.au) Date: Fri, 12 Apr 2002 00:00:16 +0800 (WST) From: Christopher Kings-Lynne To: Bruce Momjian cc: Hiroshi Inoue , Tom Lane , Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <200204110419.g3B4J8v29682@candle.pha.pa.us> Message-ID: <20020411233659.O69846-100000@houston.familyhealth.com.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: OR > Actually, what we need to do to reclaim space is to enable table > recreation without the column, now that we have relfilenode for file > renaming. It isn't hard to do, but no one has focused on it. I want to > focus on it, but have not had the time, obviously, and would be very > excited to assist someone else. > > Hiroshi's fine idea of marking certain columns as unused would not have > reclaimed the missing space, just as my idea of physical/logical column > distinction would not reclaim the space either. Again, my > physical/logical idea is more for fixing other problems and > optimization, not DROP COLUMN. Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is kinda useless - you may as well just use a view!!! So how would this occur?: 1. Lock target table for writing (allow reads) 2. Begin a table scan on target table, writing a new file with a particular filenode 3. Delete the attribute row from pg_attribute 4. Point the table in the catalog to the new filenode 5. Release locks 6. Commit transaction 7. Delete orhpan filenode i. Upon postmaster startup, remove any orphaned filenodes The real problem here is the fact that there are now missing attnos in pg_attribute. Either that's handled or we renumber the attnos - which is also quite hard? This, of course, suffers from the double size data problem - but I believe that it does not matter - we just need to document it. Interestingly enough, Oracle support ALTER TABLE foo SET UNUSED col; Which invalidates the attribute entry, and: ALTER TABLE foo DROP col CHECKPOINT 1000; Which actually reclaims the space. The optional CHECKPOINT [n] clause tells Oracle to do a checkpoint every [n] rows. "Checkpointing cuts down the amount of undo logs accumulated during the drop column operation to avoid running out of rollback segment space. However, if this statement is interrupted after a checkpoint has been applied, the table remains in an unusable state. While the table is unusable, the only operations allowed on it are DROP TABLE, TRUNCATE TABLE, and ALTER TABLE DROP COLUMNS CONTINUE (described below). " Chris From pgsql-hackers-owner+M21180@postgresql.org Thu Apr 11 12:02:54 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BG2sS03611 for ; Thu, 11 Apr 2002 12:02:54 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 6446B478F0A; Thu, 11 Apr 2002 12:01:19 -0400 (EDT) Received: from houston.familyhealth.com.au (i231-006.nv.iinet.net.au [203.59.231.6]) by postgresql.org (Postfix) with ESMTP id B6271478E4C for ; Thu, 11 Apr 2002 12:00:24 -0400 (EDT) Received: from localhost (chriskl@localhost) by houston.familyhealth.com.au (8.11.6/8.11.6) with ESMTP id g3BG0GJ70765; Fri, 12 Apr 2002 00:00:16 +0800 (WST) (envelope-from chriskl@familyhealth.com.au) Date: Fri, 12 Apr 2002 00:00:16 +0800 (WST) From: Christopher Kings-Lynne To: Bruce Momjian cc: Hiroshi Inoue , Tom Lane , Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <200204110419.g3B4J8v29682@candle.pha.pa.us> Message-ID: <20020411233659.O69846-100000@houston.familyhealth.com.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: ORr > Actually, what we need to do to reclaim space is to enable table > recreation without the column, now that we have relfilenode for file > renaming. It isn't hard to do, but no one has focused on it. I want to > focus on it, but have not had the time, obviously, and would be very > excited to assist someone else. > > Hiroshi's fine idea of marking certain columns as unused would not have > reclaimed the missing space, just as my idea of physical/logical column > distinction would not reclaim the space either. Again, my > physical/logical idea is more for fixing other problems and > optimization, not DROP COLUMN. Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is kinda useless - you may as well just use a view!!! So how would this occur?: 1. Lock target table for writing (allow reads) 2. Begin a table scan on target table, writing a new file with a particular filenode 3. Delete the attribute row from pg_attribute 4. Point the table in the catalog to the new filenode 5. Release locks 6. Commit transaction 7. Delete orhpan filenode i. Upon postmaster startup, remove any orphaned filenodes The real problem here is the fact that there are now missing attnos in pg_attribute. Either that's handled or we renumber the attnos - which is also quite hard? This, of course, suffers from the double size data problem - but I believe that it does not matter - we just need to document it. Interestingly enough, Oracle support ALTER TABLE foo SET UNUSED col; Which invalidates the attribute entry, and: ALTER TABLE foo DROP col CHECKPOINT 1000; Which actually reclaims the space. The optional CHECKPOINT [n] clause tells Oracle to do a checkpoint every [n] rows. "Checkpointing cuts down the amount of undo logs accumulated during the drop column operation to avoid running out of rollback segment space. However, if this statement is interrupted after a checkpoint has been applied, the table remains in an unusable state. While the table is unusable, the only operations allowed on it are DROP TABLE, TRUNCATE TABLE, and ALTER TABLE DROP COLUMNS CONTINUE (described below). " Chris ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From tgl@sss.pgh.pa.us Thu Apr 11 12:22:44 2002 Return-path: Received: from sss.pgh.pa.us (root@[192.204.191.242]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BGMhS05541 for ; Thu, 11 Apr 2002 12:22:43 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3BGMaF01827; Thu, 11 Apr 2002 12:22:36 -0400 (EDT) To: Christopher Kings-Lynne cc: Bruce Momjian , Hiroshi Inoue , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <20020411233659.O69846-100000@houston.familyhealth.com.au> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> Comments: In-reply-to Christopher Kings-Lynne message dated "Fri, 12 Apr 2002 00:00:16 +0800" Date: Thu, 11 Apr 2002 12:22:35 -0400 Message-ID: <1824.1018542155@sss.pgh.pa.us> From: Tom Lane Status: ORr Christopher Kings-Lynne writes: > The real problem here is the fact that there are now missing attnos in > pg_attribute. Either that's handled or we renumber the attnos - which is > also quite hard? Updating pg_attribute per se is not so hard --- just store new copies of all the rows for the table. However, propagating the changes into other places could be quite painful (I'm thinking of column numbers in stored constraints, rules, etc). It seems to me that reducing the column to NULLs already gets you the majority of the space savings. I don't think there is a case to be made that getting back that last bit is worth the pain involved, either in implementation effort or direct runtime costs (do you really want a DROP COLUMN to force an immediate rewrite of the whole table?) regards, tom lane From pgsql-hackers-owner+M21186@postgresql.org Thu Apr 11 13:03:13 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BH3DS08942 for ; Thu, 11 Apr 2002 13:03:13 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 517ED479415; Thu, 11 Apr 2002 12:29:32 -0400 (EDT) Received: from sss.pgh.pa.us (unknown [192.204.191.242]) by postgresql.org (Postfix) with ESMTP id B87BC479327 for ; Thu, 11 Apr 2002 12:22:51 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3BGMaF01827; Thu, 11 Apr 2002 12:22:36 -0400 (EDT) To: Christopher Kings-Lynne cc: Bruce Momjian , Hiroshi Inoue , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <20020411233659.O69846-100000@houston.familyhealth.com.au> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> Comments: In-reply-to Christopher Kings-Lynne message dated "Fri, 12 Apr 2002 00:00:16 +0800" Date: Thu, 11 Apr 2002 12:22:35 -0400 Message-ID: <1824.1018542155@sss.pgh.pa.us> From: Tom Lane Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Christopher Kings-Lynne writes: > The real problem here is the fact that there are now missing attnos in > pg_attribute. Either that's handled or we renumber the attnos - which is > also quite hard? Updating pg_attribute per se is not so hard --- just store new copies of all the rows for the table. However, propagating the changes into other places could be quite painful (I'm thinking of column numbers in stored constraints, rules, etc). It seems to me that reducing the column to NULLs already gets you the majority of the space savings. I don't think there is a case to be made that getting back that last bit is worth the pain involved, either in implementation effort or direct runtime costs (do you really want a DROP COLUMN to force an immediate rewrite of the whole table?) regards, tom lane ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From pgsql-hackers-owner+M21187@postgresql.org Thu Apr 11 13:25:05 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BHP4S10960 for ; Thu, 11 Apr 2002 13:25:05 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 2BC27479442; Thu, 11 Apr 2002 12:30:28 -0400 (EDT) Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) by postgresql.org (Postfix) with ESMTP id 265E5479340 for ; Thu, 11 Apr 2002 12:23:30 -0400 (EDT) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id g3BGNS405576; Thu, 11 Apr 2002 12:23:28 -0400 (EDT) From: Bruce Momjian Message-ID: <200204111623.g3BGNS405576@candle.pha.pa.us> Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <20020411233659.O69846-100000@houston.familyhealth.com.au> To: Christopher Kings-Lynne Date: Thu, 11 Apr 2002 12:23:28 -0400 (EDT) cc: Hiroshi Inoue , Tom Lane , pgsql-hackers@postgresql.org X-Mailer: ELM [version 2.4ME+ PL97 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Christopher Kings-Lynne wrote: > > Actually, what we need to do to reclaim space is to enable table > > recreation without the column, now that we have relfilenode for file > > renaming. It isn't hard to do, but no one has focused on it. I want to > > focus on it, but have not had the time, obviously, and would be very > > excited to assist someone else. > > > > Hiroshi's fine idea of marking certain columns as unused would not have > > reclaimed the missing space, just as my idea of physical/logical column > > distinction would not reclaim the space either. Again, my > > physical/logical idea is more for fixing other problems and > > optimization, not DROP COLUMN. > > Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is > kinda useless - you may as well just use a view!!! Yep, kind of a problem. It is a tradeoff between double diskspace/speed and removing column from disk. I guess that's why Oracle has both. > > So how would this occur?: > > 1. Lock target table for writing (allow reads) > 2. Begin a table scan on target table, writing > a new file with a particular filenode > 3. Delete the attribute row from pg_attribute > 4. Point the table in the catalog to the new filenode > 5. Release locks > 6. Commit transaction > 7. Delete orhpan filenode Yep, something like that. CLUSTER is a good start. DROP COLUMN just deals with the attno too. You would have to renumber them to fill the gap. > i. Upon postmaster startup, remove any orphaned filenodes Actually, we don't have a good solution for finding orphaned filenodes right now. I do have some code that tries to do this as part of VACUUM but it was not 100% perfect, so it was rejected. I am willing to open the discussion to see if a perfect solution can be found. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From pgsql-hackers-owner+M21190@postgresql.org Thu Apr 11 13:40:34 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BHeXS12137 for ; Thu, 11 Apr 2002 13:40:33 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 2BD6C479604; Thu, 11 Apr 2002 12:35:51 -0400 (EDT) Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) by postgresql.org (Postfix) with ESMTP id 2DF9D47946A for ; Thu, 11 Apr 2002 12:31:25 -0400 (EDT) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id g3BGVM806114; Thu, 11 Apr 2002 12:31:22 -0400 (EDT) From: Bruce Momjian Message-ID: <200204111631.g3BGVM806114@candle.pha.pa.us> Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <1824.1018542155@sss.pgh.pa.us> To: Tom Lane Date: Thu, 11 Apr 2002 12:31:22 -0400 (EDT) cc: Christopher Kings-Lynne , Hiroshi Inoue , pgsql-hackers@postgresql.org X-Mailer: ELM [version 2.4ME+ PL97 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Tom Lane wrote: > Christopher Kings-Lynne writes: > > The real problem here is the fact that there are now missing attnos in > > pg_attribute. Either that's handled or we renumber the attnos - which is > > also quite hard? > > Updating pg_attribute per se is not so hard --- just store new copies of > all the rows for the table. However, propagating the changes into other > places could be quite painful (I'm thinking of column numbers in stored > constraints, rules, etc). > > It seems to me that reducing the column to NULLs already gets you the > majority of the space savings. I don't think there is a case to be made > that getting back that last bit is worth the pain involved, either in > implementation effort or direct runtime costs (do you really want a DROP > COLUMN to force an immediate rewrite of the whole table?) That is an excellent point about having to fix all the places that refer to attno. In fact, we have been moving away from attname references to attno references for a while, so it only gets worse. Tom is also correct that setting it to NULL removes the problem of disk space usage quite easily. That only leaves the problem of having gaps in the pg_attribute for that relation, and as I remember, that was the problem for Hiroshi's DROP COLUMN change, but at this point, after years of delay with no great solution on the horizon, we may as well just get this working. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 ---------------------------(end of broadcast)--------------------------- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly From Inoue@tpf.co.jp Thu Apr 11 19:55:08 2002 Return-path: Received: from sd.tpf.co.jp (IDENT:qmailr@sd.tpf.co.jp [210.161.239.34]) by candle.pha.pa.us (8.11.6/8.10.1) with SMTP id g3BNt6S19759 for ; Thu, 11 Apr 2002 19:55:06 -0400 (EDT) Received: (qmail 31013 invoked from network); 11 Apr 2002 23:55:06 -0000 Received: from unknown (HELO viscomail.tpf.co.jp) (100.0.0.108) by sd2.tpf-fw-c.co.jp with SMTP; 11 Apr 2002 23:55:06 -0000 Received: from tpf.co.jp (3dgateway1 [126.0.1.60]) by viscomail.tpf.co.jp (8.8.8+Sun/8.8.8) with ESMTP id IAA22335; Fri, 12 Apr 2002 08:55:05 +0900 (JST) Message-ID: <3CB62298.88565A54@tpf.co.jp> Date: Fri, 12 Apr 2002 08:56:08 +0900 From: Hiroshi Inoue X-Mailer: Mozilla 4.73 [ja] (Windows NT 5.0; U) X-Accept-Language: ja MIME-Version: 1.0 To: Christopher Kings-Lynne cc: Bruce Momjian , Tom Lane , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate References: <20020411233659.O69846-100000@houston.familyhealth.com.au> Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Status: OR Christopher Kings-Lynne wrote: > > Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is > kinda useless - you may as well just use a view!!! > > So how would this occur?: > > 1. Lock target table for writing (allow reads) > 2. Begin a table scan on target table, writing > a new file with a particular filenode > 3. Delete the attribute row from pg_attribute > 4. Point the table in the catalog to the new filenode > 5. Release locks > 6. Commit transaction > 7. Delete orhpan filenode > > i. Upon postmaster startup, remove any orphaned filenodes > > The real problem here is the fact that there are now missing attnos in > pg_attribute. Either that's handled or we renumber the attnos - which is > also quite hard? The attnos should be renumbered and it's easy. But the above seems only 20% of the total implementation. If the attnos are renumbered, all objects which refer to the numbers must be invalidated or re-compiled ... For example the parameters of foreign key constraints triggers are consist of relname and colnames currently. There has been a proposal that change to use relid or column numbers instead. Certainly it makes RENAME happy but DROP COLUMN unhappy. If there's a foreign key a_rel/1/3 and the second column of the relation is dropped the parameter must be changed to be a_rel/1/2. If neither foreign key stuff nor DROP COLUMN take the other into account, the consistency is easily broken. regards, Hiroshi Inoue From pgsql-hackers-owner+M21205@postgresql.org Thu Apr 11 19:56:20 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3BNuJS19855 for ; Thu, 11 Apr 2002 19:56:19 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 2B38A47656E; Thu, 11 Apr 2002 19:55:57 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by postgresql.org (Postfix) with SMTP id 6C92E475C96 for ; Thu, 11 Apr 2002 19:55:04 -0400 (EDT) Received: (qmail 31013 invoked from network); 11 Apr 2002 23:55:06 -0000 Received: from unknown (HELO viscomail.tpf.co.jp) (100.0.0.108) by sd2.tpf-fw-c.co.jp with SMTP; 11 Apr 2002 23:55:06 -0000 Received: from tpf.co.jp (3dgateway1 [126.0.1.60]) by viscomail.tpf.co.jp (8.8.8+Sun/8.8.8) with ESMTP id IAA22335; Fri, 12 Apr 2002 08:55:05 +0900 (JST) Message-ID: <3CB62298.88565A54@tpf.co.jp> Date: Fri, 12 Apr 2002 08:56:08 +0900 From: Hiroshi Inoue X-Mailer: Mozilla 4.73 [ja] (Windows NT 5.0; U) X-Accept-Language: ja MIME-Version: 1.0 To: Christopher Kings-Lynne cc: Bruce Momjian , Tom Lane , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate References: <20020411233659.O69846-100000@houston.familyhealth.com.au> Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: ORr Christopher Kings-Lynne wrote: > > Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is > kinda useless - you may as well just use a view!!! > > So how would this occur?: > > 1. Lock target table for writing (allow reads) > 2. Begin a table scan on target table, writing > a new file with a particular filenode > 3. Delete the attribute row from pg_attribute > 4. Point the table in the catalog to the new filenode > 5. Release locks > 6. Commit transaction > 7. Delete orhpan filenode > > i. Upon postmaster startup, remove any orphaned filenodes > > The real problem here is the fact that there are now missing attnos in > pg_attribute. Either that's handled or we renumber the attnos - which is > also quite hard? The attnos should be renumbered and it's easy. But the above seems only 20% of the total implementation. If the attnos are renumbered, all objects which refer to the numbers must be invalidated or re-compiled ... For example the parameters of foreign key constraints triggers are consist of relname and colnames currently. There has been a proposal that change to use relid or column numbers instead. Certainly it makes RENAME happy but DROP COLUMN unhappy. If there's a foreign key a_rel/1/3 and the second column of the relation is dropped the parameter must be changed to be a_rel/1/2. If neither foreign key stuff nor DROP COLUMN take the other into account, the consistency is easily broken. regards, Hiroshi Inoue ---------------------------(end of broadcast)--------------------------- TIP 6: Have you searched our list archives? http://archives.postgresql.org From pgsql-hackers-owner+M21209@postgresql.org Thu Apr 11 22:27:40 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3C2ReS27660 for ; Thu, 11 Apr 2002 22:27:40 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id A89AF4766D0; Thu, 11 Apr 2002 22:27:35 -0400 (EDT) Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) by postgresql.org (Postfix) with ESMTP id 4CE13475EB9 for ; Thu, 11 Apr 2002 22:26:25 -0400 (EDT) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id g3C2Q5I27551; Thu, 11 Apr 2002 22:26:05 -0400 (EDT) From: Bruce Momjian Message-ID: <200204120226.g3C2Q5I27551@candle.pha.pa.us> Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <3CB62298.88565A54@tpf.co.jp> To: Hiroshi Inoue Date: Thu, 11 Apr 2002 22:26:05 -0400 (EDT) cc: Christopher Kings-Lynne , Tom Lane , pgsql-hackers@postgresql.org X-Mailer: ELM [version 2.4ME+ PL97 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Hiroshi Inoue wrote: > Christopher Kings-Lynne wrote: > > > > Hmmm. Personally, I think that a DROP COLUMN that cannot reclaim space is > > kinda useless - you may as well just use a view!!! > > > > So how would this occur?: > > > > 1. Lock target table for writing (allow reads) > > 2. Begin a table scan on target table, writing > > a new file with a particular filenode > > 3. Delete the attribute row from pg_attribute > > 4. Point the table in the catalog to the new filenode > > 5. Release locks > > 6. Commit transaction > > 7. Delete orhpan filenode > > > > i. Upon postmaster startup, remove any orphaned filenodes > > > > The real problem here is the fact that there are now missing attnos in > > pg_attribute. Either that's handled or we renumber the attnos - which is > > also quite hard? > > The attnos should be renumbered and it's easy. > But the above seems only 20% of the total implementation. > If the attnos are renumbered, all objects which refer to > the numbers must be invalidated or re-compiled ... > For example the parameters of foreign key constraints > triggers are consist of relname and colnames currently. > There has been a proposal that change to use relid or > column numbers instead. Certainly it makes RENAME happy > but DROP COLUMN unhappy. If there's a foreign key a_rel/1/3 > and the second column of the relation is dropped the > parameter must be changed to be a_rel/1/2. If neither > foreign key stuff nor DROP COLUMN take the other into > account, the consistency is easily broken. I think that is why Tom was suggesting making all the column values NULL and removing the pg_attribute row for the column. With a NULL value, it doesn't take up any room in the tuple, and with the pg_attribute column gone, no one will see that row. The only problem is the gap in attno numbering. How big a problem is that? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From pgsql-hackers-owner+M21211@postgresql.org Thu Apr 11 22:55:44 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3C2tiS29394 for ; Thu, 11 Apr 2002 22:55:44 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id B86AF476739; Thu, 11 Apr 2002 22:55:39 -0400 (EDT) Received: from sss.pgh.pa.us (unknown [192.204.191.242]) by postgresql.org (Postfix) with ESMTP id 56D8747593C for ; Thu, 11 Apr 2002 22:54:26 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3C2s1F24007; Thu, 11 Apr 2002 22:54:01 -0400 (EDT) To: Bruce Momjian cc: Hiroshi Inoue , Christopher Kings-Lynne , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate In-Reply-To: <200204120226.g3C2Q5I27551@candle.pha.pa.us> References: <200204120226.g3C2Q5I27551@candle.pha.pa.us> Comments: In-reply-to Bruce Momjian message dated "Thu, 11 Apr 2002 22:26:05 -0400" Date: Thu, 11 Apr 2002 22:54:01 -0400 Message-ID: <24004.1018580041@sss.pgh.pa.us> From: Tom Lane Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Bruce Momjian writes: > I think that is why Tom was suggesting making all the column values NULL > and removing the pg_attribute row for the column. That was not my suggestion. > With a NULL value, it > doesn't take up any room in the tuple, and with the pg_attribute column > gone, no one will see that row. The only problem is the gap in attno > numbering. How big a problem is that? You can't do it that way unless you're intending to rewrite all rows of the relation before committing the ALTER; which would be the worst of both worlds. The pg_attribute row *must* be retained to show the datatype of the former column, so that we can correctly skip over it in tuples where the column isn't yet nulled out. Hiroshi did this by renumbering the attnum; I propose leaving attnum alone and instead adding an attisdropped flag. That would avoid creating a gap in the column numbers, but either way is likely to affect some applications that inspect pg_attribute. regards, tom lane ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org From pgsql-hackers-owner+M21214@postgresql.org Fri Apr 12 00:09:26 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3C49PS05093 for ; Fri, 12 Apr 2002 00:09:25 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id B1BE6476810; Fri, 12 Apr 2002 00:09:20 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by postgresql.org (Postfix) with SMTP id A8E07476444 for ; Fri, 12 Apr 2002 00:08:22 -0400 (EDT) Received: (qmail 25808 invoked from network); 12 Apr 2002 04:08:26 -0000 Received: from unknown (HELO viscomail.tpf.co.jp) (100.0.0.108) by sd2.tpf-fw-c.co.jp with SMTP; 12 Apr 2002 04:08:26 -0000 Received: from tpf.co.jp (3dgateway1 [126.0.1.60]) by viscomail.tpf.co.jp (8.8.8+Sun/8.8.8) with ESMTP id NAA22497; Fri, 12 Apr 2002 13:08:24 +0900 (JST) Message-ID: <3CB65DF7.8FCFC024@tpf.co.jp> Date: Fri, 12 Apr 2002 13:09:28 +0900 From: Hiroshi Inoue X-Mailer: Mozilla 4.73 [ja] (Windows NT 5.0; U) X-Accept-Language: ja MIME-Version: 1.0 To: Bruce Momjian cc: Christopher Kings-Lynne , Tom Lane , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate References: <200204120226.g3C2Q5I27551@candle.pha.pa.us> Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Bruce Momjian wrote: > > Hiroshi Inoue wrote: > > Christopher Kings-Lynne wrote: > > > > I think that is why Tom was suggesting making all the column values NULL > and removing the pg_attribute row for the column. With a NULL value, it > doesn't take up any room in the tuple, and with the pg_attribute column > gone, no one will see that row. The only problem is the gap in attno > numbering. How big a problem is that? There's no problem with applications which don't inquire of system catalogs(pg_attribute). Unfortunately we have been very tolerant of users' access on system tables and there would be pretty many applications which inquire of pg_attribute. regards, Hiroshi Inoue ---------------------------(end of broadcast)--------------------------- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html From pgsql-hackers-owner+M21221@postgresql.org Fri Apr 12 05:11:00 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3C9AxS28516 for ; Fri, 12 Apr 2002 05:11:00 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 28FF0476B9D; Fri, 12 Apr 2002 04:35:54 -0400 (EDT) Received: from tele-post-20.mail.demon.net (tele-post-20.mail.demon.net [194.217.242.20]) by postgresql.org (Postfix) with ESMTP id BFDE74769AC for ; Fri, 12 Apr 2002 04:30:29 -0400 (EDT) Received: from mailgate.vale-housing.co.uk ([193.195.77.162] helo=dogbert.vale-housing.co.uk) by tele-post-20.mail.demon.net with esmtp (Exim 3.35 #1) id 16vwRc-0006GP-0K; Fri, 12 Apr 2002 08:30:08 +0000 Received: by dogbert.vale-housing.co.uk with Internet Mail Service (5.5.2650.21) id <2H2ZS6HB>; Fri, 12 Apr 2002 09:35:53 +0100 Message-ID: From: Dave Page To: "'Tom Lane'" , Bruce Momjian cc: Hiroshi Inoue , Christopher Kings-Lynne , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate Date: Fri, 12 Apr 2002 09:35:52 +0100 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR > -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: 12 April 2002 03:54 > To: Bruce Momjian > Cc: Hiroshi Inoue; Christopher Kings-Lynne; > pgsql-hackers@postgresql.org > Subject: Re: RFC: Restructuring pg_aggregate > > > Bruce Momjian writes: > > I think that is why Tom was suggesting making all the column values > > NULL and removing the pg_attribute row for the column. > > That was not my suggestion. > > > With a NULL value, it > > doesn't take up any room in the tuple, and with the pg_attribute > > column gone, no one will see that row. The only problem is > the gap in > > attno numbering. How big a problem is that? > > You can't do it that way unless you're intending to rewrite > all rows of the relation before committing the ALTER; which > would be the worst of both worlds. The pg_attribute row > *must* be retained to show the datatype of the former column, > so that we can correctly skip over it in tuples where the > column isn't yet nulled out. > > Hiroshi did this by renumbering the attnum; I propose leaving > attnum alone and instead adding an attisdropped flag. That > would avoid creating a gap in the column numbers, but either > way is likely to affect some applications that inspect pg_attribute. Applications like pgAdmin that inspect pg_attribute are being seriously hacked to incorporate schema support anyway for 7.3. Personnally I'd be glad to spend some time re-coding to allow for this, just to not have to answer the numerous 'how do I drop a column' emails I get reguarly. Regards, Dave. ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From chriskl@familyhealth.com.au Sat Apr 13 02:25:23 2002 Return-path: Received: from mail.iinet.net.au (symphony-01.iinet.net.au [203.59.3.33]) by candle.pha.pa.us (8.11.6/8.10.1) with SMTP id g3D6PLS06807 for ; Sat, 13 Apr 2002 02:25:22 -0400 (EDT) Received: (qmail 7569 invoked by uid 666); 13 Apr 2002 06:25:20 -0000 Received: from unknown (HELO SOL) (203.59.103.193) by mail.iinet.net.au with SMTP; 13 Apr 2002 06:25:20 -0000 Message-ID: <001701c1e2b2$e7b10a40$0200a8c0@SOL> From: "Christopher Kings-Lynne" To: "Tom Lane" cc: "Bruce Momjian" , "Hiroshi Inoue" , References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate Date: Sat, 13 Apr 2002 14:17:34 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Status: OR > Updating pg_attribute per se is not so hard --- just store new copies of > all the rows for the table. However, propagating the changes into other > places could be quite painful (I'm thinking of column numbers in stored > constraints, rules, etc). > > It seems to me that reducing the column to NULLs already gets you the > majority of the space savings. I don't think there is a case to be made > that getting back that last bit is worth the pain involved, either in > implementation effort or direct runtime costs (do you really want a DROP > COLUMN to force an immediate rewrite of the whole table?) OK, sounds fair. However, is there a more aggressive way of reclaiming the space? The problem with updating all the rows to null for that column is that the on-disk size is doubled anyway, right? So, could a VACUUM FULL process do the nulling for us? Vacuum works outside of normal transaction constraints anyway...? Also, it seems to me that at some point we are forced to break client compatibility. Either we add attisdropped field to pg_attribute, or we use Hiroshi's (-1 * attnum - offset) idea. Both Tom and Hiroshi have good reasons for each of these - would it be possible for you guys to post with your reasons for and against both the techniques. I just want to get to an implementation specification we all agree on that can be written up and then the coding can proceed. Maybe we should add it to the 'Postgres Major Projects' page - and remove those old ones that have already been implemented. Chris From Inoue@tpf.co.jp Sun Apr 14 23:47:08 2002 Return-path: Received: from sd.tpf.co.jp (IDENT:qmailr@sd.tpf.co.jp [210.161.239.34]) by candle.pha.pa.us (8.11.6/8.10.1) with SMTP id g3F3l6S23155 for ; Sun, 14 Apr 2002 23:47:07 -0400 (EDT) Received: (qmail 9638 invoked from network); 15 Apr 2002 03:47:06 -0000 Received: from unknown (HELO viscomail.tpf.co.jp) (100.0.0.108) by sd2.tpf-fw-c.co.jp with SMTP; 15 Apr 2002 03:47:06 -0000 Received: from tpf.co.jp (3dgateway1 [126.0.1.60]) by viscomail.tpf.co.jp (8.8.8+Sun/8.8.8) with ESMTP id MAA24068; Mon, 15 Apr 2002 12:47:04 +0900 (JST) Message-ID: <3CBA4D7A.9E61DECA@tpf.co.jp> Date: Mon, 15 Apr 2002 12:48:10 +0900 From: Hiroshi Inoue X-Mailer: Mozilla 4.73 [ja] (Windows NT 5.0; U) X-Accept-Language: ja MIME-Version: 1.0 To: Christopher Kings-Lynne cc: Tom Lane , Bruce Momjian , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] RFC: Restructuring pg_aggregate References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Status: OR Christopher Kings-Lynne wrote: > > Also, it seems to me that at some point we are forced to break client > compatibility. It's not a users' consensus at all. I'm suspicious if DROP COLUMN is such a significant feature to break client compatibility at our ease. > Either we add attisdropped field to pg_attribute, or we use > Hiroshi's (-1 * attnum - offset) idea. Both Tom and Hiroshi have good > reasons for each of these - would it be possible for you guys to post with > your reasons for and against both the techniques. I don't object to adding attisdropped field. What I meant to say is that the differene is very small. regards, Hiroshi Inoue From tgl@sss.pgh.pa.us Sat Apr 13 11:30:17 2002 Return-path: Received: from sss.pgh.pa.us (root@[192.204.191.242]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3DFUGS26218 for ; Sat, 13 Apr 2002 11:30:16 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3DFTjF15655; Sat, 13 Apr 2002 11:29:45 -0400 (EDT) To: "Christopher Kings-Lynne" cc: "Bruce Momjian" , "Hiroshi Inoue" , pgsql-hackers@postgresql.org Subject: Re: DROP COLUMN (was RFC: Restructuring pg_aggregate) In-Reply-To: <001701c1e2b2$e7b10a40$0200a8c0@SOL> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> Comments: In-reply-to "Christopher Kings-Lynne" message dated "Sat, 13 Apr 2002 14:17:34 +0800" Date: Sat, 13 Apr 2002 11:29:45 -0400 Message-ID: <15652.1018711785@sss.pgh.pa.us> From: Tom Lane Status: OR [ way past time to change the title of this thread ] "Christopher Kings-Lynne" writes: > OK, sounds fair. However, is there a more aggressive way of reclaiming the > space? The problem with updating all the rows to null for that column is > that the on-disk size is doubled anyway, right? So, could a VACUUM FULL > process do the nulling for us? Vacuum works outside of normal transaction > constraints anyway...? No, VACUUM has the same transactional constraints as everyone else (unless you'd like a crash during VACUUM to trash your table...) I do not think that we necessarily need to provide a special mechanism for this at all. The docs for DROP COLUMN could simply explain that the DROP itself doesn't reclaim the space, but that the space will be reclaimed over time as extant rows are updated or deleted. If you want to hurry the process along you could do UPDATE table SET othercol = othercol VACUUM FULL to force all the rows to be updated and then reclaim space. But given the peak-space-is-twice-as-much behavior, this is not obviously a win. I'd sure object to an implementation that *forced* that approach on me, whether during DROP itself or the next VACUUM. > Also, it seems to me that at some point we are forced to break client > compatibility. Either we add attisdropped field to pg_attribute, or we use > Hiroshi's (-1 * attnum - offset) idea. Both Tom and Hiroshi have good > reasons for each of these - would it be possible for you guys to post with > your reasons for and against both the techniques. Er, didn't we do that already? regards, tom lane From chriskl@familyhealth.com.au Sun Apr 14 01:06:31 2002 Return-path: Received: from mail.iinet.net.au (symphony-03.iinet.net.au [203.59.3.35]) by candle.pha.pa.us (8.11.6/8.10.1) with SMTP id g3E56TS03274 for ; Sun, 14 Apr 2002 01:06:30 -0400 (EDT) Received: (qmail 20365 invoked by uid 666); 14 Apr 2002 05:06:31 -0000 Received: from unknown (HELO SOL) (203.59.168.230) by mail.iinet.net.au with SMTP; 14 Apr 2002 05:06:31 -0000 Message-ID: <00c601c1e371$0e324670$0200a8c0@SOL> From: "Christopher Kings-Lynne" To: "Tom Lane" cc: "Bruce Momjian" , "Hiroshi Inoue" , References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> Subject: Re: DROP COLUMN (was RFC: Restructuring pg_aggregate) Date: Sun, 14 Apr 2002 12:58:43 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Status: OR > No, VACUUM has the same transactional constraints as everyone else > (unless you'd like a crash during VACUUM to trash your table...) Seriously, you can run VACUUM in a transaction and rollback the movement of a tuple on disk? What do you mean by same transactional constraints? Chris From pgsql-hackers-owner+M21278@postgresql.org Sat Apr 13 12:21:20 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3DGLKS29823 for ; Sat, 13 Apr 2002 12:21:20 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 9B4AF475CA6; Sat, 13 Apr 2002 12:21:12 -0400 (EDT) Received: from sss.pgh.pa.us (unknown [192.204.191.242]) by postgresql.org (Postfix) with ESMTP id 1ED76474E71 for ; Sat, 13 Apr 2002 12:20:07 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3DGJeF15983; Sat, 13 Apr 2002 12:19:40 -0400 (EDT) To: Hannu Krosing cc: Christopher Kings-Lynne , Bruce Momjian , Hiroshi Inoue , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] DROP COLUMN (was RFC: Restructuring pg_aggregate) In-Reply-To: <1018716432.3360.9.camel@taru.tm.ee> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> <1018716432.3360.9.camel@taru.tm.ee> Comments: In-reply-to Hannu Krosing message dated "13 Apr 2002 18:47:07 +0200" Date: Sat, 13 Apr 2002 12:19:40 -0400 Message-ID: <15980.1018714780@sss.pgh.pa.us> From: Tom Lane Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Hannu Krosing writes: >> No, VACUUM has the same transactional constraints as everyone else >> (unless you'd like a crash during VACUUM to trash your table...) > But can't it do the SET TO NULL thing if it knows that the transaction > that dropped the column has committed. Hmm, you're thinking of allowing VACUUM to overwrite tuples in-place? Strikes me as unsafe, but I'm not really sure. In any case it's not that easy. If the column is wide enough that reclaiming its space is actually worth doing, then presumably most of its entries are just TOAST links, and what has to be done is not just rewrite the main tuple but mark the TOAST rows deleted. This is not something that VACUUM does now; I'd be rather concerned about the locking implications (especially for lightweight VACUUM). regards, tom lane ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org From pgsql-hackers-owner+M21277@postgresql.org Sat Apr 13 11:51:02 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3DFp1S28016 for ; Sat, 13 Apr 2002 11:51:01 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id B76F5475D68; Sat, 13 Apr 2002 11:47:59 -0400 (EDT) Received: from gw.itmeedia.ee (gw.itmeedia.ee [213.180.3.226]) by postgresql.org (Postfix) with SMTP id 0635E475C6F for ; Sat, 13 Apr 2002 11:47:01 -0400 (EDT) Received: (qmail 12309 invoked from network); 13 Apr 2002 15:47:06 -0000 Received: from taru.itmeedia.ee (HELO taru.tm.ee) (213.180.3.230) by gw.itmeedia.ee with SMTP; 13 Apr 2002 15:47:06 -0000 Subject: Re: [HACKERS] DROP COLUMN (was RFC: Restructuring pg_aggregate) From: Hannu Krosing To: Tom Lane cc: Christopher Kings-Lynne , Bruce Momjian , Hiroshi Inoue , pgsql-hackers@postgresql.org In-Reply-To: <15652.1018711785@sss.pgh.pa.us> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.3.99 Date: 13 Apr 2002 18:47:07 +0200 Message-ID: <1018716432.3360.9.camel@taru.tm.ee> MIME-Version: 1.0 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Sat, 2002-04-13 at 17:29, Tom Lane wrote: > [ way past time to change the title of this thread ] > > "Christopher Kings-Lynne" writes: > > OK, sounds fair. However, is there a more aggressive way of reclaiming the > > space? The problem with updating all the rows to null for that column is > > that the on-disk size is doubled anyway, right? So, could a VACUUM FULL > > process do the nulling for us? Vacuum works outside of normal transaction > > constraints anyway...? > > No, VACUUM has the same transactional constraints as everyone else > (unless you'd like a crash during VACUUM to trash your table...) But can't it do the SET TO NULL thing if it knows that the transaction that dropped the column has committed. This could probably even be done in the light version of vacuum with a special flag (VACUUM RECLAIM). Of course running this this makes sense only if the dropped column had some significant amount of data . > I do not think that we necessarily need to provide a special mechanism > for this at all. The docs for DROP COLUMN could simply explain that > the DROP itself doesn't reclaim the space, but that the space will be > reclaimed over time as extant rows are updated or deleted. If you want > to hurry the process along you could do > UPDATE table SET othercol = othercol > VACUUM FULL If only we could do it in namageable chunks: FOR i IN 0 TO (size(table)/chunk) DO UPDATE table SET othercol = othercol OFFSET i*chunk LIMIT chunk VACUUM FULL; END FOR; or even better - "VACUUM FULL OFFSET i*chunk LIMIT chunk" and then make chunk == 1 :) -------------- Hannu ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From pgsql-hackers-owner+M21292@postgresql.org Sun Apr 14 01:07:16 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3E57FS03403 for ; Sun, 14 Apr 2002 01:07:15 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 78A86475DD7; Sun, 14 Apr 2002 01:07:12 -0400 (EDT) Received: from mail.iinet.net.au (symphony-03.iinet.net.au [203.59.3.35]) by postgresql.org (Postfix) with SMTP id DA1D447593E for ; Sun, 14 Apr 2002 01:06:32 -0400 (EDT) Received: (qmail 20365 invoked by uid 666); 14 Apr 2002 05:06:31 -0000 Received: from unknown (HELO SOL) (203.59.168.230) by mail.iinet.net.au with SMTP; 14 Apr 2002 05:06:31 -0000 Message-ID: <00c601c1e371$0e324670$0200a8c0@SOL> From: "Christopher Kings-Lynne" To: "Tom Lane" cc: "Bruce Momjian" , "Hiroshi Inoue" , References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> Subject: Re: [HACKERS] DROP COLUMN (was RFC: Restructuring pg_aggregate) Date: Sun, 14 Apr 2002 12:58:43 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR > No, VACUUM has the same transactional constraints as everyone else > (unless you'd like a crash during VACUUM to trash your table...) Seriously, you can run VACUUM in a transaction and rollback the movement of a tuple on disk? What do you mean by same transactional constraints? Chris ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From tgl@sss.pgh.pa.us Sun Apr 14 14:13:33 2002 Return-path: Received: from sss.pgh.pa.us (root@[192.204.191.242]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3EIDWS18224 for ; Sun, 14 Apr 2002 14:13:32 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3EIDMF22681; Sun, 14 Apr 2002 14:13:22 -0400 (EDT) To: "Christopher Kings-Lynne" cc: "Bruce Momjian" , "Hiroshi Inoue" , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] DROP COLUMN (was RFC: Restructuring pg_aggregate) In-Reply-To: <00c601c1e371$0e324670$0200a8c0@SOL> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> <00c601c1e371$0e324670$0200a8c0@SOL> Comments: In-reply-to "Christopher Kings-Lynne" message dated "Sun, 14 Apr 2002 12:58:43 +0800" Date: Sun, 14 Apr 2002 14:13:21 -0400 Message-ID: <22678.1018808001@sss.pgh.pa.us> From: Tom Lane Status: OR "Christopher Kings-Lynne" writes: >> No, VACUUM has the same transactional constraints as everyone else >> (unless you'd like a crash during VACUUM to trash your table...) > Seriously, you can run VACUUM in a transaction and rollback the movement of > a tuple on disk? What do you mean by same transactional constraints? In VACUUM FULL, tuples moved to compact the table aren't good until you commit. In this hypothetical column-drop-implementing VACUUM, I think there'd need to be some similar rule --- otherwise it's not clear what happens to TOASTED data if you crash partway through. (In particular, if we tried overwriting main tuples in place as Hannu was suggesting, we'd need a way of being certain the deletion of the corresponding TOAST rows occurs *before* we overwrite the only reference to them.) regards, tom lane From pgsql-hackers-owner+M21305@postgresql.org Sun Apr 14 14:14:46 2002 Return-path: Received: from postgresql.org (postgresql.org [64.49.215.8]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g3EIEkS18333 for ; Sun, 14 Apr 2002 14:14:46 -0400 (EDT) Received: from postgresql.org (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with SMTP id 8FA74475C4C; Sun, 14 Apr 2002 14:14:43 -0400 (EDT) Received: from sss.pgh.pa.us (unknown [192.204.191.242]) by postgresql.org (Postfix) with ESMTP id 8AC04475892 for ; Sun, 14 Apr 2002 14:13:52 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g3EIDMF22681; Sun, 14 Apr 2002 14:13:22 -0400 (EDT) To: "Christopher Kings-Lynne" cc: "Bruce Momjian" , "Hiroshi Inoue" , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] DROP COLUMN (was RFC: Restructuring pg_aggregate) In-Reply-To: <00c601c1e371$0e324670$0200a8c0@SOL> References: <20020411233659.O69846-100000@houston.familyhealth.com.au> <1824.1018542155@sss.pgh.pa.us> <001701c1e2b2$e7b10a40$0200a8c0@SOL> <15652.1018711785@sss.pgh.pa.us> <00c601c1e371$0e324670$0200a8c0@SOL> Comments: In-reply-to "Christopher Kings-Lynne" message dated "Sun, 14 Apr 2002 12:58:43 +0800" Date: Sun, 14 Apr 2002 14:13:21 -0400 Message-ID: <22678.1018808001@sss.pgh.pa.us> From: Tom Lane Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR "Christopher Kings-Lynne" writes: >> No, VACUUM has the same transactional constraints as everyone else >> (unless you'd like a crash during VACUUM to trash your table...) > Seriously, you can run VACUUM in a transaction and rollback the movement of > a tuple on disk? What do you mean by same transactional constraints? In VACUUM FULL, tuples moved to compact the table aren't good until you commit. In this hypothetical column-drop-implementing VACUUM, I think there'd need to be some similar rule --- otherwise it's not clear what happens to TOASTED data if you crash partway through. (In particular, if we tried overwriting main tuples in place as Hannu was suggesting, we'd need a way of being certain the deletion of the corresponding TOAST rows occurs *before* we overwrite the only reference to them.) regards, tom lane ---------------------------(end of broadcast)--------------------------- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html