mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-08-26 04:37:19 +02:00
616 lines
25 KiB
Plaintext
616 lines
25 KiB
Plaintext
From pgsql-hackers-owner+M59479@postgresql.org Thu Sep 30 15:55:23 2004
|
|
Return-path: <pgsql-hackers-owner+M59479@postgresql.org>
|
|
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i8UJtHw26165
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 15:55:19 -0400 (EDT)
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id A4BDD32A219
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 20:55:10 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 24195-05 for <pgman@candle.pha.pa.us>;
|
|
Thu, 30 Sep 2004 19:55:08 +0000 (GMT)
|
|
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 537C532A216
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 20:55:10 +0100 (BST)
|
|
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 333B932A1EF
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Thu, 30 Sep 2004 20:49:20 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 21793-04
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
|
|
Thu, 30 Sep 2004 19:49:09 +0000 (GMT)
|
|
Received: from authenticity.encs.concordia.ca (authenticity-96.encs.concordia.ca [132.205.96.93])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id BEB6A32A156
|
|
for <pgsql-hackers@postgresql.org>; Thu, 30 Sep 2004 20:49:03 +0100 (BST)
|
|
Received: from haida.cs.concordia.ca (IDENT:mokhov@haida.cs.concordia.ca [132.205.64.45])
|
|
by authenticity.encs.concordia.ca (8.12.11/8.12.11) with ESMTP id i8UJn0Xe022202;
|
|
Thu, 30 Sep 2004 15:49:00 -0400
|
|
Date: Thu, 30 Sep 2004 15:49:00 -0400 (EDT)
|
|
From: "Serguei A. Mokhov" <mokhov@cs.concordia.ca>
|
|
To: pgsql-hackers@postgresql.org
|
|
cc: "Serguei A. Mokhov" <mokhov@cs.concordia.ca>
|
|
Subject: [HACKERS] pg_upgrade project: high-level design proposal of in-place upgrade
|
|
facility
|
|
Message-ID: <Pine.LNX.4.58.0409301532390.4685@haida.cs.concordia.ca>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Mailing-List: pgsql-hackers
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on
|
|
candle.pha.pa.us
|
|
X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham
|
|
version=2.61
|
|
Status: OR
|
|
|
|
Hello dear all,
|
|
|
|
[Please CC your replies to me as I am on the digest mode]
|
|
|
|
Here's finally a very high-level design proposal of the pg_upgrade feature
|
|
I was handwaiving a couple of weeks ago. Since, I am almost done with the
|
|
moving, I can allocate some time for this for 8.1/8.2.
|
|
|
|
If this topic is of interest to you, please read on until the very end
|
|
before flaming or bashing the ideas out. I had designed that thing and
|
|
kept updating (the design) more or less regularly, and also reflected some
|
|
issues from the nearby threads [1] and [2].
|
|
|
|
This design is very high-level at the moment and is not very detailed. I
|
|
will need to figure out more stuff as I go and design some aspects in
|
|
finer detail. I started to poke around asking for initdb-forcing code
|
|
paths in [3], but got no response so far. But I guess if the general idea
|
|
or, rather, ideas accepted I will insist on more information more
|
|
aggressively :) if I can't figure something out for myself.
|
|
|
|
[1] http://archives.postgresql.org/pgsql-hackers/2004-09/msg00000.php
|
|
[2] http://archives.postgresql.org/pgsql-hackers/2004-09/msg00382.php
|
|
[3] http://archives.postgresql.org/pgsql-hackers/2004-08/msg01594.php
|
|
|
|
Comments are very welcome, especially _*CONSTRUCTIVE*_...
|
|
|
|
Thank you, and now sit back and read...
|
|
|
|
CONTENTS:
|
|
=========
|
|
|
|
1. The Need
|
|
1. Utilities and User's View of the pg_upgrade Feature
|
|
2. Storage Management
|
|
- Storage Managers and the smgr API
|
|
3. Source Code Maintenance Aspects
|
|
2. The Upgrade Sequence
|
|
4. Proposed Implementation Plan
|
|
- initdb() API
|
|
- upgrade API
|
|
|
|
|
|
1. The Need
|
|
-----------
|
|
|
|
It's been a problem for PG for quite awhile now to have a less painful
|
|
upgrade procedure with every new revision of PostgreSQL, so the
|
|
dump/restore sequence is required. That can take a while for a production
|
|
DB, while keeping it offline. The new replication-related solutions, such
|
|
as Slony I, pg_pool, and others can remedy the problem somewhat, but
|
|
require to roughly double the storage requirements of a given database
|
|
while replicating from the older server to a newer one.
|
|
|
|
The proposed implementation of an in-server pg_upgrade facility attempts
|
|
to address both issues at the same time -- a possibility to keep the
|
|
server running and upgrading lazily w/o doubling the storage requirements
|
|
(there will be some extra disk space taken, but far from doubling the
|
|
size). The in-process upgrade will not take much of down time and won't
|
|
require that much memory/disk/network resources as replication solutions
|
|
do.
|
|
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
Ideally, the (maybe not so anymore) ambitious goal is to simply be able to
|
|
"drop in" the new binaries of the new server and kick off on the older
|
|
version of data files. I think is this feasible now a lot more than before
|
|
since we have those things available, which should ease up the
|
|
implementation:
|
|
|
|
- bgwriter
|
|
- pg_autovacuum (the one to be integrated into the backend in 8.1)
|
|
- smgr API for pluggable storage managers
|
|
- initdb in C
|
|
- ...
|
|
|
|
initdb in C, bgwriter and pg_autovacuum, and pluggable storage manager
|
|
have made the possibility of creation of the Upgrade Subsystem for
|
|
PostgreSQL to be something more reasonable, complete, feasible, and sane
|
|
to a point.
|
|
|
|
|
|
Utilities and the User's (DBA) View of the Feature
|
|
--------------------------------------------------
|
|
|
|
Two instances exist:
|
|
|
|
pg_upgrade (in C)
|
|
|
|
A standalone utility to upgrade the binary on-disk format from one
|
|
version to another when the database is offline.
|
|
We should always have this as an option.
|
|
pg_upgrade will accept sub/super set of pg_dump(all)/pg_restore
|
|
options that do not require a connection. I haven't
|
|
thought through this in detail yet.
|
|
|
|
pg_autoupgrade
|
|
|
|
a postgres subprocess, modeled after bgwriter and pg_autovacuum
|
|
daemons. This will work when the database system is running
|
|
on old data directory, and lazily converting relations to the new
|
|
format.
|
|
|
|
pg_autoupgrade daemon can be triggered by the following events in addition
|
|
to the lazy upgrade process:
|
|
|
|
"SQL" level: UPGRADE <ALL | relation_name [, relation_name]> [NOW | time]
|
|
|
|
While the database won't be offline running over older database files,
|
|
SELECT/read-only queries would be allowed using older storage managers*.
|
|
Any write operation on old data will act using write-invalidate approach
|
|
that will force the upgrade the affected relations to the new format to be
|
|
scheduled after the relation-in-progress.
|
|
|
|
(* See the "Storage Management" section.)
|
|
|
|
Availability of the relations while upgrade is in progress is likely to be
|
|
the same as in VACUUM FULL for that relation, i.e. the entire relation is
|
|
locked until the upgrade is complete. Maybe we could optimize that by
|
|
locking only particular pages of relations, I have to figure that out.
|
|
|
|
The upgrade of indices can be done using REINDEX, which seems far less
|
|
complicated than trying to convert its on-disk representation. This has
|
|
to be done after the relation is converted. Alternatively, the index
|
|
upgrade can simply be done by "CREATE INDEX" after the upgrade of
|
|
relations.
|
|
|
|
The relations to be upgraded are ordered according to some priority, e.g.
|
|
system relations being first, then user-owned relations. System relations
|
|
upgrade is forced upon the postmaster startup, and then user relations are
|
|
processed lazily.
|
|
|
|
So, in a sense, pg_autoupgrade will act like a proxy choosing appropriate
|
|
storage manager (like a driver) between the new server and the old data
|
|
file upgrading them on-demand. For that purpose we might need to add a
|
|
pg_upgradeproxy to intercept backend requests and use appropriate storage
|
|
manager. There will be one proxy process per backend.
|
|
|
|
|
|
Storage Management
|
|
==================
|
|
|
|
Somebody has made a possibility to plug a different storage manager in
|
|
postgres and we even had two of them at some point . for the magnetic disk
|
|
and the main memory. The main memory one is gone, but the smgr API is
|
|
still there. Some were dubious why we would ever need another third-party
|
|
storage manager, but here I propose to "plug in" storage managers from the
|
|
older Postgres versions itself! Here is where the pluggable storage
|
|
manager API would be handy once fully resurrected. Instead of trying to
|
|
plug some third party storage managers it will primarily be used by the
|
|
storage managers of different versions of Postgres.
|
|
|
|
We can take the storage manager code from the past maintenance releases,
|
|
namely 6.5.3, 7.0.3, 7.1.3, 7.2.5, 7.3.7, 7.4.5, and 8.0, and arrange them
|
|
in appropriate fashion and have them implement the API properly. Anyone
|
|
can contribute a storage manager as they see fit, there's no need to get
|
|
them all at once. As a trial implementation I will try to do the last
|
|
three or four maybe.
|
|
|
|
|
|
Where to put relations being upgraded?
|
|
--------------------------------------
|
|
|
|
At the beginning of the upgrade process if pg detects the old version of
|
|
data files, it moves them under $PGDATA/<ver>, and keeps the old relations
|
|
there until upgraded. The relations to be upgraded will be kept in the
|
|
pg_upgrade_catalog. Once all relations upgraded, the <ver> directory is
|
|
removed and the auto and proxy processes are shut down. The contents of
|
|
the pg_upgrade_catalog emptied. The only issue remains is how to deal
|
|
with tablespaces (or LOCATION in 7.* releases) elsewhere .- this can
|
|
probably be addressed in the similar fashion, but having a
|
|
/my/tablespace/<ver> directory.
|
|
|
|
Source Code Maintenance
|
|
=======================
|
|
|
|
Now, after the above some of you may get scared on the amount of similar
|
|
code to possibly maintain in all those storage managers, but in reality
|
|
they would require as much maintenance as the corresponding releases do
|
|
get back-patched in that code area, and some are not being maintained for
|
|
quite some time already. Plus, I should be around to maintain it, should
|
|
this become realized.
|
|
|
|
Release-time Maintenance
|
|
------------------------
|
|
|
|
For maintenance of pg_upgrade itself, one will have to fork out a new
|
|
storage manager from the previous stable release and "register" it within
|
|
the system. Alternatively, the new storage manager can be forked when the
|
|
new release cycle begins. Additionally, a pg_upgrade version has to be
|
|
added implementing the API steps outlined in the pg_upgrade API section.
|
|
|
|
|
|
Implementation Steps
|
|
====================
|
|
|
|
To materialize the above idea, I'd proceed as follows:
|
|
|
|
*) Provide the initdb() API (quick)
|
|
|
|
*) Resurrect the pluggable storage manager API to be usable for the
|
|
purpose.
|
|
|
|
*) Document it
|
|
|
|
*) Implement pg_upgrade API for 8.0 and 7.4.5.
|
|
|
|
*) Extract 8.0 and 7.4.5 storage managers and have them implement the API
|
|
as a proof of concept. Massage the API as needed.
|
|
|
|
*) Document the process of adding new storage managers and pg_upgrade
|
|
drivers.
|
|
|
|
*) Extract other versions storage managers.
|
|
|
|
|
|
pg_upgrade sequence
|
|
-------------------
|
|
|
|
pg_upgrade API for the steps below to update for the next release.
|
|
|
|
What to do with WAL?? Maybe upgrade can simply be done using WAL replay
|
|
with old WAL manager? Not, fully, because not everything is in WAL, but
|
|
some WAL recovery maybe needed in case the server was not shutdown cleanly
|
|
before the upgrade.
|
|
|
|
pg_upgrade will proceed as follows:
|
|
|
|
- move PGDATA to PGDATA/<major pg version>
|
|
- move tablespaces likewise
|
|
- optional recovery from WAL in case old server was not shutdown properly
|
|
-? Shall I upgrade PITR logs of 8.x??? So one can recover to a
|
|
point-in-time in the upgraded database?
|
|
- CLUSTER all old data
|
|
- ANALYZE all old data
|
|
- initdb() new system catalogs
|
|
- Merge in modifications from old system catalogs
|
|
- upgrade schemas/users
|
|
-- variations
|
|
- upgrade user relations
|
|
|
|
Upgrade API:
|
|
------------
|
|
|
|
First draft, to be refined multiple times, but to convey the ideas behind:
|
|
|
|
moveData()
|
|
movePGData()
|
|
moveTablespaces() 8.0+
|
|
moveDbLocation() < 8.0
|
|
|
|
preliminaryRecovery()
|
|
- WAL??
|
|
- PITR 8.0+??
|
|
|
|
preliminaryCleanup()
|
|
CLUSTER -- recover some dead space
|
|
ANALYZE -- gives us stats
|
|
|
|
upgradeSystemInfo()
|
|
initdb()
|
|
mergeOldCatalogs()
|
|
mergeOldTemplates()
|
|
|
|
upgradeUsers()
|
|
|
|
upgradeSchemas()
|
|
- > 7.2, else NULL
|
|
|
|
upgradeUserRelations()
|
|
upgradeIndices()
|
|
DROP/CREATE
|
|
|
|
upgradeInit()
|
|
{
|
|
|
|
}
|
|
|
|
The main body in pseudocode:
|
|
|
|
upgradeLoop()
|
|
{
|
|
moveData();
|
|
preliminaryRecovery();
|
|
preliminaryCleanup();
|
|
upgradeSystemInfo();
|
|
upgradeUsers();
|
|
upgradeSchemas();
|
|
upgradeUserRelations();
|
|
}
|
|
|
|
Something along these lines the API would be:
|
|
|
|
typedef struct t_upgrade
|
|
{
|
|
bool (*moveData) (void);
|
|
bool (*preliminaryRecovery) (void); /* may be NULL */
|
|
bool (*preliminaryCleanup) (void); /* may be NULL */
|
|
bool (*upgradeSystemInfo) (void); /* may be NULL */
|
|
bool (*upgradeUsers) (void); /* may be NULL */
|
|
bool (*upgradeSchemas) (void); /* may be NULL */
|
|
bool (*upgradeUserRelations) (void); /* may be NULL */
|
|
} t_upgrade;
|
|
|
|
|
|
The above sequence is executed by either pg_upgrade utility uninterrupted
|
|
or by the pg_autoupgrade daemon. In the former the upgrade priority is
|
|
simply by OID, in the latter also, but can be overridden by the user using
|
|
the UPGRADE command to schedule relations upgrade, write operation can
|
|
also change such schedule, with user's selected choice to be first. The
|
|
more write requests a relation receives while in the upgrade queue, its
|
|
priority increases; thus, the relation with most hits is on top. In case
|
|
of tie, OID is the decision mark.
|
|
|
|
Some issues to look into:
|
|
|
|
- catalog merger
|
|
- a crash in the middle of upgrade
|
|
- PITR logs for 8.x+
|
|
- ...
|
|
|
|
Flames and Handwaiving
|
|
----------------------
|
|
|
|
Okay, flame is on, but before you flame, mind you, this is a very initial
|
|
version of the design. Some of the ideas may seem far fetched, the
|
|
contents may seem messy, but I believe it's now more doable than ever and
|
|
I am willing to put effort in it for the next release or two and then
|
|
maintain it afterwards. It's not going to be done in one shot maybe, but
|
|
incrementally, using input, feedback, and hints from you, guys.
|
|
|
|
Thank you for reading till this far :-) I.d like to hear from you if any
|
|
of this made sense to you.
|
|
|
|
Truly yours,
|
|
|
|
--
|
|
Serguei A. Mokhov | /~\ The ASCII
|
|
Computer Science Department | \ / Ribbon Campaign
|
|
Concordia University | X Against HTML
|
|
Montreal, Quebec, Canada | / \ Email!
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 5: Have you checked our extensive FAQ?
|
|
|
|
http://www.postgresql.org/docs/faqs/FAQ.html
|
|
|
|
From pgsql-hackers-owner+M59486@postgresql.org Thu Sep 30 16:39:54 2004
|
|
Return-path: <pgsql-hackers-owner+M59486@postgresql.org>
|
|
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i8UKdrw02740
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 16:39:53 -0400 (EDT)
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id EF25F329E3B
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 21:39:48 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 38456-02 for <pgman@candle.pha.pa.us>;
|
|
Thu, 30 Sep 2004 20:39:46 +0000 (GMT)
|
|
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 88673329C6B
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 21:39:48 +0100 (BST)
|
|
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id AA522329DAE
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Thu, 30 Sep 2004 21:37:43 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 38130-02
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
|
|
Thu, 30 Sep 2004 20:37:39 +0000 (GMT)
|
|
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 846E9329C63
|
|
for <pgsql-hackers@postgresql.org>; Thu, 30 Sep 2004 21:37:39 +0100 (BST)
|
|
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
|
|
by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i8UKa3jk025254;
|
|
Thu, 30 Sep 2004 16:36:03 -0400 (EDT)
|
|
To: "Serguei A. Mokhov" <mokhov@cs.concordia.ca>
|
|
cc: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] pg_upgrade project: high-level design proposal of in-place upgrade facility
|
|
In-Reply-To: <Pine.LNX.4.58.0409301532390.4685@haida.cs.concordia.ca>
|
|
References: <Pine.LNX.4.58.0409301532390.4685@haida.cs.concordia.ca>
|
|
Comments: In-reply-to "Serguei A. Mokhov" <mokhov@cs.concordia.ca>
|
|
message dated "Thu, 30 Sep 2004 15:49:00 -0400"
|
|
Date: Thu, 30 Sep 2004 16:36:02 -0400
|
|
Message-ID: <25253.1096576562@sss.pgh.pa.us>
|
|
From: Tom Lane <tgl@sss.pgh.pa.us>
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Mailing-List: pgsql-hackers
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on
|
|
candle.pha.pa.us
|
|
X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham
|
|
version=2.61
|
|
Status: OR
|
|
|
|
"Serguei A. Mokhov" <mokhov@cs.concordia.ca> writes:
|
|
> Comments are very welcome, especially _*CONSTRUCTIVE*_...
|
|
|
|
This is fundamentally wrong, because you are assigning the storage
|
|
manager functionality that it does not have. In particular, the
|
|
storage manager knows nothing of the contents or format of the files
|
|
it is managing, and so you can't realistically expect to use the smgr
|
|
switch as a way to support access to tables with different internal
|
|
formats. The places that change in interesting ways across versions
|
|
are usually far above the smgr switch.
|
|
|
|
I don't believe in the idea of incremental "lazy" upgrades very much
|
|
either. It certainly will not work on the system catalogs --- you have
|
|
to convert those in a big-bang fashion, because how are you going to
|
|
find the other stuff otherwise? And the real problem with it IMHO is
|
|
that if something goes wrong partway through the process, you're in
|
|
deep trouble because you have no way to back out. You can't just revert
|
|
to the old version because it won't understand your data, and your old
|
|
backups that are compatible with it are now out of date. If there are
|
|
going to be any problems, you really need to find out about them
|
|
immediately while your old backups are still current, not in a "lazy"
|
|
fashion.
|
|
|
|
The design we'd pretty much all bought into six months ago involved
|
|
being able to do in-place upgrades when the format/contents of user
|
|
relations and indexes is not changing. All you'd have to do is dump
|
|
and restore the schema data (system catalogs) which is a reasonably
|
|
short process even on a large DB, so the big-bang nature of the
|
|
conversion isn't a problem. Admittedly this will not work for every
|
|
single upgrade, but we had agreed that we could schedule upgrades
|
|
so that the majority of releases do not change user data. Historically
|
|
that's been mostly true anyway, even without any deliberate attempt to
|
|
group user-data-changing features together.
|
|
|
|
I think the last major discussion about it started here:
|
|
http://archives.postgresql.org/pgsql-hackers/2003-12/msg00379.php
|
|
(I got distracted by other stuff and never did the promised work,
|
|
but I still think the approach is sound.)
|
|
|
|
regards, tom lane
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 2: you can get off all lists at once with the unregister command
|
|
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
|
|
|
|
From pgsql-hackers-owner+M59497@postgresql.org Thu Sep 30 17:44:44 2004
|
|
Return-path: <pgsql-hackers-owner+M59497@postgresql.org>
|
|
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i8ULihw11377
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 17:44:43 -0400 (EDT)
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id D0A6B329E2E
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 22:44:38 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 55636-04 for <pgman@candle.pha.pa.us>;
|
|
Thu, 30 Sep 2004 21:44:36 +0000 (GMT)
|
|
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 6ED67329DFC
|
|
for <pgman@candle.pha.pa.us>; Thu, 30 Sep 2004 22:44:38 +0100 (BST)
|
|
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [200.46.204.144])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 040D2329E2E
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Thu, 30 Sep 2004 22:42:24 +0100 (BST)
|
|
Received: from svr1.postgresql.org ([200.46.204.71])
|
|
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
|
|
with ESMTP id 55767-04
|
|
for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
|
|
Thu, 30 Sep 2004 21:42:18 +0000 (GMT)
|
|
Received: from authenticity.encs.concordia.ca (authenticity-96.encs.concordia.ca [132.205.96.93])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id E3A4D329DFC
|
|
for <pgsql-hackers@postgresql.org>; Thu, 30 Sep 2004 22:42:19 +0100 (BST)
|
|
Received: from haida.cs.concordia.ca (IDENT:mokhov@haida.cs.concordia.ca [132.205.64.45])
|
|
by authenticity.encs.concordia.ca (8.12.11/8.12.11) with ESMTP id i8ULgJrP001049;
|
|
Thu, 30 Sep 2004 17:42:19 -0400
|
|
Date: Thu, 30 Sep 2004 17:42:19 -0400 (EDT)
|
|
From: "Serguei A. Mokhov" <mokhov@cs.concordia.ca>
|
|
To: Tom Lane <tgl@sss.pgh.pa.us>
|
|
cc: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] pg_upgrade project: high-level design proposal of
|
|
In-Reply-To: <25253.1096576562@sss.pgh.pa.us>
|
|
Message-ID: <Pine.LNX.4.58.0409301725550.4685@haida.cs.concordia.ca>
|
|
References: <Pine.LNX.4.58.0409301532390.4685@haida.cs.concordia.ca>
|
|
<25253.1096576562@sss.pgh.pa.us>
|
|
MIME-Version: 1.0
|
|
Content-Type: TEXT/PLAIN; charset=US-ASCII
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Mailing-List: pgsql-hackers
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
X-Virus-Scanned: by amavisd-new at hub.org
|
|
X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on
|
|
candle.pha.pa.us
|
|
X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham
|
|
version=2.61
|
|
Status: OR
|
|
|
|
On Thu, 30 Sep 2004, Tom Lane wrote:
|
|
|
|
> Date: Thu, 30 Sep 2004 16:36:02 -0400
|
|
>
|
|
> "Serguei A. Mokhov" <mokhov@cs.concordia.ca> writes:
|
|
> > Comments are very welcome, especially _*CONSTRUCTIVE*_...
|
|
>
|
|
> This is fundamentally wrong, because you are assigning the storage
|
|
> manager functionality that it does not have. In particular, the
|
|
|
|
Maybe, that's why I was asking of all init-db forcing paths, so I can go
|
|
level above smgr to upgrade stuff, let say in access/ and other parts. I
|
|
did ask that before and never got a reply. So the concept of "Storage
|
|
Managers" may and will go well beyond the smgt API. That's the design
|
|
refinement stage.
|
|
|
|
> I don't believe in the idea of incremental "lazy" upgrades very much
|
|
> either. It certainly will not work on the system catalogs --- you have
|
|
> to convert those in a big-bang fashion,
|
|
|
|
I never proposed to do that to system catalogs, on the contrary, I said
|
|
the system catalogs are to be upgraded upon the postmaster startup. only
|
|
user relations are upgraded lazily:
|
|
|
|
> The relations to be upgraded are ordered according to some priority,
|
|
> e.g. system relations being first, then user-owned relations. System
|
|
> relations upgrade is forced upon the postmaster startup, and then user
|
|
> relations are processed lazily.
|
|
|
|
So looks like we don't disagree here :)
|
|
|
|
> The design we'd pretty much all bought into six months ago involved
|
|
> being able to do in-place upgrades when the format/contents of user
|
|
> relations and indexes is not changing. All you'd have to do is dump and
|
|
> restore the schema data (system catalogs) which is a reasonably short
|
|
> process even on a large DB, so the big-bang nature of the conversion
|
|
> isn't a problem. Admittedly this will not work for every single
|
|
> upgrade, but we had agreed that we could schedule upgrades so that the
|
|
> majority of releases do not change user data. Historically that's been
|
|
> mostly true anyway, even without any deliberate attempt to group
|
|
> user-data-changing features together.
|
|
|
|
Annoyingly enough, they still do change.
|
|
|
|
> I think the last major discussion about it started here:
|
|
> http://archives.postgresql.org/pgsql-hackers/2003-12/msg00379.php
|
|
> (I got distracted by other stuff and never did the promised work,
|
|
> but I still think the approach is sound.)
|
|
|
|
I'll go over that discussion and maybe will combine useful ideas together.
|
|
I'll open a pgfoundry project to develop it there and then will submit for
|
|
evaluation UNLESS you reserved it for yourself, Tom, to fullfill the
|
|
promise... If anybody objects the pgfoundry idea to test the concepts,
|
|
I'll apply for a project there.
|
|
|
|
Thank you for the comments!
|
|
|
|
> regards, tom lane
|
|
>
|
|
|
|
--
|
|
Serguei A. Mokhov | /~\ The ASCII
|
|
Computer Science Department | \ / Ribbon Campaign
|
|
Concordia University | X Against HTML
|
|
Montreal, Quebec, Canada | / \ Email!
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 7: don't forget to increase your free space map settings
|
|
|