postgresql/doc/TODO.detail/wal

From cjs@cynic.net Sat Jun 22 04:41:54 2002
Return-path: <cjs@cynic.net>
Received: from academic.cynic.net ([63.144.177.3])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5M8fpF04711
	for <pgman@candle.pha.pa.us>; Sat, 22 Jun 2002 04:41:53 -0400 (EDT)
Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224])
	by academic.cynic.net (Postfix) with ESMTP
	id 415C8F820; Sat, 22 Jun 2002 08:41:33 +0000 (UTC)
Date: Sat, 22 Jun 2002 17:41:30 +0900 (JST)
From: Curt Sampson <cjs@cynic.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Bruce Momjian <pgman@candle.pha.pa.us>, Michael Loftis <mloftis@wgops.com>,
   mlw <markw@mohawksoft.com>,
   PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE 
In-Reply-To: <19332.1024668861@sss.pgh.pa.us>
Message-ID: <Pine.NEB.4.43.0206221731130.1091-100000@angelic.cynic.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: RO

On Fri, 21 Jun 2002, Tom Lane wrote:

> Curt Sampson <cjs@cynic.net> writes:
> > And now you know which parts of your page got written, and which
> > parts didn't.
>
> Yes ... and what do you *do* about it?

Ok. Here's the extract from _Inside Microsoft SQL Server 7.0_, page 207:

    torn page detection   When TRUE, this option causes a bit to be
	flipped for each 512-byte sector in a database page (8 KB)
	whenever the page is written to disk.  This option allows
	SQL Server to detect incomplete I/O operations caused by
	power failures or other system outages. If a bit is in the
	wrong state when the page is later read by SQL Server, this
	means the page was written incorrectly; a torn page has
	been detected. Although SQL Server database pages are 8
	KB, disks perform I/O operations using 512-byte sectors.
	Therefore, 16 sectors are written per database page.  A
	torn page can occur if the system crashes (for example,
	because of power failure) between the time the operating
	system writes the first 512-byte sector to disk and the
	completion of the 8-KB I/O operation.  If the first sector
	of a database page is successfully written before the crash,
	it will appear that the database page on disk was updated,
	although it might not have succeeded. Using battery-backed
	disk caches can ensure that data is [sic] successfully
	written to disk or not written at all. In this case, don't
	set torn page detection to TRUE, as it isn't needed. If a
	torn page is detected, the database will need to be restored
	from backup because it will be physically inconsistent.

As I understand it, this is not a problem for postgres becuase the
entire page is written to the log. So postgres is safe, but quite
inefficient. (It would be much more efficient to write just the
changed tuple, or even just the changed values within the tuple,
to the log.)

Adding these torn bits would allow posgres at least to write to
the log just the 512-byte sectors that have changed, rather than
the entire 8 KB page.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


From pgsql-hackers-owner+M24100@postgresql.org Mon Jun 24 13:13:41 2002
Return-path: <pgsql-hackers-owner+M24100@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OHDeF08564
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 13:13:40 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id 05602475CBE; Mon, 24 Jun 2002 13:11:10 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 13:11:10 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 929A247633B; Mon, 24 Jun 2002 09:26:54 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 962C147631A
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 08:31:43 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 08:31:43 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id C112D475C3C
	for <pgsql-hackers@postgresql.org>; Sun, 23 Jun 2002 15:35:20 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5NJYtL07449;
	Sun, 23 Jun 2002 15:34:55 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206231934.g5NJYtL07449@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <1024855044.1793.414.camel@localhost.localdomain>
To: "J. R. Nield" <jrnield@usol.com>
Date: Sun, 23 Jun 2002 15:34:55 -0400 (EDT)
cc: Tom Lane <tgl@sss.pgh.pa.us>, Curt Sampson <cjs@cynic.net>,
   Michael Loftis <mloftis@wgops.com>, mlw <markw@mohawksoft.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

J. R. Nield wrote:
> So since we have all this buffering designed especially to meet our
> needs, and since the OS buffering is in the way, can someone explain to
> me why postgresql would ever open a file without the O_DSYNC flag if the
> platform supports it? 

We sync only WAL, not the other pages, except for the sync() call we do
during checkpoint when we discard old WAL files.

> > I concur with Bruce: the reason we keep page images in WAL is to
> > minimize the number of places we have to fsync, and thus the amount of
> > head movement required for a commit.  Putting the page images elsewhere
> > cannot be a win AFAICS.
> 
> 
> Why not put all the page images in a single pre-allocated file and treat
> it as a ring? How could this be any worse than flushing them in the WAL
> log? 
> 
> Maybe fsync would be slower with two files, but I don't see how
> fdatasync would be, and most platforms support that. 

We have fdatasync option for WAL in postgresql.conf.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



From pgsql-hackers-owner+M24091@postgresql.org Mon Jun 24 12:54:22 2002
Return-path: <pgsql-hackers-owner+M24091@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OGsMF07208
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 12:54:22 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id 7DB7947679D; Mon, 24 Jun 2002 09:48:51 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 09:48:51 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 3FD37476491; Mon, 24 Jun 2002 08:55:34 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 2769E4762E3
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 08:27:39 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 08:27:39 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id ED459475C61
	for <pgsql-hackers@postgresql.org>; Sun, 23 Jun 2002 15:37:08 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5NJasa07642;
	Sun, 23 Jun 2002 15:36:54 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206231936.g5NJasa07642@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <Pine.NEB.4.43.0206240307550.511-100000@angelic.cynic.net>
To: Curt Sampson <cjs@cynic.net>
Date: Sun, 23 Jun 2002 15:36:54 -0400 (EDT)
cc: "J. R. Nield" <jrnield@usol.com>, Tom Lane <tgl@sss.pgh.pa.us>,
   Michael Loftis <mloftis@wgops.com>, mlw <markw@mohawksoft.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

Curt Sampson wrote:
> On 23 Jun 2002, J. R. Nield wrote:
> 
> > So since we have all this buffering designed especially to meet our
> > needs, and since the OS buffering is in the way, can someone explain to
> > me why postgresql would ever open a file without the O_DSYNC flag if the
> > platform supports it?
> 
> It's more code, if there are platforms out there that don't support
> O_DYSNC. (We still have to keep the old fsync code.) On the other hand,
> O_DSYNC could save us a disk arm movement over fsync() because it
> appears to me that fsync is also going to force a metadata update, which
> means that the inode blocks have to be written as well.

Again, see postgresql.conf:

#wal_sync_method = fsync        # the default varies across platforms:
#                               # fsync, fdatasync, open_sync, or open_datasync

> 
> > Maybe fsync would be slower with two files, but I don't see how
> > fdatasync would be, and most platforms support that.
> 
> Because, if both files are on the same disk, you still have to move
> the disk arm from the cylinder at the current log file write point
> to the cylinder at the current ping-pong file write point. And then back
> again to the log file write point cylinder.
> 
> In the end, having a ping-pong file as well seems to me unnecessary
> complexity, especially when anyone interested in really good
> performance is going to buy a disk subsystem that guarantees no
> torn pages and thus will want to turn off the ping-pong file writes
> entirely, anyway.

Yes, I don't see writing to two files vs. one to be any win, especially
when we need to fsync both of them.  What I would really like is to
avoid the double I/O of writing to WAL and to the data file;  improving
that would be a huge win.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



From cjs@cynic.net Sun Jun 23 23:40:59 2002
Return-path: <cjs@cynic.net>
Received: from academic.cynic.net (academic.cynic.net [63.144.177.3])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5O3evF17903
	for <pgman@candle.pha.pa.us>; Sun, 23 Jun 2002 23:40:58 -0400 (EDT)
Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224])
	by academic.cynic.net (Postfix) with ESMTP
	id 37F36F820; Mon, 24 Jun 2002 03:40:54 +0000 (UTC)
Date: Mon, 24 Jun 2002 12:40:51 +0900 (JST)
From: Curt Sampson <cjs@cynic.net>
To: "J. R. Nield" <jrnield@usol.com>
cc: Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>,
   Michael Loftis <mloftis@wgops.com>, mlw <markw@mohawksoft.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <1024882167.1793.733.camel@localhost.localdomain>
Message-ID: <Pine.NEB.4.43.0206241150500.7326-100000@angelic.cynic.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: RO

On 23 Jun 2002, J. R. Nield wrote:

> If is impossible to do what you want. You can not protect against
> partial writes without writing pages twice and calling fdatasync
> between them while going through a generic filesystem.

I agree with this.

> The best disk array will not protect you if the operating system does
> not align block writes to the structure of the underlying device.

This I don't quite understand. Assuming you're using a SCSI drive
(and this mostly applies to ATAPI/IDE, too), you can do naught but
align block writes to the structure of the underlying device. When you
initiate a SCSI WRITE command, you start by telling the device at which
block to start writing and how many blocks you intend to write. Then you
start passing the data.

(See http://www.danbbs.dk/~dino/SCSI/SCSI2-09.html#9.2.21 for parameter
details for the SCSI WRITE(10) command. You may find the SCSI 2
specification, at http://www.danbbs.dk/~dino/SCSI/ to be a useful
reference here.)

> Even with raw devices, you need special support or knowledge of the
> operating system and/or the disk device to ensure that each write
> request will be atomic to the underlying hardware.

Well, so here I guess you're talking about two things:

    1. When you request, say, an 8K block write, will the OS really
    write it to disk in a single 8K or multiple of 8K SCSI write
    command?

    2. Does the SCSI device you're writing to consider these writes to
    be transactional. That is, if the write is interrupted before being
    completed, does the SCSI device guarantee that the partially-sent
    data is not written, and the old data is maintained? And of course,
    does it guarantee that, when it acknowledges a write, that write is
    now in stable storage and will never go away?

Both of these are not hard to guarantee, actually. For a BSD-based OS,
for example, just make sure that your filesystem block size is the
same as or a multiple of the database block size. BSD will never write
anything other than a block or a sequence of blocks to a disk in a
single SCSI transaction (unless you've got a really odd SCSI driver).
And for your disk, buy a Baydel or Clarion disk array, or something
similar.

Given that it's not hard to set up a system that meets these criteria,
and this is in fact commonly done for database servers, it would seem a
good idea for postgres to have the option to take advantage of the time
and money spent and adjust its performance upward appropriately.

> All other systems rely on the fact that you can recover a damaged file
> using the log archive.

Not exactly. For MS SQL Server, at any rate, if it detects a page tear
you cannot restore based on the log file alone. You need a full or
partial backup that includes that entire torn block.

> This means downtime in the rare case, but no data loss. Until
> PostgreSQL can do this, then it will not be acceptable for real
> critical production use.

It seems to me that it is doing this right now. In fact, it's more
reliable than some commerial systems (such as SQL Server) because it can
recover from a torn block with just the logfile.

> But at the end of the day, unless you have complete understanding of
> the I/O system from write(2) through to the disk system, the only sure
> ways to protect against partial writes are by "careful writes" (in
> the WAL log or elsewhere, writing pages twice), or by requiring (and
> allowing) users to do log-replay recovery when a file is corrupted by
> a partial write.

I don't understand how, without a copy of the old data that was in the
torn block, you can restore that block from just log file entries. Can
you explain this to me? Take, as an example, a block with ten tuples,
only one of which has been changed "recently." (I.e., only that change
is in the log files.)

> If we log pages to WAL, they are useless when archived (after a
> checkpoint). So either we have a separate "log" for them (the
> ping-pong file), or we should at least remove them when archived,
> which makes log archiving more complex but is perfectly doable.

Right. That seems to me a better option, since we've now got only one
write point on the disk rather than two.

> Finally, I would love to hear why we are using the operating system
> buffer manager at all. The OS is acting as a secondary buffer manager
> for us. Why is that? What flaw in our I/O system does this reveal?

It's acting as a "second-level" buffer manager, yes, but to say it's
"secondary" may be a bit misleading. On most of the systems I've set
up, the OS buffer cache is doing the vast majority of the work, and the
postgres buffering is fairly minimal.

There are some good (and some perhaps not-so-good) reasons to do it this
way. I'll list them more or less in the order of best to worst:

    1. The OS knows where the blocks physically reside on disk, and
    postgres does not. Therefore it's in the interest of postgresql to
    dispatch write responsibility back to the OS as quickly as possible
    so that the OS can prioritize requests appropriately. Most operating
    systems use an "elevator" algorithm to minimize disk head movement;
    but if the OS does not have a block that it could write while the
    head is "on the way" to another request, it can't write it in that
    head pass.

    2. Postgres does not know about any "bank-switching" tricks for
    mapping more physical memory than it has address space. Thus, on
    32-bit machines, postgres might be limited to mapping 2 or 3 GB of
    memory, even though the machine has, say, 6 GB of physical RAM. The
    OS can use all of the available memory for caching; postgres cannot.

    3. A lot of work has been put into the seek algorithms, read-ahead
    algorithms, block allocation algorithms, etc. in the OS. Why
    duplicate all that work again in postgres?

When you say things like the following:

> We should only be writing blocks when they need to be on disk. We
> should not be expecting the OS to write them "sometime later" and
> avoid blocking (as long) for the write. If we need that, then our
> buffer management is wrong and we need to fix it.

you appear to be making the arugment that we should take the route of
other database systems, and use raw devices and our own management of
disk block allocation. If so, you might want first to look back through
the archives at the discussion I and several others had about this a
month or two ago. After looking in detail at what NetBSD, at least, does
in terms of its disk I/O algorithms and buffering, I've pretty much come
around, at least for the moment, to the attitude that we should stick
with using the OS. I wouldn't mind seeing postgres be able to manage all
of this stuff, but it's a *lot* of work for not all that much benefit
that I can see.

> The ORACLE people were not kidding when they said that they could not
> certify Linux for production use until it supported O_DSYNC. Can you
> explain why that was the case?

I'm suspecting it's because Linux at the time had no raw devices, so
O_DSYNC was the only other possible method of making sure that disk
writes actually got to disk.

You certainly don't want to use O_DSYNC if you can use another method,
because O_DSYNC still goes through the the operating system's buffer
cache, wasting memory and double-caching things. If you're doing your
own management, you need either to use a raw device or open files with
the flag that indicates that the buffer cache should not be used at all
for reads from and writes to that file.

> However, this discussion and a search of the pgsql-hackers archives
> reveals this problem to be the KEY area of PostgreSQL's failing, and
> general misunderstanding, when compared to its commercial competitors.

No, I think it's just that you're under a few minor misapprehensions
here about what postgres and the OS are actually doing. As I said, I
went through this whole exact argument a month or two ago, on this very
list, and I came around to the idea that what postgres is doing now
works quite well, at least on NetBSD. (Most other OSes have disk I/O
algorithms that are pretty much as good or better.) There might be a
very slight advantage to doing all one's own I/O management, but it's
a huge amount of work, and I think that much effort could be much more
usefully applied to other areas.

Just as a side note, I've been a NetBSD developer since about '96,
and have been delving into the details of OS design since well before
that time, so I'm coming to this with what I hope is reasonably good
knowledge of how disks work and how operating systems use them. (Not
that this should stop you from pointing out holes in my arguments. :-))

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


From pgsql-hackers-owner+M24112@postgresql.org Mon Jun 24 18:16:36 2002
Return-path: <pgsql-hackers-owner+M24112@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OMGaF00910
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 18:16:36 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id A2EF1476475; Mon, 24 Jun 2002 16:43:38 -0400 (EDT)
Mailbox-Line: From tgl@sss.pgh.pa.us  Mon Jun 24 16:43:38 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id BA57D476148; Mon, 24 Jun 2002 14:14:00 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 93D6A477214
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 13:59:17 -0400 (EDT)
Mailbox-Line: From tgl@sss.pgh.pa.us  Mon Jun 24 13:59:17 2002
Received: from sss.pgh.pa.us (unknown [192.204.191.242])
	by postgresql.org (Postfix) with ESMTP id D70AA476401
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 10:06:26 -0400 (EDT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OE6J117666;
	Mon, 24 Jun 2002 10:06:19 -0400 (EDT)
To: Curt Sampson <cjs@cynic.net>
cc: Bruce Momjian <pgman@candle.pha.pa.us>, "J. R. Nield" <jrnield@usol.com>,
   Michael Loftis <mloftis@wgops.com>, mlw <markw@mohawksoft.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE 
In-Reply-To: <Pine.NEB.4.43.0206240907160.511-100000@angelic.cynic.net> 
References: <Pine.NEB.4.43.0206240907160.511-100000@angelic.cynic.net>
Comments: In-reply-to Curt Sampson <cjs@cynic.net>
	message dated "Mon, 24 Jun 2002 09:09:30 +0900"
Date: Mon, 24 Jun 2002 10:06:19 -0400
Message-ID: <17663.1024927579@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-5.3 required=5.0
	tests=IN_REP_TO,X_NOT_PRESENT
	version=2.30
Status: RO

> On Sun, 23 Jun 2002, Bruce Momjian wrote:
>> Yes, I don't see writing to two files vs. one to be any win, especially
>> when we need to fsync both of them.  What I would really like is to
>> avoid the double I/O of writing to WAL and to the data file;  improving
>> that would be a huge win.

I don't believe it's possible to eliminate the double I/O.  Keep in mind
though that in the ideal case (plenty of shared buffers) you are only
paying two writes per modified block per checkpoint interval --- one to
the WAL during the first write of the interval, and then a write to the
real datafile issued by the checkpoint process.  Anything that requires
transaction commits to write data blocks will likely result in more I/O
not less, at least for blocks that are modified by several successive
transactions.

The only thing I've been able to think of that seems like it might
improve matters is to make the WAL writing logic aware of the layout
of buffer pages --- specifically, to know that our pages generally
contain an uninteresting "hole" in the middle, and not write the hole.
Optimistically this might reduce the WAL data volume by something
approaching 50%; though pessimistically (if most pages are near full)
it wouldn't help much.

This was not very feasible when the WAL code was designed because the
buffer manager needed to cope with both normal pages and pg_log pages,
but as of 7.2 I think it'd be safe to assume that all pages have the
standard layout.

			regards, tom lane



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)



From pgsql-hackers-owner+M24116@postgresql.org Mon Jun 24 20:32:07 2002
Return-path: <pgsql-hackers-owner+M24116@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P0W7F10985
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 20:32:07 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id EBCE547632E; Mon, 24 Jun 2002 18:54:34 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 18:54:34 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 3EB93476D85; Mon, 24 Jun 2002 17:12:18 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id EBC20476E2E
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 14:54:40 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 14:54:40 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id 1C8874760C2
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 12:40:53 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5OGeVY06116;
	Mon, 24 Jun 2002 12:40:31 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206241640.g5OGeVY06116@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <17663.1024927579@sss.pgh.pa.us>
To: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 24 Jun 2002 12:40:31 -0400 (EDT)
cc: Curt Sampson <cjs@cynic.net>, "J. R. Nield" <jrnield@usol.com>,
   Michael Loftis <mloftis@wgops.com>, mlw <markw@mohawksoft.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

Tom Lane wrote:
> > On Sun, 23 Jun 2002, Bruce Momjian wrote:
> >> Yes, I don't see writing to two files vs. one to be any win, especially
> >> when we need to fsync both of them.  What I would really like is to
> >> avoid the double I/O of writing to WAL and to the data file;  improving
> >> that would be a huge win.
> 
> I don't believe it's possible to eliminate the double I/O.  Keep in mind
> though that in the ideal case (plenty of shared buffers) you are only
> paying two writes per modified block per checkpoint interval --- one to
> the WAL during the first write of the interval, and then a write to the
> real datafile issued by the checkpoint process.  Anything that requires
> transaction commits to write data blocks will likely result in more I/O
> not less, at least for blocks that are modified by several successive
> transactions.
> 
> The only thing I've been able to think of that seems like it might
> improve matters is to make the WAL writing logic aware of the layout
> of buffer pages --- specifically, to know that our pages generally
> contain an uninteresting "hole" in the middle, and not write the hole.
> Optimistically this might reduce the WAL data volume by something
> approaching 50%; though pessimistically (if most pages are near full)
> it wouldn't help much.

Good idea.  How about putting the page through or TOAST compression
routine before writing it to WAL?  Should be pretty easy and fast and
doesn't require any knowledge of the page format.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org



From pgsql-hackers-owner+M24128@postgresql.org Mon Jun 24 22:01:58 2002
Return-path: <pgsql-hackers-owner+M24128@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P21vF19918
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 22:01:57 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id 540B8475B33; Mon, 24 Jun 2002 21:34:40 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 21:34:40 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 0A13F476965; Mon, 24 Jun 2002 19:30:14 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id B4F62476E4A
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 18:53:59 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 18:53:59 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id 36043475BF6
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 17:25:28 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5OLPFG26140;
	Mon, 24 Jun 2002 17:25:15 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206242125.g5OLPFG26140@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <1024951786.1793.865.camel@localhost.localdomain>
To: "J. R. Nield" <jrnield@usol.com>
Date: Mon, 24 Jun 2002 17:25:14 -0400 (EDT)
cc: Curt Sampson <cjs@cynic.net>, Tom Lane <tgl@sss.pgh.pa.us>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

J. R. Nield wrote:
> > This I don't quite understand. Assuming you're using a SCSI drive
> > (and this mostly applies to ATAPI/IDE, too), you can do naught but
> > align block writes to the structure of the underlying device. When you
> > initiate a SCSI WRITE command, you start by telling the device at which
> > block to start writing and how many blocks you intend to write. Then you
> > start passing the data.
> > 
> 
> All I'm saying is that the entire postgresql block write must be
> converted into exactly one SCSI write command in all cases, and I don't
> know a portable way to ensure this. 

...

> I agree with this. My point was only that you need to know what
> guarantees your operating system/hardware combination provides on a
> case-by-case basis, and there is no standard way for a program to
> discover this. Most system administrators are not going to know this
> either, unless databases are their main responsibility.

Yes, agreed.  >1% are going to know the answer to this question so we
have to assume worst case.

> > It seems to me that it is doing this right now. In fact, it's more
> > reliable than some commerial systems (such as SQL Server) because it can
> > recover from a torn block with just the logfile.
> 
> Again, what I meant to say is that the commercial systems can recover
> with an old file backup + logs. How old the backup can be depends only
> on how much time you are willing to spend playing the logs forward. So
> if you do a full backup once a week, and multiplex and backup the logs,
> then even if a backup tape gets destroyed you can still survive. It just
> takes longer.
> 
> Also, postgreSQL can't recover from any other type of block corruption,
> while the commercial systems can. That's what I meant by the "critical
> production use" comment, which was sort-of unfair.
> 
> So I would say they are equally reliable for torn pages (but not bad
> blocks), and the commercial systems let you trade potential recovery
> time for not having to write the blocks twice. You do need to back-up
> the log archives though.

Yes, good tradeoff analysis.  We recover from partial writes quicker,
and don't require saving of log files, _but_ we don't recover from bad
disk blocks.  Good summary.

> I'll back off on that. I don't know if we want to use the OS buffer
> manager, but shouldn't we try to have our buffer manager group writes
> together by files, and pro-actively get them out to disk? Right now, it
> looks like all our write requests are delayed as long as possible and
> the order in which they are written is pretty-much random, as is the
> backend that writes the block, so there is no locality of reference even
> when the blocks are adjacent on disk, and the write calls are spread-out
> over all the backends.
> 
> Would it not be the case that things like read-ahead, grouping writes,
> and caching written data are probably best done by PostgreSQL, because
> only our buffer manager can understand when they will be useful or when
> they will thrash the cache?

The OS should handle all of this.  We are doing main table writes but no
sync until checkpoint, so the OS can keep those blocks around and write
them at its convenience.  It knows the size of the buffer cache and when
stuff is forced to disk.  We can't second-guess that.

> I may likely be wrong on this, and I haven't done any performance
> testing. I shouldn't have brought this up alongside the logging issues,
> but there seemed to be some question about whether the OS was actually
> doing all these things behind the scene.

It had better.  Looking at the kernel source is the way to know.

> Does anyone know what the major barriers to infinite log replay are in
> PostgreSQL? I'm trying to look for everything that might need to be
> changed outside xlog.c, but surely this has come up before. Searching
> the archives hasn't revealed much.

This has been brought up.  Could we just save WAL files and get replay? 
I believe some things have to be added to WAL to allow this, but it
seems possible.  However, the pg_dump is just a data dump and does not
have the file offsets and things.  Somehow you would need a tar-type
backup of the database, and with a running db, it is hard to get a valid
snapshot of that.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly



From tgl@sss.pgh.pa.us Mon Jun 24 17:31:57 2002
Return-path: <tgl@sss.pgh.pa.us>
Received: from sss.pgh.pa.us (root@[192.204.191.242])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OLVuF26684
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 17:31:56 -0400 (EDT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OLVu121485;
	Mon, 24 Jun 2002 17:31:56 -0400 (EDT)
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "J. R. Nield" <jrnield@usol.com>, Curt Sampson <cjs@cynic.net>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE 
In-Reply-To: <200206242125.g5OLPFG26140@candle.pha.pa.us> 
References: <200206242125.g5OLPFG26140@candle.pha.pa.us>
Comments: In-reply-to Bruce Momjian <pgman@candle.pha.pa.us>
	message dated "Mon, 24 Jun 2002 17:25:14 -0400"
Date: Mon, 24 Jun 2002 17:31:56 -0400
Message-ID: <21482.1024954316@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
Status: ROr

Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Does anyone know what the major barriers to infinite log replay are in
>> PostgreSQL? I'm trying to look for everything that might need to be
>> changed outside xlog.c, but surely this has come up before. Searching
>> the archives hasn't revealed much.

> This has been brought up.  Could we just save WAL files and get replay? 
> I believe some things have to be added to WAL to allow this, but it
> seems possible.

The Red Hat group has been looking at this somewhat; so far there seem
to be some minor tweaks that would be needed, but no showstoppers.

> Somehow you would need a tar-type
> backup of the database, and with a running db, it is hard to get a valid
> snapshot of that.

But you don't *need* a "valid snapshot", only a correct copy of
every block older than the first checkpoint in your WAL log series.
Any inconsistencies in your tar dump will look like repairable damage;
replaying the WAL log will fix 'em.

			regards, tom lane

From pgsql-hackers-owner+M24131@postgresql.org Mon Jun 24 21:15:06 2002
Return-path: <pgsql-hackers-owner+M24131@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P1F5F15390
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 21:15:05 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id B76174768CC; Mon, 24 Jun 2002 20:59:56 -0400 (EDT)
Mailbox-Line: From tgl@sss.pgh.pa.us  Mon Jun 24 20:59:56 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 8724C47742E; Mon, 24 Jun 2002 20:17:44 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 4E472476875
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 18:37:46 -0400 (EDT)
Mailbox-Line: From tgl@sss.pgh.pa.us  Mon Jun 24 18:37:46 2002
Received: from sss.pgh.pa.us (unknown [192.204.191.242])
	by postgresql.org (Postfix) with ESMTP id CFCC9476A7A
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 17:32:02 -0400 (EDT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OLVu121485;
	Mon, 24 Jun 2002 17:31:56 -0400 (EDT)
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "J. R. Nield" <jrnield@usol.com>, Curt Sampson <cjs@cynic.net>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE 
In-Reply-To: <200206242125.g5OLPFG26140@candle.pha.pa.us> 
References: <200206242125.g5OLPFG26140@candle.pha.pa.us>
Comments: In-reply-to Bruce Momjian <pgman@candle.pha.pa.us>
	message dated "Mon, 24 Jun 2002 17:25:14 -0400"
Date: Mon, 24 Jun 2002 17:31:56 -0400
Message-ID: <21482.1024954316@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-5.3 required=5.0
	tests=IN_REP_TO,X_NOT_PRESENT
	version=2.30
Status: RO

Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Does anyone know what the major barriers to infinite log replay are in
>> PostgreSQL? I'm trying to look for everything that might need to be
>> changed outside xlog.c, but surely this has come up before. Searching
>> the archives hasn't revealed much.

> This has been brought up.  Could we just save WAL files and get replay? 
> I believe some things have to be added to WAL to allow this, but it
> seems possible.

The Red Hat group has been looking at this somewhat; so far there seem
to be some minor tweaks that would be needed, but no showstoppers.

> Somehow you would need a tar-type
> backup of the database, and with a running db, it is hard to get a valid
> snapshot of that.

But you don't *need* a "valid snapshot", only a correct copy of
every block older than the first checkpoint in your WAL log series.
Any inconsistencies in your tar dump will look like repairable damage;
replaying the WAL log will fix 'em.

			regards, tom lane



---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org



From pgsql-hackers-owner+M24133@postgresql.org Mon Jun 24 22:19:55 2002
Return-path: <pgsql-hackers-owner+M24133@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P2JsF21543
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 22:19:54 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id 42391476E53; Mon, 24 Jun 2002 22:09:49 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 22:09:49 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 191654774EB; Mon, 24 Jun 2002 20:26:08 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 8EB90476101
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 19:43:19 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Mon Jun 24 19:43:19 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id 08018476931
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 17:33:53 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5OLXhl26908;
	Mon, 24 Jun 2002 17:33:43 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206242133.g5OLXhl26908@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <21482.1024954316@sss.pgh.pa.us>
To: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 24 Jun 2002 17:33:43 -0400 (EDT)
cc: "J. R. Nield" <jrnield@usol.com>, Curt Sampson <cjs@cynic.net>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Does anyone know what the major barriers to infinite log replay are in
> >> PostgreSQL? I'm trying to look for everything that might need to be
> >> changed outside xlog.c, but surely this has come up before. Searching
> >> the archives hasn't revealed much.
> 
> > This has been brought up.  Could we just save WAL files and get replay? 
> > I believe some things have to be added to WAL to allow this, but it
> > seems possible.
> 
> The Red Hat group has been looking at this somewhat; so far there seem
> to be some minor tweaks that would be needed, but no showstoppers.


Good.

> > Somehow you would need a tar-type
> > backup of the database, and with a running db, it is hard to get a valid
> > snapshot of that.
> 
> But you don't *need* a "valid snapshot", only a correct copy of
> every block older than the first checkpoint in your WAL log series.
> Any inconsistencies in your tar dump will look like repairable damage;
> replaying the WAL log will fix 'em.

Yes, my point was that you need physical file backups, not pg_dump, and
you have to be tricky about the files changing during the backup.  You
_can_ work around changes to the files during backup.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
/usr/local/bin/mime: cannot create /dev/ttyp3: permission denied



---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org



From pgsql-hackers-owner+M24139@postgresql.org Tue Jun 25 00:00:22 2002
Return-path: <pgsql-hackers-owner+M24139@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P40LF00838
	for <pgman@candle.pha.pa.us>; Tue, 25 Jun 2002 00:00:21 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id CBAE8476E94; Mon, 24 Jun 2002 23:44:51 -0400 (EDT)
Mailbox-Line: From jrnield@usol.com  Mon Jun 24 23:44:51 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id C5076476871; Mon, 24 Jun 2002 22:25:46 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 8DF57476979
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 22:08:31 -0400 (EDT)
Mailbox-Line: From jrnield@usol.com  Mon Jun 24 22:08:31 2002
Received: from hades.usol.com (hades.usol.com [208.232.58.41])
	by postgresql.org (Postfix) with ESMTP id 298D2476101
	for <pgsql-hackers@postgresql.org>; Mon, 24 Jun 2002 20:27:46 -0400 (EDT)
Received: from 08-159.024.popsite.net (08-159.024.popsite.net [66.19.4.159])
	by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5P0RbV01261;
	Mon, 24 Jun 2002 20:27:37 -0400
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
From: "J. R. Nield" <jrnield@usol.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Curt Sampson <cjs@cynic.net>, Bruce Momjian <pgman@candle.pha.pa.us>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
In-Reply-To: <21376.1024953361@sss.pgh.pa.us>
References: <Pine.NEB.4.43.0206241150500.7326-100000@angelic.cynic.net>
	<1024951786.1793.865.camel@localhost.localdomain> 
	<21376.1024953361@sss.pgh.pa.us>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) 
Date: 24 Jun 2002 20:28:00 -0400
Message-ID: <1024964884.3031.876.camel@localhost.localdomain>
MIME-Version: 1.0
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

On Mon, 2002-06-24 at 17:16, Tom Lane wrote:
 
> I think you have been missing the point...  
Yes, this appears to be the case. Thanks especially to Curt for clearing
things up for me.

-- 
J. R. Nield
jrnield@usol.com






---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html



From jrnield@usol.com Mon Jun 24 20:27:45 2002
Return-path: <jrnield@usol.com>
Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P0RhF10711
	for <pgman@candle.pha.pa.us>; Mon, 24 Jun 2002 20:27:44 -0400 (EDT)
Received: from 08-159.024.popsite.net (08-159.024.popsite.net [66.19.4.159])
	by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5P0RbV01261;
	Mon, 24 Jun 2002 20:27:37 -0400
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
From: "J. R. Nield" <jrnield@usol.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Curt Sampson <cjs@cynic.net>, Bruce Momjian <pgman@candle.pha.pa.us>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
In-Reply-To: <21376.1024953361@sss.pgh.pa.us>
References: <Pine.NEB.4.43.0206241150500.7326-100000@angelic.cynic.net>
	<1024951786.1793.865.camel@localhost.localdomain> 
	<21376.1024953361@sss.pgh.pa.us>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) 
Date: 24 Jun 2002 20:28:00 -0400
Message-ID: <1024964884.3031.876.camel@localhost.localdomain>
MIME-Version: 1.0
Status: RO

On Mon, 2002-06-24 at 17:16, Tom Lane wrote:
 
> I think you have been missing the point...  
Yes, this appears to be the case. Thanks especially to Curt for clearing
things up for me.

-- 
J. R. Nield
jrnield@usol.com




From cjs@cynic.net Tue Jun 25 01:09:02 2002
Return-path: <cjs@cynic.net>
Received: from academic.cynic.net (academic.cynic.net [63.144.177.3])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P591F07292
	for <pgman@candle.pha.pa.us>; Tue, 25 Jun 2002 01:09:01 -0400 (EDT)
Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224])
	by academic.cynic.net (Postfix) with ESMTP
	id 517BEF820; Tue, 25 Jun 2002 05:09:02 +0000 (UTC)
Date: Tue, 25 Jun 2002 14:08:59 +0900 (JST)
From: Curt Sampson <cjs@cynic.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: "J. R. Nield" <jrnield@usol.com>, Bruce Momjian <pgman@candle.pha.pa.us>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE 
In-Reply-To: <21376.1024953361@sss.pgh.pa.us>
Message-ID: <Pine.NEB.4.43.0206251406390.17448-100000@angelic.cynic.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: ROr

On Mon, 24 Jun 2002, Tom Lane wrote:

> There are a lot of other things we desperately need to spend time
> on that would not amount to re-engineering large quantities of OS-level
> code.  Given that most Unixen have perfectly respectable disk management
> subsystems, we prefer to tune our code to make use of that stuff, rather
> than follow the "conventional wisdom" that databases need to bypass it.
> ...
> Oracle can afford to do that sort of thing because they have umpteen
> thousand developers available.  Postgres does not.

Well, Oracle also started out, a long long time ago, on systems without
unified buffer cache and so on, and so they *had* to write this stuff
because otherwise data would not be cached. So Oracle can also afford to
maintain it now because the code already exists.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


From pgsql-hackers-owner+M24154@postgresql.org Tue Jun 25 09:22:38 2002
Return-path: <pgsql-hackers-owner+M24154@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5PDMbF03932
	for <pgman@candle.pha.pa.us>; Tue, 25 Jun 2002 09:22:37 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP
	id C12C3475E4A; Tue, 25 Jun 2002 09:22:32 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Tue Jun 25 09:22:32 2002
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 65471475C7A; Tue, 25 Jun 2002 09:22:23 -0400 (EDT)
Received: from localhost.localdomain (postgresql.org [64.49.215.8])
	by localhost (Postfix) with ESMTP id 97C8C475A7C
	for <pgsql-hackers@postgresql.org>; Tue, 25 Jun 2002 09:22:20 -0400 (EDT)
Mailbox-Line: From pgman@candle.pha.pa.us  Tue Jun 25 09:22:20 2002
Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35])
	by postgresql.org (Postfix) with ESMTP id 42C0B475A64
	for <pgsql-hackers@postgresql.org>; Tue, 25 Jun 2002 09:22:19 -0400 (EDT)
Received: (from pgman@localhost)
	by candle.pha.pa.us (8.11.6/8.10.1) id g5PDM5B03772;
	Tue, 25 Jun 2002 09:22:05 -0400 (EDT)
From: Bruce Momjian <pgman@candle.pha.pa.us>
Message-ID: <200206251322.g5PDM5B03772@candle.pha.pa.us>
Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE
In-Reply-To: <Pine.NEB.4.43.0206251406390.17448-100000@angelic.cynic.net>
To: Curt Sampson <cjs@cynic.net>
Date: Tue, 25 Jun 2002 09:22:05 -0400 (EDT)
cc: Tom Lane <tgl@sss.pgh.pa.us>, "J. R. Nield" <jrnield@usol.com>,
   PostgreSQL Hacker <pgsql-hackers@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL97 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Spam-Status: No, hits=-3.4 required=5.0
	tests=IN_REP_TO
	version=2.30
Status: RO

Curt Sampson wrote:
> On Mon, 24 Jun 2002, Tom Lane wrote:
> 
> > There are a lot of other things we desperately need to spend time
> > on that would not amount to re-engineering large quantities of OS-level
> > code.  Given that most Unixen have perfectly respectable disk management
> > subsystems, we prefer to tune our code to make use of that stuff, rather
> > than follow the "conventional wisdom" that databases need to bypass it.
> > ...
> > Oracle can afford to do that sort of thing because they have umpteen
> > thousand developers available.  Postgres does not.
> 
> Well, Oracle also started out, a long long time ago, on systems without
> unified buffer cache and so on, and so they *had* to write this stuff
> because otherwise data would not be cached. So Oracle can also afford to
> maintain it now because the code already exists.

Well, actually, it isn't unified buffer cache that is the issue, but
rather the older SysV file system had pretty poor performance so
bypassing it was a bigger win that it is today.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



From pgsql-hackers-owner+M31893@postgresql.org Fri Nov 15 11:25:58 2002
Return-path: <pgsql-hackers-owner+M31893@postgresql.org>
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id gAFHPvR10276
	for <pgman@candle.pha.pa.us>; Fri, 15 Nov 2002 12:25:57 -0500 (EST)
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP
	id A2D5A4774A1; Fri, 15 Nov 2002 11:34:54 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with SMTP
	id 5E898477132; Fri, 15 Nov 2002 11:15:45 -0500 (EST)
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP id 90CF1475B85
	for <pgsql-hackers@postgresql.org>; Mon, 11 Nov 2002 15:33:47 -0500 (EST)
Received: from Curtis-Vaio (unknown [63.164.0.45])
	by postgresql.org (Postfix) with SMTP id C6CB1475A3F
	for <pgsql-hackers@postgresql.org>; Mon, 11 Nov 2002 15:33:46 -0500 (EST)
Received: from [127.0.0.1] by Curtis-Vaio
  (ArGoSoft Mail Server Freeware, Version 1.8 (1.8.1.7)); Mon, 11 Nov 2002 16:33:42 -0400
From: "Curtis Faith" <curtis@galtcapital.com>
To: <pgsql-hackers@postgresql.org>
Subject: [HACKERS] 500 tpsQL + WAL log implementation
Date: Mon, 11 Nov 2002 16:33:41 -0400
Message-ID: <DMEEJMCDOJAKPPFACMPMCEBMCFAA.curtis@galtcapital.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700
Importance: Normal
X-Virus-Scanned: by AMaViS new-20020517
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Virus-Scanned: by AMaViS new-20020517
Status: ROr

I have been experimenting with empirical tests of file system and device
level writes to determine the actual constraints in order to speed up the WAL
logging code.

Using a raw file partition and a time-based technique for determining the
optimal write position, I am able to get 8K writes physically written to disk
synchronously in the range of 500 to 650 writes per second using FreeBSD raw
device partitions on IDE disks (with write cache disabled).  I will be
testing it soon under linux with 10,00RPM SCSI which should be even better.
It is my belief that the mechanism used to achieve these speeds could be
incorporated into the existing WAL logging code as an abstraction that looks
to the WAL code just like the file level access currently used. The current
speeds are limited by the speed of a single disk rotation. For a 7,200 RPM
disk this is 120/second, for a 10,000 RPM disk this is 166.66/second

The mechanism works by adjusting the seek offset of the write by using
gettimeofday to determine approximately where the disk head is in its
rotation. The mechanism does not use any AIO calls.

Assuming the following:

1) Disk rotation time is 8.333ms or 8333us (7200 RPM).

2) A write at offset 1,500K completes at system time 103s 000ms 000us

3) A new write is requested at system time 103s 004ms 166us

4) A 390K per rotation alignment of the data on the disk.

5) A write must be sent at least 20K ahead of the current head position to
ensure that it is written in less than one rotation.

It can be determined from the above that a write for an offset of something
slightly more than 195K past the last write, or offset 1,695K will be ahead
of the current location of the head and will therefore complete in less than
a single rotation's time.

The disk specific metrics (rotation speed, bytes per rotation, base write
time, etc.) can be derived empirically through a tester program that would
take a few minutes to run and which could be run at log setup time.

The obvious problem with the above mechanism is that the WAL log needs to be
able to read from the log file in transaction order during recovery. This
could be provided for using an abstraction that prepends the logical order
for each block written to the disk and makes sure that the log blocks contain
either a valid logical order number or some other marker indicating that the
block is not being used.

A bitmap of blocks that have already been used would be kept in memory for
quickly determining the next set of possible unused blocks but this bitmap
would not need to be written to disk except during normal shutdown since in
the even of a failure the bitmaps would be reconstructed by reading all the
blocks from the disk.

Checkpointing and something akin to log rotation could be handled using this
mechanism as well.

So, MY REAL QUESTION is whether or not this is the sort of speed improvement
that warrants the work of writing the required abstraction layer and making
this very robust. The WAL code should remain essentially unchanged, with
perhaps new calls for the five or six routines used to access the log files,
and handle the equivalent of log rotation for raw device access. These new
calls would either use the current file based implementation or the new
logging mechanism depending on the configuration.

I anticipate that the extra work required for a PostgreSQL administrator to
use the proposed logging mechanism would be to:

1) Create a raw device partition of the appropriate size
2) Run the metrics tester for that device partition
3) Set the appropriate configuration parameters to indicate raw WAL logging

I anticipate that the additional space requirements for this system would be
on the order of 10% to 15% beyond the current file-based implementation's
requirements.

So, is this worth doing? Would a robust implementation likely be accepted for
7.4 assuming it can demonstrate speed improvements in the range of 500tps?

- Curtis

















---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

On Sun, 2005-04-10 at 21:12 -0400, Bruce Momjian wrote:
> Jim C. Nasby wrote:
> > Maybe better for -hackers, but here it goes anyway...
> > 
> > Has anyone looked at compressing WAL's before writing to disk? On a
> > system generating a lot of WAL it seems there might be some gains to be
> > had WAL data could be compressed before going to disk, since today's
> > machines are generally more I/O bound than CPU bound. And unlike the
> > base tables, you generally don't need to read the WAL, so you don't
> > really need to worry about not being able to quickly scan through the
> > data without decompressing it.
> 
> I have never heard anyone talk about it, but it seems useful.  I think
> compressing the page images written on first page modification since
> checkpoint would be a big win.

Well it was discussed 2-3 years ago as part of the PITR preamble. You
may be surprised to read that over...

A summary of thoughts to date on this are:

xlog.c XLogInsert places backup blocks into the wal buffers before
insertion, so is the right place to do this. It would be possible to do
this before any LWlocks are taken, so would not not necessarily impair
scalability.

Currently XLogInsert is a severe CPU bottleneck around the CRC
calculation, as identified recently by Tom. Digging further, the code
used seems to cause processor stalls on Intel CPUs, possibly responsible
for much of the CPU time. Discussions to move to a 32-bit CRC would also
be effected by this because of the byte-by-byte nature of the algorithm,
whatever the length of the generating polynomial. PostgreSQL's CRC
algorithm is the fastest BSD code available. Until improvement is made
there, I would not investigate compression further. Some input from
hardware tuning specialists is required...

The current LZW compression code uses a 4096 byte lookback size, so that
would need to be modified to extend across a whole block. An
alternative, suggested originally by Tom and rediscovered by me because
I just don't read everybody's fine words in history, is to simply take
out the freespace in the middle of every heap block that consists of
zeros.

Any solution in this area must take into account the variability of the
size of freespace in database blocks. Some databases have mostly full
blocks, others vary. There would also be considerable variation in
compressability of blocks, especially since some blocks (e.g. TOAST) are
likely to already be compressed. There'd need to be some testing done to
see exactly the point where the costs of compression produce realisable
benefits.

So any solution must be able to cope with both compressed blocks and
non-compressed blocks. My current thinking is that this could be
achieved by using the spare fourth bit of the BkpBlocks portion of the
XLog structure, so that either all included BkpBlocks are compressed or
none of them are, and hope that allows benefit to shine through. Not
thought about heap/index issues.

It is possible that an XLogWriter process could be used to assist in the
CRC and compression calculations also, an a similar process used to
assist decompression for recovery, in time.

I regret I do not currently have time to pursue further.

Best Regards, Simon Riggs

From pgsql-hackers-owner+M65147=pgman=candle.pha.pa.us@postgresql.org Fri Mar 11 12:35:29 2005
Return-path: <pgsql-hackers-owner+M65147=pgman=candle.pha.pa.us@postgresql.org>
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j2BIZSk26269
	for <pgman@candle.pha.pa.us>; Fri, 11 Mar 2005 13:35:29 -0500 (EST)
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 7C7DD545B4
	for <pgman@candle.pha.pa.us>; Fri, 11 Mar 2005 18:35:25 +0000 (GMT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 31351-02 for <pgman@candle.pha.pa.us>;
	Fri, 11 Mar 2005 18:35:25 +0000 (GMT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by svr1.postgresql.org (Postfix) with ESMTP id D316E545A2
	for <pgman@candle.pha.pa.us>; Fri, 11 Mar 2005 18:35:24 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id E70B65640C
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Fri, 11 Mar 2005 18:33:21 +0000 (GMT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 29957-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Fri, 11 Mar 2005 18:33:19 +0000 (GMT)
Received: from hal.kabsi.at (bottom.kabsi.at [195.202.128.74])
	by svr1.postgresql.org (Postfix) with ESMTP id 9D9EE545EE
	for <pgsql-hackers@postgresql.org>; Fri, 11 Mar 2005 18:33:17 +0000 (GMT)
Received: from [192.168.0.5] (h062040243020.plc.cm.kabsi.at [62.40.243.20])
	by hal.kabsi.at (8.11.1/) with ESMTP id j2BIX7a0000748810;
	Fri, 11 Mar 2005 19:33:07 +0100 (CET)
Message-ID: <4231E416.4030900@cybertec.at>
Date: Fri, 11 Mar 2005 19:31:50 +0100
From: =?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= <postgres@cybertec.at>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20040922
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Simon Riggs <simon@2ndquadrant.com>
cc: Tom Lane <tgl@sss.pgh.pa.us>,
   Mark Cave-Ayland <m.cave-ayland@webbased.co.uk>,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
References: <9EB50F1A91413F4FA63019487FCD251D113169@WEBBASEDDC.webbasedltd.local>	 <23031.1110206390@sss.pgh.pa.us> <1110239639.6117.197.camel@localhost.localdomain>
In-Reply-To: <1110239639.6117.197.camel@localhost.localdomain>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by amavisd-new at hub.org
X-Mailing-List: pgsql-hackers
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
X-Virus-Scanned: by amavisd-new at hub.org
Status: RO


> One of the things I was thinking about was whether we could use up those
> cycles more effectively. If we were to include a compression routine
> before we calculated the CRC that would 
> - reduce the size of the blocks to be written, hence reduce size of xlog
> - reduce the following CRC calculation
> 
> I was thinking about using a simple run-length encoding to massively
> shrink half-empty blocks with lots of zero padding, but we've already
> got code to LZW the data down also.
> 
> Best Regards, Simon Riggs
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


Simon,

I think having a compression routine in there could make real sense.
We have done some major I/O testing involving compression for a large 
customer some time ago. We have seen that compressing / decompressing on 
the fly is in MOST cases much faster than uncompressed I/O (try a simple 
"cat file | ..." vs." zcat file.gz | ...") - the zcat version will be 
faster on all platforms we have tried (Linux, AIX, Sun on some SAN 
system, etc. ...).
Also, when building up a large database within one transaction the xlog 
will eat a lot of storage - this can be quite annoying when you have to 
deal with a lot of data).
Are there any technical reasons which would prevent somebody from 
implementing compression?

	Best regards,

		Hans

-- 
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/660/816 40 77
www.cybertec.at, www.postgresql.at


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq