485 lines
22 KiB
Plaintext
485 lines
22 KiB
Plaintext
From pgsql-hackers-owner+M5149@postgresql.org Mon Feb 26 03:32:49 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA04497
|
|
for <pgman@candle.pha.pa.us>; Mon, 26 Feb 2001 03:32:48 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f1Q8TSx48319;
|
|
Mon, 26 Feb 2001 03:29:28 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M5149@postgresql.org)
|
|
Received: from store.d.zembu.com (nat.zembu.com [209.128.96.253])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f1Q8LPx47243
|
|
for <pgsql-hackers@postgreSQL.org>; Mon, 26 Feb 2001 03:21:25 -0500 (EST)
|
|
(envelope-from ncm@zembu.com)
|
|
Received: by store.d.zembu.com (Postfix, from userid 509)
|
|
id 58E39A782; Mon, 26 Feb 2001 00:21:25 -0800 (PST)
|
|
Date: Mon, 26 Feb 2001 00:21:25 -0800
|
|
To: pgsql-hackers@postgresql.org
|
|
Subject: Re: [HACKERS] Re: [PATCHES] A patch for xlog.c
|
|
Message-ID: <20010226002125.A2430@store.zembu.com>
|
|
Reply-To: pgsql-hackers@postgresql.org
|
|
References: <200102260200.VAA17397@candle.pha.pa.us> <22318.983161726@sss.pgh.pa.us>
|
|
Mime-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
User-Agent: Mutt/1.2.5i
|
|
In-Reply-To: <22318.983161726@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Sun, Feb 25, 2001 at 11:28:46PM -0500
|
|
From: ncm@zembu.com (Nathan Myers)
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: ORr
|
|
|
|
On Sun, Feb 25, 2001 at 11:28:46PM -0500, Tom Lane wrote:
|
|
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
|
|
> > It allows no backing store on disk.
|
|
|
|
I.e. it allows you to map memory without an associated inode; the memory
|
|
may still be swapped. Of course, there is no problem with mapping an
|
|
inode too, so that unrelated processes can join in. Solarix has a flag
|
|
to pin the shared pages in RAM so they can't be swapped out.
|
|
|
|
> > It is the BSD solution to SysV
|
|
> > share memory. Here are all the BSDi flags:
|
|
>
|
|
> > MAP_ANON Map anonymous memory not associated with any specific
|
|
> > file. The file descriptor used for creating MAP_ANON
|
|
> > must be -1. The offset parameter is ignored.
|
|
>
|
|
> Hmm. Now that I read down to the "nonstandard extensions" part of the
|
|
> HPUX man page for mmap(), I find
|
|
>
|
|
> If MAP_ANONYMOUS is set in flags:
|
|
>
|
|
> o A new memory region is created and initialized to all zeros.
|
|
> This memory region can be shared only with descendants of
|
|
> the current process.
|
|
|
|
This is supported on Linux and BSD, but not on Solarix 7. It's not
|
|
necessary; you can just map /dev/zero on SysV systems that don't
|
|
have MAP_ANON.
|
|
|
|
> While I've said before that I don't think it's really necessary for
|
|
> processes that aren't children of the postmaster to access the shared
|
|
> memory, I'm not sure that I want to go over to a mechanism that makes it
|
|
> *impossible* for that to be done. Especially not if the only motivation
|
|
> is to avoid having to configure the kernel's shared memory settings.
|
|
|
|
There are enormous advantages to avoiding the need to configure kernel
|
|
settings. It makes PG a better citizen. PG is much easier to drop in
|
|
and use if you don't need attention from the IT department.
|
|
|
|
But I don't know of any reason to avoid mapping an actual inode,
|
|
so using mmap doesn't necessarily mean giving up sharing among
|
|
unrelated processes.
|
|
|
|
> Besides, what makes you think there's not a limit on the size of shmem
|
|
> allocatable via mmap()?
|
|
|
|
I've never seen any mmap limit documented. Since mmap() is how
|
|
everybody implements shared libraries, such a limit would be equivalent
|
|
to a limit on how much/many shared libraries are used. mmap() with
|
|
MAP_ANONYMOUS (or its SysV /dev/zero equivalent) is a common, modern
|
|
way to get raw storage for malloc(), so such a limit would be a limit
|
|
on malloc() too.
|
|
|
|
The mmap architecture comes to us from the Mach microkernel memory
|
|
manager, backported into BSD and then copied widely. Since it was
|
|
the fundamental mechanism for all memory operations in Mach, arbitrary
|
|
limits would make no sense. That it worked so well is the reason it
|
|
was copied everywhere else, so adding arbitrary limits while copying
|
|
it would be silly. I don't think we'll see any systems like that.
|
|
|
|
Nathan Myers
|
|
ncm@zembu.com
|
|
|
|
From pgsql-hackers-owner+M6138@postgresql.org Mon Mar 19 07:57:59 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id HAA26926
|
|
for <pgman@candle.pha.pa.us>; Mon, 19 Mar 2001 07:57:59 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f2JCug641835;
|
|
Mon, 19 Mar 2001 07:56:42 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M6138@postgresql.org)
|
|
Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20])
|
|
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f2JCt7641684
|
|
for <pgsql-hackers@postgresql.org>; Mon, 19 Mar 2001 07:55:07 -0500 (EST)
|
|
(envelope-from bright@fw.wintelcom.net)
|
|
Received: (from bright@localhost)
|
|
by fw.wintelcom.net (8.10.0/8.10.0) id f2JCt2325289;
|
|
Mon, 19 Mar 2001 04:55:02 -0800 (PST)
|
|
Date: Mon, 19 Mar 2001 04:55:01 -0800
|
|
From: Alfred Perlstein <bright@wintelcom.net>
|
|
To: Rod Taylor <rod.taylor@inquent.com>
|
|
Cc: Hackers List <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Fw: [vorbis-dev] ogg123: shared memory by mmap()
|
|
Message-ID: <20010319045500.T29888@fw.wintelcom.net>
|
|
References: <018301c0b070$16049a40$2205010a@jester>
|
|
Mime-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
User-Agent: Mutt/1.2.5i
|
|
In-Reply-To: <018301c0b070$16049a40$2205010a@jester>; from rod.taylor@inquent.com on Mon, Mar 19, 2001 at 07:28:21AM -0500
|
|
X-all-your-base: are belong to us.
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: ORr
|
|
|
|
WOOT WOOT! DANGER WILL ROBINSON!
|
|
|
|
> ----- Original Message -----
|
|
> From: "Christian Weisgerber" <naddy@mips.inka.de>
|
|
> Newsgroups: list.vorbis.dev
|
|
> To: <vorbis-dev@xiph.org>
|
|
> Sent: Saturday, March 17, 2001 12:01 PM
|
|
> Subject: [vorbis-dev] ogg123: shared memory by mmap()
|
|
>
|
|
>
|
|
> > The patch below adds:
|
|
> >
|
|
> > - acinclude.m4: A new macro A_FUNC_SMMAP to check that sharing
|
|
> pages
|
|
> > through mmap() works. This is taken from Joerg Schilling's star.
|
|
> > - configure.in: A_FUNC_SMMAP
|
|
> > - ogg123/buffer.c: If we have a working mmap(), use it to create
|
|
> > a region of shared memory instead of using System V IPC.
|
|
> >
|
|
> > Works on BSD. Should also work on SVR4 and offspring (Solaris),
|
|
> > and Linux.
|
|
|
|
This is a really bad idea performance wise. Solaris has a special
|
|
code path for SYSV shared memory that doesn't require tons of swap
|
|
tracking structures per-page/per-process. FreeBSD also has this
|
|
optimization (it's off by default, but should work since FreeBSD
|
|
4.2 via the sysctl kern.ipc.shm_use_phys=1)
|
|
|
|
Both OS's use a trick of making the pages non-pageable, this allows
|
|
signifigant savings in kernel space required for each attached
|
|
process, as well as the use of large pages which reduce the amount
|
|
of TLB faults your processes will incurr.
|
|
|
|
Anyhow, if you could make this a runtime option it wouldn't be so
|
|
evil, but as a compile time option, it's a really bad idea for
|
|
Solaris and FreeBSD.
|
|
|
|
--
|
|
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 2: you can get off all lists at once with the unregister command
|
|
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
|
|
|
|
From pgsql-hackers-owner+M6255@postgresql.org Tue Mar 20 18:46:33 2001
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id SAA02887
|
|
for <pgman@candle.pha.pa.us>; Tue, 20 Mar 2001 18:46:33 -0500 (EST)
|
|
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by mail.postgresql.org (8.11.3/8.11.1) with SMTP id f2KNjtH22390;
|
|
Tue, 20 Mar 2001 18:45:55 -0500 (EST)
|
|
(envelope-from pgsql-hackers-owner+M6255@postgresql.org)
|
|
Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20])
|
|
by mail.postgresql.org (8.11.3/8.11.1) with ESMTP id f2KNiFH22033
|
|
for <pgsql-hackers@postgresql.org>; Tue, 20 Mar 2001 18:44:15 -0500 (EST)
|
|
(envelope-from bright@fw.wintelcom.net)
|
|
Received: (from bright@localhost)
|
|
by fw.wintelcom.net (8.10.0/8.10.0) id f2KNiAW02417;
|
|
Tue, 20 Mar 2001 15:44:10 -0800 (PST)
|
|
Date: Tue, 20 Mar 2001 15:44:10 -0800
|
|
From: Alfred Perlstein <bright@wintelcom.net>
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>
|
|
Cc: Rod Taylor <rod.taylor@inquent.com>,
|
|
Hackers List <pgsql-hackers@postgresql.org>
|
|
Subject: Re: [HACKERS] Fw: [vorbis-dev] ogg123: shared memory by mmap()
|
|
Message-ID: <20010320154410.H29888@fw.wintelcom.net>
|
|
References: <20010319045500.T29888@fw.wintelcom.net> <200103202210.RAA23981@candle.pha.pa.us>
|
|
Mime-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
User-Agent: Mutt/1.2.5i
|
|
In-Reply-To: <200103202210.RAA23981@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Mar 20, 2001 at 05:10:33PM -0500
|
|
X-all-your-base: are belong to us.
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: OR
|
|
|
|
* Bruce Momjian <pgman@candle.pha.pa.us> [010320 14:10] wrote:
|
|
> > > > The patch below adds:
|
|
> > > >
|
|
> > > > - acinclude.m4: A new macro A_FUNC_SMMAP to check that sharing
|
|
> > > pages
|
|
> > > > through mmap() works. This is taken from Joerg Schilling's star.
|
|
> > > > - configure.in: A_FUNC_SMMAP
|
|
> > > > - ogg123/buffer.c: If we have a working mmap(), use it to create
|
|
> > > > a region of shared memory instead of using System V IPC.
|
|
> > > >
|
|
> > > > Works on BSD. Should also work on SVR4 and offspring (Solaris),
|
|
> > > > and Linux.
|
|
> >
|
|
> > This is a really bad idea performance wise. Solaris has a special
|
|
> > code path for SYSV shared memory that doesn't require tons of swap
|
|
> > tracking structures per-page/per-process. FreeBSD also has this
|
|
> > optimization (it's off by default, but should work since FreeBSD
|
|
> > 4.2 via the sysctl kern.ipc.shm_use_phys=1)
|
|
>
|
|
> >
|
|
> > Both OS's use a trick of making the pages non-pageable, this allows
|
|
> > signifigant savings in kernel space required for each attached
|
|
> > process, as well as the use of large pages which reduce the amount
|
|
> > of TLB faults your processes will incurr.
|
|
>
|
|
> That is interesting. BSDi has SysV shared memory as non-pagable, and I
|
|
> always thought of that as a bug. Seems you are saying that having it
|
|
> pagable has a significant performance penalty. Interesting.
|
|
|
|
Yes, having it pageable is actually sort of bad.
|
|
|
|
It doesn't allow you to do several important optimizations.
|
|
|
|
--
|
|
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 4: Don't 'kill -9' the postmaster
|
|
|
|
From pgsql-general-owner+M14300@postgresql.org Mon Aug 27 13:07:32 2001
|
|
Return-path: <pgsql-general-owner+M14300@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238])
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f7RH7VF04800
|
|
for <pgman@candle.pha.pa.us>; Mon, 27 Aug 2001 13:07:31 -0400 (EDT)
|
|
Received: from postgresql.org.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f7RH7Tq17721;
|
|
Mon, 27 Aug 2001 12:07:29 -0500 (CDT)
|
|
(envelope-from pgsql-general-owner+M14300@postgresql.org)
|
|
Received: from svana.org (svana.org [210.9.66.30])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f7RFE1f13269
|
|
for <pgsql-general@postgresql.org>; Mon, 27 Aug 2001 11:14:01 -0400 (EDT)
|
|
(envelope-from kleptog@svana.org)
|
|
Received: from kleptog by svana.org with local (Exim 3.12 #1 (Debian))
|
|
id 15bO5x-0000Fd-00; Tue, 28 Aug 2001 01:14:33 +1000
|
|
Date: Tue, 28 Aug 2001 01:14:33 +1000
|
|
From: Martijn van Oosterhout <kleptog@svana.org>
|
|
To: Andrew Snow <andrew@modulus.org>
|
|
cc: pgsql-general@postgresql.org
|
|
Subject: Re: [GENERAL] raw partition
|
|
Message-ID: <20010828011433.E32309@svana.org>
|
|
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
|
|
References: <20010827233815.B32309@svana.org> <000101c12f00$dc5814b0$fa01b5ca@avon>
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
User-Agent: Mutt/1.2.5i
|
|
In-Reply-To: <000101c12f00$dc5814b0$fa01b5ca@avon>; from andrew@modulus.org on Tue, Aug 28, 2001 at 12:02:08AM +1000
|
|
Precedence: bulk
|
|
Sender: pgsql-general-owner@postgresql.org
|
|
Status: OR
|
|
|
|
On Tue, Aug 28, 2001 at 12:02:08AM +1000, Andrew Snow wrote:
|
|
>
|
|
> What I think would be better would be moving postgresql to a system of
|
|
> using memory-mapped I/O. instead of the shared buffer cache, files
|
|
> would be directly memory-mapped and the OS would do the caching. I
|
|
> can't see this happening though because of platform dependancy, but I
|
|
> think its worth another look soon because many unix platforms support
|
|
> mmap(). I think it would improve the performance of disk-intensive
|
|
> tasks noticeably.
|
|
|
|
Well, this has other problems. Consider tables that are larger than your
|
|
system memory. You'd have to continuously map and unmap different sections.
|
|
That can have odd side effects (witness mozilla on linux having 15,000
|
|
mapped areas or so...)
|
|
|
|
You would still however get the advantage that you wouldn't have to copy the
|
|
data from the disk buffers to user space, you simply get the disk buffer
|
|
mapped into your address space.
|
|
|
|
I think that for commonly used tables that are under 100K in size (most of
|
|
the system tables), this is quite a workable idea. If you don't mind keeping
|
|
them mapped the whole time.
|
|
|
|
--
|
|
Martijn van Oosterhout <kleptog@svana.org>
|
|
http://svana.org/kleptog/
|
|
> It would be nice if someone came up with a certification system that
|
|
> actually separated those who can barely regurgitate what they crammed over
|
|
> the last few weeks from those who command secret ninja networking powers.
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 3: if posting/reading through Usenet, please send an appropriate
|
|
subscribe-nomail command to majordomo@postgresql.org so that your
|
|
message can get through to the mailing list cleanly
|
|
|
|
From pgsql-general-owner+M14319@postgresql.org Mon Aug 27 16:57:10 2001
|
|
Return-path: <pgsql-general-owner+M14319@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238])
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f7RKv9F16849
|
|
for <pgman@candle.pha.pa.us>; Mon, 27 Aug 2001 16:57:09 -0400 (EDT)
|
|
Received: from postgresql.org.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f7RKv9q31456;
|
|
Mon, 27 Aug 2001 15:57:09 -0500 (CDT)
|
|
(envelope-from pgsql-general-owner+M14319@postgresql.org)
|
|
Received: from sss.pgh.pa.us ([192.204.191.242])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f7RJrsf55472
|
|
for <pgsql-general@postgresql.org>; Mon, 27 Aug 2001 15:53:54 -0400 (EDT)
|
|
(envelope-from tgl@sss.pgh.pa.us)
|
|
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
|
|
by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f7RJrGK19431;
|
|
Mon, 27 Aug 2001 15:53:16 -0400 (EDT)
|
|
To: Martijn van Oosterhout <kleptog@svana.org>
|
|
cc: Andrew Snow <andrew@modulus.org>, pgsql-general@postgresql.org
|
|
Subject: Re: [GENERAL] raw partition
|
|
In-Reply-To: <20010828011433.E32309@svana.org>
|
|
References: <20010827233815.B32309@svana.org> <000101c12f00$dc5814b0$fa01b5ca@avon> <20010828011433.E32309@svana.org>
|
|
Comments: In-reply-to Martijn van Oosterhout <kleptog@svana.org>
|
|
message dated "Tue, 28 Aug 2001 01:14:33 +1000"
|
|
Date: Mon, 27 Aug 2001 15:53:15 -0400
|
|
Message-ID: <19428.998941995@sss.pgh.pa.us>
|
|
From: Tom Lane <tgl@sss.pgh.pa.us>
|
|
Precedence: bulk
|
|
Sender: pgsql-general-owner@postgresql.org
|
|
Status: OR
|
|
|
|
Martijn van Oosterhout <kleptog@svana.org> writes:
|
|
> You would still however get the advantage that you wouldn't have to copy the
|
|
> data from the disk buffers to user space, you simply get the disk buffer
|
|
> mapped into your address space.
|
|
|
|
AFAICS this would be the *only* advantage. While it's not negligible,
|
|
it's quite unclear that it's worth the bookkeeping and portability
|
|
headaches of managing lots of mmap'd areas, either.
|
|
|
|
Before I take this idea seriously at all, I'd want to see a design that
|
|
addresses a couple of critical issues:
|
|
|
|
1. Postgres' shared buffers are *shared*, potentially across many
|
|
processes. How will you deal with buffers for files that have been
|
|
mmap'd by only some of the processes? (Maybe this means that the
|
|
whole concept of shared buffers goes away, and each process does its
|
|
own buffer management based on its own mmaps. Not sure. That would be
|
|
a pretty radical restructuring though, and would completely invalidate
|
|
our present approach to page-level locking.)
|
|
|
|
2. How do you deal with extending a file? My system's mmap man page
|
|
says
|
|
If the size of the mapped file changes after the call to mmap(), the
|
|
effect of references to portions of the mapped region that correspond
|
|
to added or removed portions of the file is unspecified.
|
|
This suggests that the only portable way to cope is to issue a separate
|
|
mmap for every disk page. Will typical Unix systems perform well with
|
|
umpteen thousand small mmap requests?
|
|
|
|
3. How do you persuade the other backends to drop their mmaps of a table
|
|
you are deleting?
|
|
|
|
There are probably other gotchas, but without an understanding of how
|
|
to address these, I doubt it's worth looking further ...
|
|
|
|
regards, tom lane
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 5: Have you checked our extensive FAQ?
|
|
|
|
http://www.postgresql.org/users-lounge/docs/faq.html
|
|
|
|
From pgsql-hackers-owner+M13750=candle.pha.pa.us=pgman@postgresql.org Mon Oct 1 05:59:15 2001
|
|
Return-path: <pgsql-hackers-owner+M13750=candle.pha.pa.us=pgman@postgresql.org>
|
|
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f919xF512590
|
|
for <pgman@candle.pha.pa.us>; Mon, 1 Oct 2001 05:59:15 -0400 (EDT)
|
|
Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
|
|
by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f919xA207817
|
|
for <pgman@candle.pha.pa.us>; Mon, 1 Oct 2001 04:59:10 -0500 (CDT)
|
|
(envelope-from pgsql-hackers-owner+M13750=candle.pha.pa.us=pgman@postgresql.org)
|
|
Received: from mrsgntmail01.mediaring.com.sg (mserver.mediaring.com.sg [203.208.141.175])
|
|
by postgresql.org (8.11.3/8.11.4) with ESMTP id f919rE320926
|
|
for <pgsql-hackers@postgreSQL.org>; Mon, 1 Oct 2001 05:53:15 -0400 (EDT)
|
|
(envelope-from jana-reddy@mediaring.com.sg)
|
|
Received: by MRSGNTMAIL01 with Internet Mail Service (5.5.2650.21)
|
|
id <PMTCM7SJ>; Mon, 1 Oct 2001 18:03:34 +0800
|
|
Received: from mediaring.com.sg (10.1.0.131 [10.1.0.131]) by mrsgntmail01.mediaring.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21)
|
|
id PMTCM7SH; Mon, 1 Oct 2001 18:03:25 +0800
|
|
From: Janardhana Reddy <jana-reddy@mediaring.com.sg>
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>
|
|
cc: PostgreSQL-development <pgsql-hackers@postgresql.org>,
|
|
janareddy
|
|
<jana-reddy@mediaring.com.sg>
|
|
Message-ID: <3BB83DF0.8946973@mediaring.com.sg>
|
|
Date: Mon, 01 Oct 2001 17:57:04 +0800
|
|
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.0 i686)
|
|
X-Accept-Language: en
|
|
MIME-Version: 1.0
|
|
Subject: Re: [HACKERS] PERFORMANCE IMPROVEMENT by mapping WAL FILES
|
|
References: <200109282137.f8SLbpm01890@candle.pha.pa.us>
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Transfer-Encoding: 7bit
|
|
Precedence: bulk
|
|
Sender: pgsql-hackers-owner@postgresql.org
|
|
Status: ORr
|
|
|
|
I have just completed the functional testing the WAL using mmap , it is
|
|
|
|
working fine, I have tested by commenting out the "CreateCheckPoint "
|
|
functionality so that
|
|
when i kill the postgres and restart it will redo all the records from the
|
|
WAL log file which
|
|
is updated using mmap.
|
|
Just i need to clean code and to do some stress testing.
|
|
By the end of this week i should able to complete the stress test and
|
|
generate the patch file .
|
|
As Tom Lane mentioned i see the problem in portability to all platforms,
|
|
|
|
what i propose is to use mmap for only WAL for some platforms like
|
|
linux,freebsd etc . For other platforms we can use the existing method by
|
|
slightly modifying the
|
|
write() routine to write only the modified part of the page.
|
|
|
|
Regards
|
|
jana
|
|
|
|
>
|
|
>
|
|
> OK, I have talked to Tom Lane about this on the phone and we have a few
|
|
> ideas.
|
|
>
|
|
> Historically, we have avoided mmap() because of portability problems,
|
|
> and because using mmap() to write to large tables could consume lots of
|
|
> address space with little benefit. However, I perhaps can see WAL as
|
|
> being a good use of mmap.
|
|
>
|
|
> First, there is the issue of using mmap(). For OS's that have the
|
|
> mmap() MAP_SHARED flag, different backends could mmap the same file and
|
|
> each see the changes. However, keep in mind we still have to fsync()
|
|
> WAL, so we need to use msync().
|
|
>
|
|
> So, looking at the benefits of using mmap(), we have overhead of
|
|
> different backends having to mmap something that now sits quite easily
|
|
> in shared memory. Now, I can see mmap reducing the copy from user to
|
|
> kernel, but there are other ways to fix that. We could modify the
|
|
> write() routines to write() 8k on first WAL page write and later write
|
|
> only the modified part of the page to the kernel buffers. The old
|
|
> kernel buffer is probably still around so it is unlikely to require a
|
|
> read from the file system to read in the rest of the page. This reduces
|
|
> the write from 8k to something probably less than 4k which is better
|
|
> than we can do with mmap.
|
|
>
|
|
> I will add a TODO item to this effect.
|
|
>
|
|
> As far as reducing the write to disk from 8k to 4k, if we have to
|
|
> fsync/msync, we have to wait for the disk to spin to the proper location
|
|
> and at that point writing 4k or 8k doesn't seem like much of a win.
|
|
>
|
|
> In summary, I think it would be nice to reduce the 8k transfer from user
|
|
> to kernel on secondary page writes to only the modified part of the
|
|
> page. I am uncertain if mmap() or anything else will help the physical
|
|
> write to the disk.
|
|
>
|
|
> --
|
|
> Bruce Momjian | http://candle.pha.pa.us
|
|
> pgman@candle.pha.pa.us | (610) 853-3000
|
|
> + If your life is a hard drive, | 830 Blythe Avenue
|
|
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 6: Have you searched our list archives?
|
|
|
|
http://archives.postgresql.org
|
|
|