mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-08-19 10:03:23 +02:00
506 lines
24 KiB
Plaintext
506 lines
24 KiB
Plaintext
From pgsql-performance-owner+M3897@postgresql.org Sat Oct 4 19:50:57 2003
|
|
Return-path: <pgsql-performance-owner+M3897@postgresql.org>
|
|
Received: from svr5.postgresql.org (svr5.postgresql.org [64.117.225.181])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h94NotQ08911
|
|
for <pgman@candle.pha.pa.us>; Sat, 4 Oct 2003 19:50:56 -0400 (EDT)
|
|
Received: from postgresql.org (svr1.postgresql.org [64.117.224.193])
|
|
by svr5.postgresql.org (Postfix) with ESMTP
|
|
id DB0F072DC9E; Sat, 4 Oct 2003 23:50:50 +0000 (GMT)
|
|
X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [64.117.224.130])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 70DDDD1B4EC
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>; Sat, 4 Oct 2003 23:50:42 +0000 (GMT)
|
|
Received: from svr1.postgresql.org ([64.117.224.193])
|
|
by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024)
|
|
with ESMTP id 14368-03
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
|
|
Sat, 4 Oct 2003 20:49:56 -0300 (ADT)
|
|
Received: from news.hub.org (unknown [64.117.224.194])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 2FBF7D1B4F0
|
|
for <pgsql-performance@postgresql.org>; Sat, 4 Oct 2003 20:49:53 -0300 (ADT)
|
|
Received: from news.hub.org (host-64-117-224-194.altec1.com [64.117.224.194] (may be forged))
|
|
by news.hub.org (8.12.9/8.12.9) with ESMTP id h94NnqQh076664
|
|
for <pgsql-performance@postgresql.org>; Sat, 4 Oct 2003 23:49:52 GMT
|
|
(envelope-from news@news.hub.org)
|
|
Received: (from news@localhost)
|
|
by news.hub.org (8.12.9/8.12.9/Submit) id h94NaQEP075478
|
|
for pgsql-performance@postgresql.org; Sat, 4 Oct 2003 23:36:26 GMT
|
|
From: Christopher Browne <cbbrowne@acm.org>
|
|
X-Newsgroups: comp.databases.postgresql.performance
|
|
Subject: Re: [PERFORM] count(*) slow on large tables
|
|
Date: Sat, 04 Oct 2003 19:33:46 -0400
|
|
Organization: cbbrowne Computing Inc
|
|
Lines: 77
|
|
Message-ID: <m3u16ovaqt.fsf@wolfe.cbbrowne.com>
|
|
References: <200310041556.h94Fuek24423@candle.pha.pa.us> <6743.1065286173@sss.pgh.pa.us>
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
X-Complaints-To: usenet@news.hub.org
|
|
X-message-flag: Outlook is rather hackable, isn't it?
|
|
X-Home-Page: http://www.cbbrowne.com/info/
|
|
X-Affero: http://svcs.affero.net/rm.php?r=cbbrowne
|
|
User-Agent: Gnus/5.1003 (Gnus v5.10.3) XEmacs/21.4 (Reasonable Discussion, linux)
|
|
Cancel-Lock: sha1:lLXE17xNVoXrMYZPn8CzzK9g1mc=
|
|
To: pgsql-performance@postgresql.org
|
|
X-Virus-Scanned: by amavisd-new at postgresql.org
|
|
X-Mailing-List: pgsql-performance
|
|
Precedence: bulk
|
|
Sender: pgsql-performance-owner@postgresql.org
|
|
Status: OR
|
|
|
|
Quoth tgl@sss.pgh.pa.us (Tom Lane):
|
|
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
|
|
>> We do have a TODO item:
|
|
>> * Consider using MVCC to cache count(*) queries with no WHERE clause
|
|
>
|
|
>> The idea is to cache a recent count of the table, then have
|
|
>> insert/delete add +/- records to the count. A COUNT(*) would get the
|
|
>> main cached record plus any visible +/- records. This would allow the
|
|
>> count to return the proper value depending on the visibility of the
|
|
>> requesting transaction, and it would require _no_ heap or index scan.
|
|
>
|
|
> ... and it would give the wrong answers. Unless the cache is somehow
|
|
> snapshot-aware, so that it can know which other transactions should be
|
|
> included in your count.
|
|
|
|
[That's an excellent summary that Bruce did of what came out of the
|
|
previous discussion...]
|
|
|
|
If this "cache" was a table, itself, the visibility of its records
|
|
should be identical to that of the visibility of the "real" records.
|
|
+/- records would become visible when the transaction COMMITed, at the
|
|
very same time the source records became visible.
|
|
|
|
I thought, at one point, that it would be a slick idea for "record
|
|
compression" to take place automatically; when you do a COUNT(*), the
|
|
process would include compressing multiple records down to one.
|
|
Unfortunately, that turns out to be Tremendously Evil if the same
|
|
COUNT(*) were being concurrently processed in multiple transactions.
|
|
Both would repeat much the same work, and this would ultimately lead
|
|
to one of the transactions aborting. [I recently saw this effect
|
|
occur, um, a few times...]
|
|
|
|
For this not to have Evil Effects on unsuspecting transactions, we
|
|
would instead require some process analagous to VACUUM, where a single
|
|
transaction would be used to compress the "counts table" down to one
|
|
record per table. Being independent of "user transactions," it could
|
|
safely compress the data without injuring unsuspecting transactions.
|
|
|
|
But in most cases, the cost of this would be pretty prohibitive.
|
|
Every transaction that adds a record to a table leads to a record
|
|
being added to table "pg_exact_row_counts". If transactions typically
|
|
involve adding ONE row to any given table, this effectively doubles
|
|
the update traffic. Ouch. That means that in a _real_
|
|
implementation, it would make sense to pick and choose the tables that
|
|
would be so managed.
|
|
|
|
In my earlier arguing of "You don't really want that!", while I may
|
|
have been guilty of engaging in a _little_ hyperbole, I was certainly
|
|
_not_ being facetious overall. At work, we tell the developers "avoid
|
|
doing COUNT(*) inside ordinary transactions!", and that is certainly
|
|
NOT facetious comment. I recall a case a while back where system
|
|
performance was getting brutalized by a lurking COUNT(*). (Combined
|
|
with some other pathological behaviour, of course!) And note that
|
|
this wasn't a query that the TODO item could address; it was of the
|
|
form "SELECT COUNT(*) FROM SOME_TABLE WHERE OWNER = VALUE;"
|
|
|
|
As you have commented elsewhere in the thread, much of the time, the
|
|
point of asking for COUNT(*) is often to get some idea of table size,
|
|
where the precise number isn't terribly important in comparison with
|
|
getting general magnitude. Improving the ability to get approximate
|
|
values would be of some value.
|
|
|
|
I would further argue that "SELECT COUNT(*) FROM TABLE" isn't
|
|
particularly useful even when precision _is_ important. If I'm
|
|
working on reports that would be used to reconcile things, the queries
|
|
I use are a whole lot more involved than that simple form. It is far
|
|
more likely that I'm using a GROUP BY.
|
|
|
|
It is legitimate to get wishful and imagine that it would be nice if
|
|
we could get the value of that query "instantaneously." It is also
|
|
legitimate to think that the effort required to implement that might
|
|
be better used on improving other things.
|
|
--
|
|
(reverse (concatenate 'string "ac.notelrac.teneerf" "@" "454aa"))
|
|
http://www3.sympatico.ca/cbbrowne/
|
|
"very few people approach me in real life and insist on proving they
|
|
are drooling idiots." -- Erik Naggum, comp.lang.lisp
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 5: Have you checked our extensive FAQ?
|
|
|
|
http://www.postgresql.org/docs/faqs/FAQ.html
|
|
|
|
From josh@agliodbs.com Sun Oct 5 14:59:07 2003
|
|
Return-path: <josh@agliodbs.com>
|
|
Received: from davinci.ethosmedia.com (vista1-228.percepticon.net [209.128.84.228] (may be forged))
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h95Ix5Q17861
|
|
for <pgman@candle.pha.pa.us>; Sun, 5 Oct 2003 14:59:06 -0400 (EDT)
|
|
Received: from [63.195.55.98] (HELO spooky)
|
|
by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.0.2)
|
|
with ESMTP id 3728969; Sun, 05 Oct 2003 11:59:26 -0700
|
|
Content-Type: text/plain;
|
|
charset="iso-8859-1"
|
|
From: Josh Berkus <josh@agliodbs.com>
|
|
Organization: Aglio Database Solutions
|
|
To: Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>
|
|
Subject: Re: [PERFORM] count(*) slow on large tables
|
|
Date: Sun, 5 Oct 2003 11:57:21 -0700
|
|
User-Agent: KMail/1.4.3
|
|
cc: Christopher Browne <cbbrowne@libertyrms.info>,
|
|
pgsql-performance@postgresql.org
|
|
References: <200310041819.h94IJkV07596@candle.pha.pa.us>
|
|
In-Reply-To: <200310041819.h94IJkV07596@candle.pha.pa.us>
|
|
MIME-Version: 1.0
|
|
Content-Transfer-Encoding: 8bit
|
|
Message-ID: <200310051157.21555.josh@agliodbs.com>
|
|
Status: OR
|
|
|
|
Bruce,
|
|
|
|
> OK, I beefed up the TODO:
|
|
>
|
|
> * Use a fixed row count and a +/- count with MVCC visibility rules
|
|
> to allow fast COUNT(*) queries with no WHERE clause(?)
|
|
>
|
|
> I can always give the details if someone asks. It doesn't seem complex
|
|
> enough for a separate TODO.detail item.
|
|
|
|
Hmmm ... this doesn't seem effort-worthy to me. How often does anyone do
|
|
COUNT with no where clause, except GUIs that give you a record count? (of
|
|
course, as always, if someone wants to code it, feel free ...)
|
|
|
|
And for those GUIs, wouldn't it be 97% as good to run an ANALYZE and give the
|
|
approximate record counts for large tables?
|
|
|
|
As for counts with a WHERE clause, this is obviously up to the user. Joe
|
|
Conway and I tested using a C trigger to track some COUNT ... GROUP BY values
|
|
for large tables based on additive numbers. It worked fairly well for
|
|
accuracy, but the performance penalty on data writes was significant ... 8%
|
|
to 25% penalty for UPDATES, depending on the frequency and batch size (>
|
|
frequency > batch size --> > penalty)
|
|
|
|
It's possible that this could be improved through some mechanism more tightly
|
|
integrated with the source code. However,the coding effort would be
|
|
significant ( 12-20 hours ) and it's possible that there would be no
|
|
improvement, which is why we didn't do it.
|
|
|
|
We also discussed an asynchronous aggregates collector that would work
|
|
something like the statistics collector, and keep pre-programmmed aggregate
|
|
data, updating during "low-activity" periods. This would significantly
|
|
reduce the performance penalty, but at the cost of accuracy ... that is, a
|
|
1%-5% variance on high-activity tables would be unavoidable, and all cached
|
|
aggregates would have to be recalculated on database restart, significantly
|
|
slowing down startup. Again, we felt that the effort-result payoff was not
|
|
worthwhile.
|
|
|
|
Overall, I think the stuff we already have planned ... the hash aggregates in
|
|
7.4 and Tom's suggestion of adding an indexable flag to pg_aggs ... are far
|
|
more likely to yeild useful fruit than any caching plan.
|
|
|
|
--
|
|
Josh Berkus
|
|
Aglio Database Solutions
|
|
San Francisco
|
|
|
|
From pgsql-performance-owner+M3915@postgresql.org Mon Oct 6 02:08:33 2003
|
|
Return-path: <pgsql-performance-owner+M3915@postgresql.org>
|
|
Received: from svr5.postgresql.org (svr5.postgresql.org [64.117.225.181])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h9668VQ15914
|
|
for <pgman@candle.pha.pa.us>; Mon, 6 Oct 2003 02:08:32 -0400 (EDT)
|
|
Received: from postgresql.org (svr1.postgresql.org [64.117.224.193])
|
|
by svr5.postgresql.org (Postfix) with ESMTP
|
|
id DC70672E71E; Mon, 6 Oct 2003 06:08:24 +0000 (GMT)
|
|
X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [64.117.224.130])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id BFA49D1B4F6
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>; Mon, 6 Oct 2003 06:07:33 +0000 (GMT)
|
|
Received: from svr1.postgresql.org ([64.117.224.193])
|
|
by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024)
|
|
with ESMTP id 90800-06
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
|
|
Mon, 6 Oct 2003 03:06:44 -0300 (ADT)
|
|
Received: from smtp.pspl.co.in (unknown [202.54.11.65])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 9033ED1B4EB
|
|
for <pgsql-performance@postgresql.org>; Mon, 6 Oct 2003 03:06:41 -0300 (ADT)
|
|
Received: (from root@localhost)
|
|
by smtp.pspl.co.in (8.12.9/8.12.9) id h966AmTk013993
|
|
for <pgsql-performance@postgresql.org>; Mon, 6 Oct 2003 11:40:49 +0530
|
|
Received: from persistent.co.in (daithan.intranet.pspl.co.in [192.168.7.161])
|
|
(authenticated bits=0)
|
|
by persistent.co.in (8.12.9/8.12.9) with ESMTP id h966AlYM013922
|
|
for <pgsql-performance@postgresql.org>; Mon, 6 Oct 2003 11:40:48 +0530
|
|
Message-ID: <3F81066C.90402@persistent.co.in>
|
|
Date: Mon, 06 Oct 2003 11:36:36 +0530
|
|
From: Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
|
|
Organization: Persistent Systems Pvt. Ltd.
|
|
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5b) Gecko/20030917 Thunderbird/0.3a
|
|
X-Accept-Language: en-us, en
|
|
MIME-Version: 1.0
|
|
To: pgsql-performance@postgresql.org
|
|
Subject: Re: [PERFORM] count(*) slow on large tables
|
|
References: <200310041819.h94IJkV07596@candle.pha.pa.us>
|
|
In-Reply-To: <200310041819.h94IJkV07596@candle.pha.pa.us>
|
|
Content-Type: text/plain; charset=us-ascii; format=flowed
|
|
Content-Transfer-Encoding: 7bit
|
|
X-Virus-Scanned: by amavisd-new at postgresql.org
|
|
X-Mailing-List: pgsql-performance
|
|
Precedence: bulk
|
|
Sender: pgsql-performance-owner@postgresql.org
|
|
Status: OR
|
|
|
|
Bruce Momjian wrote:
|
|
> OK, I beefed up the TODO:
|
|
>
|
|
> * Use a fixed row count and a +/- count with MVCC visibility rules
|
|
> to allow fast COUNT(*) queries with no WHERE clause(?)
|
|
>
|
|
> I can always give the details if someone asks. It doesn't seem complex
|
|
> enough for a separate TODO.detail item.
|
|
|
|
May I propose alternate approach for this optimisation?
|
|
|
|
- Postgresql allows to maintain user defined variables in shared memory.
|
|
- These variables obey transactions but do not get written to disk at all.
|
|
- There should be a facility to detect whether such a variable is initialized or
|
|
not.
|
|
|
|
How it will help? This is in addition to trigger proposal that came up earlier.
|
|
With triggers it's not possible to make values visible across backends unless
|
|
trigger updates a table, which eventually leads to vacuum/dead tuples problem.
|
|
|
|
1. User creates a trigger to check updates/inserts for certain conditions.
|
|
2. It updates the count as and when required.
|
|
3. If the trigger detects the count is not initialized, it would issue the same
|
|
query first time. There is no avoiding this issue.
|
|
|
|
Besides providing facility of resident variables could be used imaginatively as
|
|
well.
|
|
|
|
Does this make sense? IMO this is more generalised approach over all.
|
|
|
|
Just a thought.
|
|
|
|
Shridhar
|
|
|
|
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 6: Have you searched our list archives?
|
|
|
|
http://archives.postgresql.org
|
|
|
|
From pgsql-performance-owner+M3938@postgresql.org Mon Oct 6 16:04:10 2003
|
|
Return-path: <pgsql-performance-owner+M3938@postgresql.org>
|
|
Received: from svr5.postgresql.org (svr5.postgresql.org [64.117.225.181])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h96K49i20610
|
|
for <pgman@candle.pha.pa.us>; Mon, 6 Oct 2003 16:04:10 -0400 (EDT)
|
|
Received: from postgresql.org (svr1.postgresql.org [64.117.224.193])
|
|
by svr5.postgresql.org (Postfix) with ESMTP
|
|
id 9B73272DC4D; Mon, 6 Oct 2003 20:04:08 +0000 (GMT)
|
|
X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [64.117.224.130])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id 3770CD1B567
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>; Mon, 6 Oct 2003 15:11:08 +0000 (GMT)
|
|
Received: from svr1.postgresql.org ([64.117.224.193])
|
|
by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024)
|
|
with ESMTP id 81338-08
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
|
|
Mon, 6 Oct 2003 12:10:22 -0300 (ADT)
|
|
Received: from main.gmane.org (main.gmane.org [80.91.224.249])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id E71D7D1B51E
|
|
for <pgsql-performance@postgresql.org>; Mon, 6 Oct 2003 12:10:21 -0300 (ADT)
|
|
Received: from root by main.gmane.org with local (Exim 3.35 #1 (Debian))
|
|
id 1A6X08-0003KO-00
|
|
for <pgsql-performance@postgresql.org>; Mon, 06 Oct 2003 17:10:20 +0200
|
|
X-Injected-Via-Gmane: http://gmane.org/
|
|
To: pgsql-performance@postgresql.org
|
|
Received: from sea.gmane.org ([80.91.224.252])
|
|
by main.gmane.org with esmtp (Exim 3.35 #1 (Debian))
|
|
id 1A6Wxn-0003Hh-00
|
|
for <gmane-comp-db-postgresql-performance@m.gmane.org>; Mon, 06 Oct 2003 17:07:55 +0200
|
|
Received: from news by sea.gmane.org with local (Exim 3.35 #1 (Debian))
|
|
id 1A6Wxn-0006U8-00
|
|
for <gmane-comp-db-postgresql-performance@m.gmane.org>; Mon, 06 Oct 2003 17:07:55 +0200
|
|
From: Harald Fuchs <nospam@sap.com>
|
|
Subject: Re: [PERFORM] count(*) slow on large tables
|
|
Date: 06 Oct 2003 17:08:36 +0200
|
|
Organization: Linux Private Site
|
|
Lines: 21
|
|
Message-ID: <pupthae74b.fsf@srv.protecting.net>
|
|
References: <20031002191547.GZ87525@rlx11.zapatec.com> <20031002193905.GD18417@wolff.to> <3F7C98B8.C892D0E5@nsd.ca> <60brszcng5.fsf@dev6.int.libertyrms.info> <20031002223313.GE87525@rlx11.zapatec.com> <m3vfr7f4z1.fsf@wolfe.cbbrowne.com> <20031003042754.GH87525@rlx11.zapatec.com> <3F7D172E.3060107@persistent.co.in>
|
|
Reply-To: hf99@protecting.net
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=US-ASCII
|
|
X-Complaints-To: usenet@sea.gmane.org
|
|
X-No-Archive: yes
|
|
User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7
|
|
X-Virus-Scanned: by amavisd-new at postgresql.org
|
|
X-Mailing-List: pgsql-performance
|
|
Precedence: bulk
|
|
Sender: pgsql-performance-owner@postgresql.org
|
|
Status: OR
|
|
|
|
In article <3F7D172E.3060107@persistent.co.in>,
|
|
Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
|
|
|
|
> Dror Matalon wrote:
|
|
>> I smell a religious war in the aii:-). Can you go several days in a
|
|
>> row without doing select count(*) on any
|
|
>> of your tables? I suspect that this is somewhat a domain specific
|
|
>> issue. In some areas
|
|
>> you don't need to know the total number of rows in your tables, in
|
|
>> others you do.
|
|
|
|
> If I were you, I would have an autovacuum daemon running and rather
|
|
> than doing select count(*), I would look at stats generated by
|
|
> vacuums. They give approximate number of tuples and it should be good
|
|
> enough it is accurate within a percent.
|
|
|
|
The stats might indeed be a good estimate presumed there were not many
|
|
changes since the last VACUUM. But how about a variant of COUNT(*)
|
|
using an index? It would not be quite exact since it might contain
|
|
tuples not visible in the current transaction, but it might be a much
|
|
better estimate than the stats.
|
|
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 2: you can get off all lists at once with the unregister command
|
|
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
|
|
|
|
From pgsql-performance-owner+M3930@postgresql.org Mon Oct 6 13:03:02 2003
|
|
Return-path: <pgsql-performance-owner+M3930@postgresql.org>
|
|
Received: from svr4.postgresql.org (svr4.postgresql.org [64.117.224.192])
|
|
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h96H30Q06466
|
|
for <pgman@candle.pha.pa.us>; Mon, 6 Oct 2003 13:03:00 -0400 (EDT)
|
|
Received: from postgresql.org (svr1.postgresql.org [64.117.224.193])
|
|
by svr4.postgresql.org (Postfix) with ESMTP
|
|
id 314A01CB46D6; Mon, 6 Oct 2003 17:02:55 +0000 (GMT)
|
|
X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
|
|
Received: from localhost (unknown [64.117.224.130])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id E83D7D1B4F2
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>; Mon, 6 Oct 2003 17:02:38 +0000 (GMT)
|
|
Received: from svr1.postgresql.org ([64.117.224.193])
|
|
by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024)
|
|
with ESMTP id 03671-08
|
|
for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
|
|
Mon, 6 Oct 2003 14:01:53 -0300 (ADT)
|
|
Received: from perrin.nxad.com (internal.nxad.com [69.1.70.251])
|
|
by svr1.postgresql.org (Postfix) with ESMTP id DEADDD1B4EC
|
|
for <pgsql-performance@postgresql.org>; Mon, 6 Oct 2003 14:01:51 -0300 (ADT)
|
|
Received: by perrin.nxad.com (Postfix, from userid 1001)
|
|
id 64CEC21068; Mon, 6 Oct 2003 10:01:36 -0700 (PDT)
|
|
Date: Mon, 6 Oct 2003 10:01:36 -0700
|
|
From: Sean Chittenden <sean@chittenden.org>
|
|
To: Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
|
|
cc: pgsql-performance@postgresql.org
|
|
Subject: Re: [PERFORM] count(*) slow on large tables
|
|
Message-ID: <20031006170136.GB94718@perrin.nxad.com>
|
|
References: <200310041819.h94IJkV07596@candle.pha.pa.us> <3F81066C.90402@persistent.co.in>
|
|
MIME-Version: 1.0
|
|
Content-Type: text/plain; charset=us-ascii
|
|
Content-Disposition: inline
|
|
In-Reply-To: <3F81066C.90402@persistent.co.in>
|
|
X-PGP-Key: finger seanc@FreeBSD.org
|
|
X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341
|
|
X-Web-Homepage: http://sean.chittenden.org/
|
|
User-Agent: Mutt/1.5.4i
|
|
X-Virus-Scanned: by amavisd-new at postgresql.org
|
|
X-Mailing-List: pgsql-performance
|
|
Precedence: bulk
|
|
Sender: pgsql-performance-owner@postgresql.org
|
|
Status: OR
|
|
|
|
> How it will help? This is in addition to trigger proposal that came
|
|
> up earlier. With triggers it's not possible to make values visible
|
|
> across backends unless trigger updates a table, which eventually
|
|
> leads to vacuum/dead tuples problem.
|
|
>
|
|
> 1. User creates a trigger to check updates/inserts for certain conditions.
|
|
> 2. It updates the count as and when required.
|
|
> 3. If the trigger detects the count is not initialized, it would issue the
|
|
> same query first time. There is no avoiding this issue.
|
|
>
|
|
> Besides providing facility of resident variables could be used
|
|
> imaginatively as well.
|
|
>
|
|
> Does this make sense? IMO this is more generalised approach over all.
|
|
|
|
I do this _VERY_ frequently in my databases, only I have my stored
|
|
procs do the aggregate in a predefined MVCC table that's always there.
|
|
Here's a denormalized version for public consumption/thought:
|
|
|
|
CREATE TABLE global.dba_aggregate_cache (
|
|
dbl TEXT NOT NULL, -- The database location, doesn't need to be
|
|
-- qualified (ex: schema.table.col)
|
|
op TEXT NOT NULL, -- The operation, SUM, COUNT, etc.
|
|
qual TEXT, -- Any kind of conditional, such as a where clause
|
|
val_int INT, -- Whatever the value is, of type INT
|
|
val_bigint BIGINT, -- Whatever the value is, of type BIGINT
|
|
val_text TEXT, -- Whatever the value is, of type TEXT
|
|
val_bytea BYTEA, -- Whatever the value is, of type BYTEA
|
|
);
|
|
CREATE UNIQUE INDEX dba_aggregate_cache_dbl_op_udx ON global.dba_aggregate_cache(dbl,op);
|
|
|
|
Then, I use a function to retrieve this value instead of a SELECT
|
|
COUNT(*).
|
|
|
|
SELECT public.cache_count('dbl','qual'); -- In this case, the op is COUNT
|
|
SELECT public.cache_count('dbl'); -- Returns the COUNT for the table listed in the dbl
|
|
|
|
Then, I create 4 or 5 functions (depends on the op I'm performing):
|
|
|
|
1) A private function that _doesn't_ run as security definer, that
|
|
populates the global.dba_aggregate_cache row if it's empty.
|
|
2) A STABLE function for SELECTs, if the row doesn't exist, then it
|
|
calls function #1 to populate its existence.
|
|
3) A STABLE function for INSERTs, if the row doesn't exist, then it
|
|
calls function #1 to populate its existence, then adds the
|
|
necessary bits to make it accurate.
|
|
4) A STABLE function for DELETEs, if the row doesn't exist, then it
|
|
calls function #1 to populate its existence, then deletes the
|
|
necessary bits to make it accurate.
|
|
5) A STABLE function for UPDATEs, if the row doesn't exist, then it
|
|
calls function #1 to populate its existence, then updates the
|
|
necessary bits to make it accurate. It's not uncommon for me to
|
|
not have an UPDATE function/trigger.
|
|
|
|
Create triggers for functions 2-5, and test away. It's MVCC,
|
|
searching through a table that's INDEX'ed for a single row is
|
|
obviously vastly faster than a seqscan/aggregate. If I need any kind
|
|
of an aggregate to be fast, I use this system with a derivation of the
|
|
above table. The problem with it being that I have to retrain others
|
|
to use cache_count(), or some other function instead of using
|
|
COUNT(*).
|
|
|
|
That said, it'd be nice if there were a way to tell PostgreSQL to do
|
|
the above for you and teach COUNT(*), SUM(*), or other aggregates to
|
|
use an MVCC backed cache similar to the above. If people want their
|
|
COUNT's to be fast, then they have to live with the INSERT, UPDATE,
|
|
DELETE cost. The above doesn't work with anything complex such as
|
|
join's, but it's certainly a start and I think satisfies everyone's
|
|
gripes other than the tuple churn that _does_ happen (*nudge nudge*,
|
|
pg_autovacuum could be integrated into the backend to handle this).
|
|
Those worried about performance, the pages that are constantly being
|
|
recycled would likely stay in disk cache (PG or the OS). There's
|
|
still some commit overhead, but still... no need to over optimize by
|
|
requiring the table to be stored in the out dated, slow, and over used
|
|
shm (also, *nudge nudge*).
|
|
|
|
Anyway, let me throw that out there as a solution that I use and it
|
|
works quite well. I didn't explain the use of the qual column, but I
|
|
think those who grasp the above way of handling things probably grok
|
|
how to use the qual column in a dynamically executed query.
|
|
|
|
CREATE AGGREGATE CACHE anyone?
|
|
|
|
-sc
|
|
|
|
--
|
|
Sean Chittenden
|
|
|
|
---------------------------(end of broadcast)---------------------------
|
|
TIP 7: don't forget to increase your free space map settings
|
|
|