2010-09-20 22:08:53 +02:00
|
|
|
<!-- doc/src/sgml/regress.sgml -->
|
2000-10-17 17:26:40 +02:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<chapter id="regress">
|
2010-04-03 09:23:02 +02:00
|
|
|
<title>Regression Tests</title>
|
2000-05-02 22:02:03 +02:00
|
|
|
|
2003-08-31 19:32:24 +02:00
|
|
|
<indexterm zone="regress">
|
|
|
|
<primary>regression tests</primary>
|
|
|
|
</indexterm>
|
|
|
|
|
|
|
|
<indexterm zone="regress">
|
|
|
|
<primary>test</primary>
|
|
|
|
</indexterm>
|
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
The regression tests are a comprehensive set of tests for the SQL
|
|
|
|
implementation in <productname>PostgreSQL</productname>. They test
|
|
|
|
standard SQL operations as well as the extended capabilities of
|
2004-12-28 20:08:58 +01:00
|
|
|
<productname>PostgreSQL</productname>.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
2001-09-21 20:37:05 +02:00
|
|
|
<sect1 id="regress-run">
|
|
|
|
<title>Running the Tests</title>
|
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2004-12-28 20:08:58 +01:00
|
|
|
The regression tests can be run against an already installed and
|
2000-10-17 17:26:40 +02:00
|
|
|
running server, or using a temporary installation within the build
|
|
|
|
tree. Furthermore, there is a <quote>parallel</quote> and a
|
|
|
|
<quote>sequential</quote> mode for running the tests. The
|
2010-02-03 18:25:06 +01:00
|
|
|
sequential method runs each test script alone, while the
|
2000-10-17 17:26:40 +02:00
|
|
|
parallel method starts up multiple server processes to run groups
|
2014-02-14 22:50:22 +01:00
|
|
|
of tests in parallel. Parallel testing adds confidence that
|
2010-02-03 18:25:06 +01:00
|
|
|
interprocess communication and locking are working correctly.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
2011-01-11 21:47:58 +01:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Running the Tests Against a Temporary Installation</title>
|
2011-01-11 21:47:58 +01:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2010-02-03 18:25:06 +01:00
|
|
|
To run the parallel regression tests after building but before installation,
|
2009-04-27 18:27:36 +02:00
|
|
|
type:
|
2000-10-17 17:26:40 +02:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check
|
2000-10-17 17:26:40 +02:00
|
|
|
</screen>
|
|
|
|
in the top-level directory. (Or you can change to
|
|
|
|
<filename>src/test/regress</filename> and run the command there.)
|
2014-02-14 22:50:22 +01:00
|
|
|
At the end you should see something like:
|
2000-10-17 17:26:40 +02:00
|
|
|
<screen>
|
|
|
|
<computeroutput>
|
2008-05-30 02:04:32 +02:00
|
|
|
=======================
|
|
|
|
All 115 tests passed.
|
|
|
|
=======================
|
2000-10-17 17:26:40 +02:00
|
|
|
</computeroutput>
|
|
|
|
</screen>
|
2003-11-02 22:56:15 +01:00
|
|
|
or otherwise a note about which tests failed. See <xref
|
2004-12-28 20:08:58 +01:00
|
|
|
linkend="regress-evaluation"> below before assuming that a
|
|
|
|
<quote>failure</> represents a serious problem.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-17 17:24:32 +01:00
|
|
|
Because this test method runs a temporary server, it will not work
|
|
|
|
if you did the build as the root user, since the server will not start as
|
|
|
|
root. Recommended procedure is not to do the build as root, or else to
|
|
|
|
perform testing after completing the installation.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
2004-12-03 18:46:19 +01:00
|
|
|
<para>
|
|
|
|
If you have configured <productname>PostgreSQL</productname> to install
|
|
|
|
into a location where an older <productname>PostgreSQL</productname>
|
2014-02-12 23:29:19 +01:00
|
|
|
installation already exists, and you perform <literal>make check</>
|
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
|
|
|
before installing the new version, you might find that the tests fail
|
2004-12-03 18:46:19 +01:00
|
|
|
because the new programs try to use the already-installed shared
|
|
|
|
libraries. (Typical symptoms are complaints about undefined symbols.)
|
|
|
|
If you wish to run the tests before overwriting the old installation,
|
|
|
|
you'll need to build with <literal>configure --disable-rpath</>.
|
|
|
|
It is not recommended that you use this option for the final installation,
|
|
|
|
however.
|
|
|
|
</para>
|
|
|
|
|
2001-12-04 02:49:17 +01:00
|
|
|
<para>
|
|
|
|
The parallel regression test starts quite a few processes under your
|
2002-01-20 23:19:57 +01:00
|
|
|
user ID. Presently, the maximum concurrency is twenty parallel test
|
2006-07-19 19:02:59 +02:00
|
|
|
scripts, which means forty processes: there's a server process and a
|
|
|
|
<application>psql</> process for each test script.
|
2001-12-04 02:49:17 +01:00
|
|
|
So if your system enforces a per-user limit on the number of processes,
|
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
|
|
|
make sure this limit is at least fifty or so, else you might get
|
2001-12-04 02:49:17 +01:00
|
|
|
random-seeming failures in the parallel test. If you are not in
|
2003-11-02 22:56:15 +01:00
|
|
|
a position to raise the limit, you can cut down the degree of parallelism
|
2009-04-27 18:27:36 +02:00
|
|
|
by setting the <literal>MAX_CONNECTIONS</> parameter. For example:
|
2003-11-02 22:56:15 +01:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make MAX_CONNECTIONS=10 check
|
2003-11-02 22:56:15 +01:00
|
|
|
</screen>
|
|
|
|
runs no more than ten tests concurrently.
|
2001-12-04 02:49:17 +01:00
|
|
|
</para>
|
2011-01-11 21:47:58 +01:00
|
|
|
</sect2>
|
|
|
|
|
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Running the Tests Against an Existing Installation</title>
|
2001-12-04 02:49:17 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
To run the tests after installation (see <xref linkend="installation">),
|
2001-09-21 20:37:05 +02:00
|
|
|
initialize a data area and start the
|
2014-02-14 22:50:22 +01:00
|
|
|
server as explained in <xref linkend="runtime">, then type:
|
2000-10-17 17:26:40 +02:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make installcheck
|
2004-12-28 20:08:58 +01:00
|
|
|
</screen>
|
2009-04-27 18:27:36 +02:00
|
|
|
or for a parallel test:
|
2004-12-28 20:08:58 +01:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make installcheck-parallel
|
2000-10-17 17:26:40 +02:00
|
|
|
</screen>
|
2001-03-20 01:09:36 +01:00
|
|
|
The tests will expect to contact the server at the local host and the
|
2004-12-28 20:08:58 +01:00
|
|
|
default port number, unless directed otherwise by <envar>PGHOST</envar> and
|
2014-02-03 16:27:47 +01:00
|
|
|
<envar>PGPORT</envar> environment variables. The tests will be run in a
|
|
|
|
database named <literal>regression</>; any existing database by this name
|
|
|
|
will be dropped.
|
2014-02-14 22:50:22 +01:00
|
|
|
The tests will also transiently create some cluster-wide objects, such as
|
|
|
|
user identities named <literal>regressuser<replaceable>N</></literal>.
|
2005-10-14 01:41:07 +02:00
|
|
|
</para>
|
2011-01-11 21:47:58 +01:00
|
|
|
</sect2>
|
|
|
|
|
|
|
|
<sect2>
|
2014-02-14 22:50:22 +01:00
|
|
|
<title>Additional Test Suites</title>
|
2010-05-02 14:22:40 +02:00
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
The <literal>make check</> and <literal>make installcheck</> commands
|
|
|
|
run only the <quote>core</> regression tests, which test built-in
|
|
|
|
functionality of the <productname>PostgreSQL</> server. The source
|
|
|
|
distribution also contains additional test suites, most of them having
|
|
|
|
to do with add-on functionality such as optional procedural languages.
|
2010-05-02 14:22:40 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
To run all test suites applicable to the modules that have been selected
|
|
|
|
to be built, including the core tests, type one of these commands at the
|
|
|
|
top of the build tree:
|
2010-05-02 14:22:40 +02:00
|
|
|
<screen>
|
2014-02-14 22:50:22 +01:00
|
|
|
make check-world
|
|
|
|
make installcheck-world
|
2010-05-02 14:22:40 +02:00
|
|
|
</screen>
|
2014-02-14 22:50:22 +01:00
|
|
|
These commands run the tests using temporary servers or an
|
|
|
|
already-installed server, respectively, just as previously explained
|
|
|
|
for <literal>make check</> and <literal>make installcheck</>. Other
|
|
|
|
considerations are the same as previously explained for each method.
|
|
|
|
Note that <literal>make check-world</> builds a separate temporary
|
|
|
|
installation tree for each tested module, so it requires a great deal
|
|
|
|
more time and disk space than <literal>make installcheck-world</>.
|
2010-05-02 14:22:40 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
Alternatively, you can run individual test suites by typing
|
|
|
|
<literal>make check</> or <literal>make installcheck</> in the appropriate
|
|
|
|
subdirectory of the build tree. Keep in mind that <literal>make
|
|
|
|
installcheck</> assumes you've installed the relevant module(s), not
|
|
|
|
only the core server.
|
2010-05-02 14:22:40 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
The additional tests that can be invoked this way include:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Regression tests for optional procedural languages (other than
|
|
|
|
<application>PL/pgSQL</>, which is tested by the core tests).
|
|
|
|
These are located under <filename>src/pl</>.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Regression tests for <filename>contrib</> modules,
|
|
|
|
located under <filename>contrib</>.
|
|
|
|
Not all <filename>contrib</> modules have tests.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Regression tests for the ECPG interface library,
|
|
|
|
located in <filename>src/interfaces/ecpg/test</>.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Tests stressing behavior of concurrent sessions,
|
|
|
|
located in <filename>src/test/isolation</>.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
2014-04-15 03:33:46 +02:00
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
Tests of client programs under <filename>src/bin</filename>. See
|
|
|
|
also <xref linkend="regress-tap">.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
2014-02-14 22:50:22 +01:00
|
|
|
</itemizedlist>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When using <literal>installcheck</> mode, these tests will destroy any
|
|
|
|
existing databases named <literal>pl_regression</>,
|
|
|
|
<literal>contrib_regression</>, <literal>isolationtest</>,
|
|
|
|
<literal>regress1</>, or <literal>connectdb</>, as well as
|
|
|
|
<literal>regression</>.
|
2010-05-02 14:22:40 +02:00
|
|
|
</para>
|
2011-01-11 21:47:58 +01:00
|
|
|
</sect2>
|
2011-01-13 08:13:12 +01:00
|
|
|
|
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Locale and Encoding</title>
|
2011-01-13 08:13:12 +01:00
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
By default, tests using a temporary installation use the
|
2011-01-13 08:13:12 +01:00
|
|
|
locale defined in the current environment and the corresponding
|
|
|
|
database encoding as determined by <command>initdb</command>. It
|
|
|
|
can be useful to test different locales by setting the appropriate
|
|
|
|
environment variables, for example:
|
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check LANG=C
|
|
|
|
make check LC_COLLATE=en_US.utf8 LC_CTYPE=fr_CA.utf8
|
2011-01-13 08:13:12 +01:00
|
|
|
</screen>
|
|
|
|
For implementation reasons, setting <envar>LC_ALL</envar> does not
|
|
|
|
work for this purpose; all the other locale-related environment
|
|
|
|
variables do work.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When testing against an existing installation, the locale is
|
|
|
|
determined by the existing database cluster and cannot be set
|
|
|
|
separately for the test run.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
You can also choose the database encoding explicitly by setting
|
2011-04-15 07:42:05 +02:00
|
|
|
the variable <envar>ENCODING</envar>, for example:
|
2011-01-13 08:13:12 +01:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check LANG=C ENCODING=EUC_JP
|
2011-01-13 08:13:12 +01:00
|
|
|
</screen>
|
|
|
|
Setting the database encoding this way typically only makes sense
|
|
|
|
if the locale is C; otherwise the encoding is chosen automatically
|
|
|
|
from the locale, and specifying an encoding that does not match
|
|
|
|
the locale will result in an error.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
The database encoding can be set for tests against either a temporary or
|
|
|
|
an existing installation, though in the latter case it must be
|
|
|
|
compatible with the installation's locale.
|
2011-01-13 08:13:12 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
2011-02-08 22:04:18 +01:00
|
|
|
|
|
|
|
<sect2>
|
2011-04-22 23:43:18 +02:00
|
|
|
<title>Extra Tests</title>
|
2011-02-08 22:04:18 +01:00
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
The core regression test suite contains a few test files that are not
|
2011-02-08 22:04:18 +01:00
|
|
|
run by default, because they might be platform-dependent or take a
|
|
|
|
very long time to run. You can run these or other extra test
|
|
|
|
files by setting the variable <envar>EXTRA_TESTS</envar>. For
|
|
|
|
example, to run the <literal>numeric_big</literal> test:
|
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check EXTRA_TESTS=numeric_big
|
2011-02-08 22:04:18 +01:00
|
|
|
</screen>
|
|
|
|
To run the collation tests:
|
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check EXTRA_TESTS=collate.linux.utf8 LANG=en_US.utf8
|
2011-02-08 22:04:18 +01:00
|
|
|
</screen>
|
2011-04-22 23:43:18 +02:00
|
|
|
The <literal>collate.linux.utf8</> test works only on Linux/glibc
|
2011-04-23 18:51:47 +02:00
|
|
|
platforms, and only when run in a database that uses UTF-8 encoding.
|
2011-02-08 22:04:18 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
2014-02-14 22:50:22 +01:00
|
|
|
|
|
|
|
<sect2>
|
|
|
|
<title>Testing Hot Standby</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The source distribution also contains regression tests for the static
|
|
|
|
behavior of Hot Standby. These tests require a running primary server
|
|
|
|
and a running standby server that is accepting new WAL changes from the
|
|
|
|
primary (using either file-based log shipping or streaming replication).
|
|
|
|
Those servers are not automatically created for you, nor is replication
|
|
|
|
setup documented here. Please check the various sections of the
|
|
|
|
documentation devoted to the required commands and related issues.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To run the Hot Standby tests, first create a database
|
|
|
|
called <literal>regression</> on the primary:
|
|
|
|
<screen>
|
|
|
|
psql -h primary -c "CREATE DATABASE regression"
|
|
|
|
</screen>
|
|
|
|
Next, run the preparatory script
|
|
|
|
<filename>src/test/regress/sql/hs_primary_setup.sql</filename>
|
|
|
|
on the primary in the regression database, for example:
|
|
|
|
<screen>
|
|
|
|
psql -h primary -f src/test/regress/sql/hs_primary_setup.sql regression
|
|
|
|
</screen>
|
|
|
|
Allow these changes to propagate to the standby.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Now arrange for the default database connection to be to the standby
|
|
|
|
server under test (for example, by setting the <envar>PGHOST</envar> and
|
|
|
|
<envar>PGPORT</envar> environment variables).
|
|
|
|
Finally, run <literal>make standbycheck</> in the regression directory:
|
|
|
|
<screen>
|
|
|
|
cd src/test/regress
|
|
|
|
make standbycheck
|
|
|
|
</screen>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Some extreme behaviors can also be generated on the primary using the
|
|
|
|
script <filename>src/test/regress/sql/hs_primary_extremes.sql</filename>
|
|
|
|
to allow the behavior of the standby to be tested.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
2001-09-21 20:37:05 +02:00
|
|
|
</sect1>
|
2000-10-17 17:26:40 +02:00
|
|
|
|
|
|
|
<sect1 id="regress-evaluation">
|
2010-11-23 21:27:50 +01:00
|
|
|
<title>Test Evaluation</title>
|
2000-05-02 22:02:03 +02:00
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Some properly installed and fully functional
|
|
|
|
<productname>PostgreSQL</productname> installations can
|
|
|
|
<quote>fail</quote> some of these regression tests due to
|
2002-01-07 03:29:15 +01:00
|
|
|
platform-specific artifacts such as varying floating-point representation
|
2005-07-24 19:07:18 +02:00
|
|
|
and message wording. The tests are currently evaluated using a simple
|
2003-03-13 02:30:29 +01:00
|
|
|
<command>diff</command> comparison against the outputs
|
2000-10-17 17:26:40 +02:00
|
|
|
generated on a reference system, so the results are sensitive to
|
|
|
|
small system differences. When a test is reported as
|
|
|
|
<quote>failed</quote>, always examine the differences between
|
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
|
|
|
expected and actual results; you might find that the
|
2000-10-17 17:26:40 +02:00
|
|
|
differences are not significant. Nonetheless, we still strive to
|
|
|
|
maintain accurate reference files across all supported platforms,
|
|
|
|
so it can be expected that all tests pass.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
The actual outputs of the regression tests are in files in the
|
|
|
|
<filename>src/test/regress/results</filename> directory. The test
|
2003-03-13 02:30:29 +01:00
|
|
|
script uses <command>diff</command> to compare each output
|
2000-10-17 17:26:40 +02:00
|
|
|
file against the reference outputs stored in the
|
|
|
|
<filename>src/test/regress/expected</filename> directory. Any
|
|
|
|
differences are saved for your inspection in
|
2014-02-14 22:50:22 +01:00
|
|
|
<filename>src/test/regress/regression.diffs</filename>.
|
|
|
|
(When running a test suite other than the core tests, these files
|
|
|
|
of course appear in the relevant subdirectory,
|
|
|
|
not <filename>src/test/regress</>.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If you don't
|
2013-01-30 04:58:38 +01:00
|
|
|
like the <command>diff</command> options that are used by default, set the
|
|
|
|
environment variable <envar>PG_REGRESS_DIFF_OPTS</envar>, for
|
|
|
|
instance <literal>PG_REGRESS_DIFF_OPTS='-u'</literal>. (Or you
|
2003-03-13 02:30:29 +01:00
|
|
|
can run <command>diff</command> yourself, if you prefer.)
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
|
2005-07-24 19:07:18 +02:00
|
|
|
<para>
|
|
|
|
If for some reason a particular platform generates a <quote>failure</>
|
|
|
|
for a given test, but inspection of the output convinces you that
|
|
|
|
the result is valid, you can add a new comparison file to silence
|
|
|
|
the failure report in future test runs. See
|
|
|
|
<xref linkend="regress-variant"> for details.
|
|
|
|
</para>
|
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Error Message Differences</title>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Some of the regression tests involve intentional invalid input
|
|
|
|
values. Error messages can come from either the
|
|
|
|
<productname>PostgreSQL</productname> code or from the host
|
Update documentation on may/can/might:
Standard English uses "may", "can", and "might" in different ways:
may - permission, "You may borrow my rake."
can - ability, "I can lift that log."
might - possibility, "It might rain today."
Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice. Similarly, "It may crash" is better stated, "It might crash".
Also update two error messages mentioned in the documenation to match.
2007-01-31 21:56:20 +01:00
|
|
|
platform system routines. In the latter case, the messages can
|
2000-10-17 17:26:40 +02:00
|
|
|
vary between platforms, but should reflect similar
|
|
|
|
information. These differences in messages will result in a
|
2000-12-22 19:57:50 +01:00
|
|
|
<quote>failed</quote> regression test that can be validated by
|
2000-10-17 17:26:40 +02:00
|
|
|
inspection.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
</sect2>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2001-03-20 01:09:36 +01:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Locale Differences</title>
|
2001-03-20 01:09:36 +01:00
|
|
|
|
|
|
|
<para>
|
2009-02-11 15:03:42 +01:00
|
|
|
If you run the tests against a server that was
|
2003-11-02 22:56:15 +01:00
|
|
|
initialized with a collation-order locale other than C, then
|
2010-02-03 18:25:06 +01:00
|
|
|
there might be differences due to sort order and subsequent
|
2002-05-14 15:05:43 +02:00
|
|
|
failures. The regression test suite is set up to handle this
|
2010-02-03 18:25:06 +01:00
|
|
|
problem by providing alternate result files that together are
|
2005-07-24 19:07:18 +02:00
|
|
|
known to handle a large number of locales.
|
2001-03-20 01:09:36 +01:00
|
|
|
</para>
|
2009-02-11 15:03:42 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
To run the tests in a different locale when using the
|
|
|
|
temporary-installation method, pass the appropriate
|
|
|
|
locale-related environment variables on
|
|
|
|
the <command>make</command> command line, for example:
|
|
|
|
<programlisting>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check LANG=de_DE.utf8
|
2009-02-11 15:03:42 +01:00
|
|
|
</programlisting>
|
2009-02-12 14:26:03 +01:00
|
|
|
(The regression test driver unsets <envar>LC_ALL</envar>, so it
|
|
|
|
does not work to choose the locale using that variable.) To use
|
|
|
|
no locale, either unset all locale-related environment variables
|
|
|
|
(or set them to <literal>C</literal>) or use the following
|
|
|
|
special invocation:
|
2009-02-11 15:03:42 +01:00
|
|
|
<programlisting>
|
2014-02-12 23:29:19 +01:00
|
|
|
make check NO_LOCALE=1
|
2009-02-11 15:03:42 +01:00
|
|
|
</programlisting>
|
|
|
|
When running the tests against an existing installation, the
|
|
|
|
locale setup is determined by the existing installation. To
|
|
|
|
change it, initialize the database cluster with a different
|
|
|
|
locale by passing the appropriate options
|
|
|
|
to <command>initdb</command>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 22:50:22 +01:00
|
|
|
In general, it is advisable to try to run the
|
2009-02-11 15:03:42 +01:00
|
|
|
regression tests in the locale setup that is wanted for
|
|
|
|
production use, as this will exercise the locale- and
|
|
|
|
encoding-related code portions that will actually be used in
|
|
|
|
production. Depending on the operating system environment, you
|
|
|
|
might get failures, but then you will at least know what
|
|
|
|
locale-specific behaviors to expect when running real
|
|
|
|
applications.
|
|
|
|
</para>
|
2001-03-20 01:09:36 +01:00
|
|
|
</sect2>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Date and Time Differences</title>
|
2001-08-07 00:53:26 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Most of the date and time results are dependent on the time zone
|
|
|
|
environment. The reference files are generated for time zone
|
2004-08-09 07:34:39 +02:00
|
|
|
<literal>PST8PDT</literal> (Berkeley, California), and there will be
|
|
|
|
apparent failures if the tests are not run with that time zone setting.
|
2000-10-17 17:26:40 +02:00
|
|
|
The regression test driver sets environment variable
|
2001-08-07 00:53:26 +02:00
|
|
|
<envar>PGTZ</envar> to <literal>PST8PDT</literal>, which normally
|
2004-08-09 07:34:39 +02:00
|
|
|
ensures proper results.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
</sect2>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Floating-Point Differences</title>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
2003-03-13 02:30:29 +01:00
|
|
|
Some of the tests involve computing 64-bit floating-point numbers (<type>double
|
|
|
|
precision</type>) from table columns. Differences in
|
2000-10-17 17:26:40 +02:00
|
|
|
results involving mathematical functions of <type>double
|
2002-11-08 21:26:12 +01:00
|
|
|
precision</type> columns have been observed. The <literal>float8</> and
|
|
|
|
<literal>geometry</> tests are particularly prone to small differences
|
2014-02-14 22:50:22 +01:00
|
|
|
across platforms, or even with different compiler optimization settings.
|
2000-10-17 17:26:40 +02:00
|
|
|
Human eyeball comparison is needed to determine the real
|
|
|
|
significance of these differences which are usually 10 places to
|
|
|
|
the right of the decimal point.
|
|
|
|
</para>
|
1999-05-20 04:46:40 +02:00
|
|
|
|
2002-11-08 21:26:12 +01:00
|
|
|
<para>
|
|
|
|
Some systems display minus zero as <literal>-0</>, while others
|
|
|
|
just show <literal>0</>.
|
|
|
|
</para>
|
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Some systems signal errors from <function>pow()</function> and
|
|
|
|
<function>exp()</function> differently from the mechanism
|
|
|
|
expected by the current <productname>PostgreSQL</productname>
|
|
|
|
code.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
2001-01-02 06:56:02 +01:00
|
|
|
|
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Row Ordering Differences</title>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2001-01-02 06:56:02 +01:00
|
|
|
<para>
|
2001-09-21 20:37:05 +02:00
|
|
|
You might see differences in which the same rows are output in a
|
2001-01-02 06:56:02 +01:00
|
|
|
different order than what appears in the expected file. In most cases
|
|
|
|
this is not, strictly speaking, a bug. Most of the regression test
|
2003-03-13 02:30:29 +01:00
|
|
|
scripts are not so pedantic as to use an <literal>ORDER BY</> for every single
|
|
|
|
<literal>SELECT</>, and so their result row orderings are not well-defined
|
2010-02-03 18:25:06 +01:00
|
|
|
according to the SQL specification. In practice, since we are
|
2001-01-02 06:56:02 +01:00
|
|
|
looking at the same queries being executed on the same data by the same
|
2010-02-03 18:25:06 +01:00
|
|
|
software, we usually get the same result ordering on all platforms,
|
|
|
|
so the lack of <literal>ORDER BY</> is not a problem. Some queries do exhibit
|
2005-03-07 03:00:28 +01:00
|
|
|
cross-platform ordering differences, however. When testing against an
|
|
|
|
already-installed server, ordering differences can also be caused by
|
|
|
|
non-C locale settings or non-default parameter settings, such as custom values
|
|
|
|
of <varname>work_mem</> or the planner cost parameters.
|
2001-01-02 06:56:02 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Therefore, if you see an ordering difference, it's not something to
|
2005-03-07 03:00:28 +01:00
|
|
|
worry about, unless the query does have an <literal>ORDER BY</> that your
|
2010-02-03 18:25:06 +01:00
|
|
|
result is violating. However, please report it anyway, so that we can add an
|
|
|
|
<literal>ORDER BY</> to that particular query to eliminate the bogus
|
2001-01-02 06:56:02 +01:00
|
|
|
<quote>failure</quote> in future releases.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2005-03-07 03:00:28 +01:00
|
|
|
You might wonder why we don't order all the regression test queries explicitly
|
|
|
|
to get rid of this issue once and for all. The reason is that that would
|
2001-01-02 06:56:02 +01:00
|
|
|
make the regression tests less useful, not more, since they'd tend
|
|
|
|
to exercise query plan types that produce ordered results to the
|
|
|
|
exclusion of those that don't.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2005-10-18 23:43:33 +02:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>Insufficient Stack Depth</title>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2005-10-18 23:43:33 +02:00
|
|
|
<para>
|
|
|
|
If the <literal>errors</literal> test results in a server crash
|
|
|
|
at the <literal>select infinite_recurse()</> command, it means that
|
|
|
|
the platform's limit on process stack size is smaller than the
|
2014-02-14 22:50:22 +01:00
|
|
|
<xref linkend="guc-max-stack-depth"> parameter indicates. This
|
2006-06-18 17:38:37 +02:00
|
|
|
can be fixed by running the server under a higher stack
|
2010-11-23 21:27:50 +01:00
|
|
|
size limit (4MB is recommended with the default value of
|
2005-10-18 23:43:33 +02:00
|
|
|
<varname>max_stack_depth</>). If you are unable to do that, an
|
|
|
|
alternative is to reduce the value of <varname>max_stack_depth</>.
|
|
|
|
</para>
|
2014-02-14 22:50:22 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
On platforms supporting <function>getrlimit()</>, the server should
|
|
|
|
automatically choose a safe value of <varname>max_stack_depth</>;
|
|
|
|
so unless you've manually overridden this setting, a failure of this
|
|
|
|
kind is a reportable bug.
|
|
|
|
</para>
|
2005-10-18 23:43:33 +02:00
|
|
|
</sect2>
|
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
2011-01-29 19:00:18 +01:00
|
|
|
<title>The <quote>random</quote> Test</title>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
2010-11-23 21:27:50 +01:00
|
|
|
The <literal>random</literal> test script is intended to produce
|
2014-02-14 22:50:22 +01:00
|
|
|
random results. In very rare cases, this causes that regression
|
2007-02-01 01:28:19 +01:00
|
|
|
test to fail. Typing:
|
2000-10-17 17:26:40 +02:00
|
|
|
<programlisting>
|
|
|
|
diff results/random.out expected/random.out
|
|
|
|
</programlisting>
|
|
|
|
should produce only one or a few lines of differences. You need
|
2004-12-28 20:08:58 +01:00
|
|
|
not worry unless the random test fails repeatedly.
|
2000-10-17 17:26:40 +02:00
|
|
|
</para>
|
|
|
|
</sect2>
|
2014-02-03 16:27:47 +01:00
|
|
|
|
|
|
|
<sect2>
|
|
|
|
<title>Configuration Parameters</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When running the tests against an existing installation, some non-default
|
2014-02-04 02:10:48 +01:00
|
|
|
parameter settings could cause the tests to fail. For example, changing
|
|
|
|
parameters such as <varname>enable_seqscan</varname> or
|
|
|
|
<varname>enable_indexscan</varname> could cause plan changes that would
|
2014-02-14 22:50:22 +01:00
|
|
|
affect the results of tests that use <command>EXPLAIN</>.
|
2014-02-03 16:27:47 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
2000-05-02 22:02:03 +02:00
|
|
|
</sect1>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<!-- We might want to move the following section into the developer's guide. -->
|
2005-07-24 19:07:18 +02:00
|
|
|
<sect1 id="regress-variant">
|
|
|
|
<title>Variant Comparison Files</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Since some of the tests inherently produce environment-dependent
|
2010-02-03 18:25:06 +01:00
|
|
|
results, we have provided ways to specify alternate <quote>expected</>
|
2005-07-24 19:07:18 +02:00
|
|
|
result files. Each regression test can have several comparison files
|
|
|
|
showing possible results on different platforms. There are two
|
|
|
|
independent mechanisms for determining which comparison file is used
|
|
|
|
for each test.
|
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
2005-07-24 19:07:18 +02:00
|
|
|
The first mechanism allows comparison files to be selected for
|
|
|
|
specific platforms. There is a mapping file,
|
|
|
|
<filename>src/test/regress/resultmap</filename>, that defines
|
|
|
|
which comparison file to use for each platform.
|
|
|
|
To eliminate bogus test <quote>failures</quote> for a particular platform,
|
|
|
|
you first choose or make a variant result file, and then add a line to the
|
|
|
|
<filename>resultmap</filename> file.
|
2000-11-22 00:40:28 +01:00
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
|
|
|
Each line in the mapping file is of the form
|
|
|
|
<synopsis>
|
2007-06-12 19:49:12 +02:00
|
|
|
testname:output:platformpattern=comparisonfilename
|
2000-11-22 00:40:28 +01:00
|
|
|
</synopsis>
|
|
|
|
The test name is just the name of the particular regression test
|
2010-11-23 21:27:50 +01:00
|
|
|
module. The output value indicates which output file to check. For the
|
2007-06-12 19:49:12 +02:00
|
|
|
standard regression tests, this is always <literal>out</literal>. The
|
|
|
|
value corresponds to the file extension of the output file.
|
|
|
|
The platform pattern is a pattern in the style of the Unix
|
2003-03-13 02:30:29 +01:00
|
|
|
tool <command>expr</> (that is, a regular expression with an implicit
|
2006-07-19 04:37:00 +02:00
|
|
|
<literal>^</literal> anchor at the start). It is matched against the
|
|
|
|
platform name as printed by <command>config.guess</command>.
|
|
|
|
The comparison file name is the base name of the substitute result
|
|
|
|
comparison file.
|
2000-11-22 00:40:28 +01:00
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
2004-05-21 07:08:06 +02:00
|
|
|
For example: some systems interpret very small floating-point values
|
|
|
|
as zero, rather than reporting an underflow error. This causes a
|
|
|
|
few differences in the <filename>float8</> regression test.
|
2000-11-22 00:40:28 +01:00
|
|
|
Therefore, we provide a variant comparison file,
|
2004-05-21 07:08:06 +02:00
|
|
|
<filename>float8-small-is-zero.out</filename>, which includes
|
2001-08-07 00:53:26 +02:00
|
|
|
the results to be expected on these systems. To silence the bogus
|
2004-05-21 07:08:06 +02:00
|
|
|
<quote>failure</quote> message on <systemitem>OpenBSD</systemitem>
|
2007-02-01 01:28:19 +01:00
|
|
|
platforms, <filename>resultmap</filename> includes:
|
2000-11-22 00:40:28 +01:00
|
|
|
<programlisting>
|
2007-06-12 19:49:12 +02:00
|
|
|
float8:out:i.86-.*-openbsd=float8-small-is-zero.out
|
2000-11-22 00:40:28 +01:00
|
|
|
</programlisting>
|
2010-02-03 18:25:06 +01:00
|
|
|
which will trigger on any machine where the output of
|
2004-05-21 07:08:06 +02:00
|
|
|
<command>config.guess</command> matches <literal>i.86-.*-openbsd</literal>.
|
2003-09-13 00:17:24 +02:00
|
|
|
Other lines
|
2001-09-10 01:52:12 +02:00
|
|
|
in <filename>resultmap</> select the variant comparison file for other
|
2000-11-22 00:40:28 +01:00
|
|
|
platforms where it's appropriate.
|
|
|
|
</para>
|
2005-07-24 19:07:18 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
The second selection mechanism for variant comparison files is
|
|
|
|
much more automatic: it simply uses the <quote>best match</> among
|
|
|
|
several supplied comparison files. The regression test driver
|
|
|
|
script considers both the standard comparison file for a test,
|
|
|
|
<literal><replaceable>testname</>.out</>, and variant files named
|
|
|
|
<literal><replaceable>testname</>_<replaceable>digit</>.out</>
|
|
|
|
(where the <replaceable>digit</> is any single digit
|
|
|
|
<literal>0</>-<literal>9</>). If any such file is an exact match,
|
|
|
|
the test is considered to pass; otherwise, the one that generates
|
|
|
|
the shortest diff is used to create the failure report. (If
|
|
|
|
<filename>resultmap</filename> includes an entry for the particular
|
|
|
|
test, then the base <replaceable>testname</> is the substitute
|
|
|
|
name given in <filename>resultmap</filename>.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
For example, for the <literal>char</literal> test, the comparison file
|
|
|
|
<filename>char.out</filename> contains results that are expected
|
|
|
|
in the <literal>C</> and <literal>POSIX</> locales, while
|
|
|
|
the file <filename>char_1.out</filename> contains results sorted as
|
|
|
|
they appear in many other locales.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The best-match mechanism was devised to cope with locale-dependent
|
|
|
|
results, but it can be used in any situation where the test results
|
|
|
|
cannot be predicted easily from the platform name alone. A limitation of
|
|
|
|
this mechanism is that the test driver cannot tell which variant is
|
|
|
|
actually <quote>correct</> for the current environment; it will just pick
|
|
|
|
the variant that seems to work best. Therefore it is safest to use this
|
|
|
|
mechanism only for variant results that you are willing to consider
|
|
|
|
equally valid in all contexts.
|
|
|
|
</para>
|
2010-11-23 21:27:50 +01:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
</sect1>
|
2008-09-05 14:11:18 +02:00
|
|
|
|
2014-04-15 03:33:46 +02:00
|
|
|
<sect1 id="regress-tap">
|
|
|
|
<title>TAP Tests</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The client program tests under <filename>src/bin</filename> use the Perl
|
|
|
|
TAP tools and are run by <command>prove</command>. You can pass
|
|
|
|
command-line options to <command>prove</command> by setting
|
|
|
|
the <command>make</command> variable <varname>PROVE_FLAGS</>, for example:
|
|
|
|
<programlisting>
|
|
|
|
make -C src/bin check PROVE_FLAGS='--reverse'
|
|
|
|
</programlisting>
|
|
|
|
The default is <literal>--verbose</literal>. See the manual page
|
|
|
|
of <command>prove</command> for more information.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The tests written in Perl require the Perl
|
2014-11-02 15:14:36 +01:00
|
|
|
module <literal>IPC::Run</literal>.
|
2014-04-15 03:33:46 +02:00
|
|
|
This module is available from CPAN or an operating system package.
|
|
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
|
2008-09-05 14:11:18 +02:00
|
|
|
<sect1 id="regress-coverage">
|
|
|
|
<title>Test Coverage Examination</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The PostgreSQL source code can be compiled with coverage testing
|
|
|
|
instrumentation, so that it becomes possible to examine which
|
|
|
|
parts of the code are covered by the regression tests or any other
|
|
|
|
test suite that is run with the code. This is currently supported
|
|
|
|
when compiling with GCC and requires the <command>gcov</command>
|
|
|
|
and <command>lcov</command> programs.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
A typical workflow would look like this:
|
|
|
|
<screen>
|
|
|
|
./configure --enable-coverage ... OTHER OPTIONS ...
|
2014-02-12 23:29:19 +01:00
|
|
|
make
|
|
|
|
make check # or other test suite
|
|
|
|
make coverage-html
|
2008-09-05 14:11:18 +02:00
|
|
|
</screen>
|
|
|
|
Then point your HTML browser
|
|
|
|
to <filename>coverage/index.html</filename>.
|
2014-02-12 23:29:19 +01:00
|
|
|
The <command>make</command> commands also work in subdirectories.
|
2008-09-05 14:11:18 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2009-04-27 18:27:36 +02:00
|
|
|
To reset the execution counts between test runs, run:
|
2008-09-05 14:11:18 +02:00
|
|
|
<screen>
|
2014-02-12 23:29:19 +01:00
|
|
|
make coverage-clean
|
2008-09-05 14:11:18 +02:00
|
|
|
</screen>
|
|
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
</chapter>
|