2000-11-22 00:40:28 +01:00
|
|
|
<!-- $Header: /cvsroot/pgsql/doc/src/sgml/regress.sgml,v 1.13 2000/11/21 23:40:27 petere Exp $ -->
|
2000-10-17 17:26:40 +02:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<chapter id="regress">
|
2000-10-17 17:26:40 +02:00
|
|
|
<title id="regress-title">Regression Tests</title>
|
2000-05-02 22:02:03 +02:00
|
|
|
|
|
|
|
<abstract>
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Regression test instructions and analysis
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
</abstract>
|
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
The regression tests are a comprehensive set of tests for the SQL
|
|
|
|
implementation in <productname>PostgreSQL</productname>. They test
|
|
|
|
standard SQL operations as well as the extended capabilities of
|
|
|
|
<productname>PostgreSQL</productname>. The test suite was
|
|
|
|
originally developed by Jolly Chen and Andrew Yu, and was
|
|
|
|
extensively revised and repackaged by Marc Fournier and Thomas
|
|
|
|
Lockhart. From <productname>PostgreSQL</productname> 6.1 onward
|
|
|
|
the regression tests are current for every official release.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
The regression test can be run against an already installed and
|
|
|
|
running server, or using a temporary installation within the build
|
|
|
|
tree. Furthermore, there is a <quote>parallel</quote> and a
|
|
|
|
<quote>sequential</quote> mode for running the tests. The
|
|
|
|
sequential method runs each test script in turn, whereas the
|
|
|
|
parallel method starts up multiple server processes to run groups
|
|
|
|
of tests in parallel. Parallel testing gives confidence that
|
|
|
|
interprocess communication and locking are working correctly. For
|
|
|
|
historical reasons, the sequential test is usually run against an
|
2000-10-22 21:11:05 +02:00
|
|
|
existing installation and the parallel method against a temporary
|
|
|
|
installation, but there are no technical reasons for this.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
To run the regression tests after building but before installation,
|
|
|
|
type
|
|
|
|
<screen>
|
|
|
|
<prompt>$ </prompt><userinput>gmake check</userinput>
|
|
|
|
</screen>
|
|
|
|
in the top-level directory. (Or you can change to
|
|
|
|
<filename>src/test/regress</filename> and run the command there.)
|
|
|
|
This will first build several auxiliary files, such as
|
|
|
|
platform-dependent <quote>expected</quote> files and some sample
|
|
|
|
user-defined trigger functions, and then run the test driver
|
|
|
|
script. At the end you should see something like
|
|
|
|
<screen>
|
|
|
|
<computeroutput>
|
|
|
|
======================
|
|
|
|
All 75 tests passed.
|
|
|
|
======================
|
|
|
|
</computeroutput>
|
|
|
|
</screen>
|
|
|
|
or otherwise a note about what tests failed. See <xref
|
|
|
|
linkend="regress-evaluation"> below for more.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<note>
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Because this test method runs a temporary server, it will not work
|
|
|
|
when you are the root user (the server will not start as root).
|
|
|
|
If you already did the build as root, you do not have to start all
|
|
|
|
over. Instead, make the regression test directory writable by
|
|
|
|
some other user, log in as that user, and restart the tests.
|
|
|
|
<screen>
|
|
|
|
<prompt>root# </prompt><userinput>chmod -R a+w src/test/regress</userinput>
|
|
|
|
<prompt>root# </prompt><userinput>su - joeuser</userinput>
|
|
|
|
<prompt>joeuser$ </prompt><userinput>gmake check</userinput>
|
|
|
|
</screen>
|
|
|
|
(The only possible <quote>security risk</quote> here is that other
|
|
|
|
users might be able to alter the regression test results behind
|
|
|
|
your back. Use common sense when managing user permissions.)
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Alternatively, run the tests after installation.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
</note>
|
2000-05-02 22:02:03 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<tip>
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
On some systems, the default Bourne-compatible shell
|
|
|
|
(<filename>/bin/sh</filename>) gets confused when it has to manage
|
|
|
|
too many child processes in parallel. This may cause the parallel
|
|
|
|
test run to lock up or fail. In such cases, specify a different
|
|
|
|
Bourne-compatible shell on the command line, for example:
|
|
|
|
<informalexample>
|
|
|
|
<screen>
|
|
|
|
<prompt>$ </prompt><userinput>gmake SHELL=/bin/ksh check</userinput>
|
|
|
|
</screen>
|
|
|
|
</informalexample>
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
</tip>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To run the tests after installation (see <xref
|
|
|
|
linkend="installation">), initialize a data area and start the
|
|
|
|
server, as explained in <xref linkend="runtime">, then type
|
|
|
|
<screen>
|
|
|
|
<prompt>$ </prompt><userinput>gmake installcheck</userinput>
|
|
|
|
</screen>
|
|
|
|
The server is expected to be running on the local host with the
|
|
|
|
default port number.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<sect1 id="regress-evaluation">
|
|
|
|
<title>Test Evaluation</title>
|
2000-05-02 22:02:03 +02:00
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Some properly installed and fully functional
|
|
|
|
<productname>PostgreSQL</productname> installations can
|
|
|
|
<quote>fail</quote> some of these regression tests due to
|
|
|
|
artifacts of floating point representation and time zone
|
|
|
|
support. The tests are currently evaluated using a simple
|
|
|
|
<application>diff</application> comparison against the outputs
|
|
|
|
generated on a reference system, so the results are sensitive to
|
|
|
|
small system differences. When a test is reported as
|
|
|
|
<quote>failed</quote>, always examine the differences between
|
|
|
|
expected and actual results; you may well find that the
|
|
|
|
differences are not significant. Nonetheless, we still strive to
|
|
|
|
maintain accurate reference files across all supported platforms,
|
|
|
|
so it can be expected that all tests pass.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
The actual outputs of the regression tests are in files in the
|
|
|
|
<filename>src/test/regress/results</filename> directory. The test
|
|
|
|
script uses <application>diff</application> to compare each output
|
|
|
|
file against the reference outputs stored in the
|
|
|
|
<filename>src/test/regress/expected</filename> directory. Any
|
|
|
|
differences are saved for your inspection in
|
|
|
|
<filename>src/test/regress/regression.diffs</filename>. (Or you
|
|
|
|
can run <application>diff</application> yourself, if you prefer.)
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
|
|
|
|
<sect2>
|
|
|
|
<title>Error message differences</title>
|
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Some of the regression tests involve intentional invalid input
|
|
|
|
values. Error messages can come from either the
|
|
|
|
<productname>PostgreSQL</productname> code or from the host
|
|
|
|
platform system routines. In the latter case, the messages may
|
|
|
|
vary between platforms, but should reflect similar
|
|
|
|
information. These differences in messages will result in a
|
|
|
|
<quote>failed</quote> regression test which can be validated by
|
|
|
|
inspection.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
</sect2>
|
1999-05-20 04:46:40 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
|
|
|
<title>Date and time differences</title>
|
2000-03-26 09:01:19 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Most of the date and time results are dependent on the time zone
|
|
|
|
environment. The reference files are generated for time zone
|
|
|
|
PST8PDT (Berkeley, California) and there will be apparent
|
|
|
|
failures if the tests are not run with that time zone setting.
|
|
|
|
The regression test driver sets environment variable
|
|
|
|
<envar>PGTZ</envar> to <literal>PST8PDT</literal> to ensure
|
|
|
|
proper results. However, your system must provide library
|
|
|
|
support for the PST8PDT time zone, or the time zone-dependent
|
|
|
|
tests will fail. To verify that your machine does have this
|
|
|
|
support, type the following:
|
|
|
|
<screen>
|
|
|
|
<prompt>$ </prompt><userinput>env TZ=PST8PDT date</userinput>
|
|
|
|
</screen>
|
|
|
|
The command above should have returned the current system time in
|
|
|
|
the PST8PDT time zone. If the PST8PDT database is not available,
|
|
|
|
then your system may have returned the time in GMT. If the
|
|
|
|
PST8PDT time zone is not available, you can set the time zone
|
|
|
|
rules explicitly:
|
|
|
|
<programlisting>
|
|
|
|
PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
1999-05-20 04:46:40 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
There appear to be some systems which do not accept the
|
|
|
|
recommended syntax for explicitly setting the local time zone
|
|
|
|
rules; you may need to use a different <envar>PGTZ</envar>
|
|
|
|
setting on such machines.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
2000-10-17 17:26:40 +02:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
<para>
|
2000-10-17 17:26:40 +02:00
|
|
|
Some systems using older time zone libraries fail to apply
|
|
|
|
daylight-savings corrections to dates before 1970, causing
|
|
|
|
pre-1970 PDT times to be displayed in PST instead. This will
|
|
|
|
result in localized differences in the test results.
|
2000-05-02 22:02:03 +02:00
|
|
|
</para>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Some of the queries in the <quote>timestamp</quote> test will
|
|
|
|
fail if you run the test on the day of a daylight-savings time
|
|
|
|
changeover, or the day before or after one. These queries assume
|
|
|
|
that the intervals between midnight yesterday, midnight today and
|
|
|
|
midnight tomorrow are exactly twenty-four hours -- which is wrong
|
|
|
|
if daylight-savings time went into or out of effect meanwhile.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
|
|
|
<title>Floating point differences</title>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Some of the tests involve computing 64-bit (<type>double
|
|
|
|
precision</type>) numbers from table columns. Differences in
|
|
|
|
results involving mathematical functions of <type>double
|
|
|
|
precision</type> columns have been observed. The float8 and
|
|
|
|
geometry tests are particularly prone to small differences across
|
|
|
|
platforms, or even with different compiler optimization options.
|
|
|
|
Human eyeball comparison is needed to determine the real
|
|
|
|
significance of these differences which are usually 10 places to
|
|
|
|
the right of the decimal point.
|
|
|
|
</para>
|
1999-05-20 04:46:40 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Some systems signal errors from <function>pow()</function> and
|
|
|
|
<function>exp()</function> differently from the mechanism
|
|
|
|
expected by the current <productname>PostgreSQL</productname>
|
|
|
|
code.
|
|
|
|
</para>
|
|
|
|
</sect2>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
|
|
|
<title>Polygon differences</title>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
Several of the tests involve operations on geographic data about
|
|
|
|
the Oakland/Berkeley, CA street map. The map data is expressed as
|
|
|
|
polygons whose vertices are represented as pairs of <type>double
|
|
|
|
precision</type> numbers (decimal latitude and
|
|
|
|
longitude). Initially, some tables are created and loaded with
|
|
|
|
geographic data, then some views are created which join two
|
|
|
|
tables using the polygon intersection operator
|
|
|
|
(<literal>##</literal>), then a select is done on the view.
|
|
|
|
</para>
|
1999-05-20 04:46:40 +02:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
When comparing the results from different platforms, differences
|
|
|
|
occur in the 2nd or 3rd place to the right of the decimal
|
|
|
|
point. The SQL statements where these problems occur are the
|
|
|
|
following:
|
|
|
|
<programlisting>
|
|
|
|
SELECT * from street;
|
|
|
|
SELECT * from iexit;
|
|
|
|
</programlisting>
|
|
|
|
</para>
|
|
|
|
</sect2>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<sect2>
|
|
|
|
<title>The <quote>random</quote> test</title>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<para>
|
|
|
|
There is at least one case in the <quote>random</quote> test
|
|
|
|
script that is intended to produce random results. This causes
|
|
|
|
random to fail the regression test once in a while (perhaps once
|
|
|
|
in every five to ten trials). Typing
|
|
|
|
<programlisting>
|
|
|
|
diff results/random.out expected/random.out
|
|
|
|
</programlisting>
|
|
|
|
should produce only one or a few lines of differences. You need
|
|
|
|
not worry unless the random test always fails in repeated
|
|
|
|
attempts. (On the other hand, if the random test is
|
|
|
|
<emphasis>never</emphasis> reported to fail even in many trials
|
|
|
|
of the regress tests, you probably <emphasis>should</emphasis>
|
|
|
|
worry.)
|
|
|
|
</para>
|
|
|
|
</sect2>
|
2000-05-02 22:02:03 +02:00
|
|
|
</sect1>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-10-17 17:26:40 +02:00
|
|
|
<!-- We might want to move the following section into the developer's guide. -->
|
2000-09-29 22:21:34 +02:00
|
|
|
<sect1 id="regress-platform">
|
2000-11-22 00:40:28 +01:00
|
|
|
<title>Platform-specific comparison files</title>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
|
|
|
Since some of the tests inherently produce platform-specific
|
|
|
|
results, we have provided a way to supply platform-specific result
|
|
|
|
comparison files. Frequently, the same variation applies to
|
|
|
|
multiple platforms; rather than supplying a separate comparison
|
|
|
|
file for every platform, there is a mapping file that defines
|
|
|
|
which comparison file to use. So, to eliminate bogus test
|
|
|
|
<quote>failures</quote> for a particular platform, you must choose
|
|
|
|
or make a variant result file, and then add a line to the mapping
|
|
|
|
file, which is <filename>resultmap</filename>.
|
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
|
|
|
Each line in the mapping file is of the form
|
|
|
|
<synopsis>
|
|
|
|
testname/platformnamepattern=comparisonfilename
|
|
|
|
</synopsis>
|
|
|
|
The test name is just the name of the particular regression test
|
|
|
|
module. The platform name pattern is a pattern in the style of
|
|
|
|
expr(1) (that is, a regular expression with an implicit ^ anchor
|
|
|
|
at the start). It is matched against the platform name as printed
|
|
|
|
by <filename>config.guess</filename> with an appended
|
|
|
|
<literal>:gcc</literal> or <literal>:cc</literal>, depending on
|
|
|
|
whether you use the GNU compiler or the system's native compiler
|
|
|
|
(on systems where there is a difference). The comparison file
|
|
|
|
name is the name of the substitute result comparison file.
|
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-11-22 00:40:28 +01:00
|
|
|
<para>
|
|
|
|
For example: the int2 regression test includes a deliberate entry
|
|
|
|
of a value that is too large to fit in int2. The specific error
|
|
|
|
message that is produced is platform-dependent; our reference
|
|
|
|
platform emits
|
|
|
|
<screen>
|
|
|
|
<computeroutput>ERROR: pg_atoi: error reading "100000": Numerical result out of range</computeroutput>
|
|
|
|
</screen>
|
|
|
|
but a fair number of other Unix platforms emit
|
|
|
|
<screen>
|
|
|
|
<computeroutput>ERROR: pg_atoi: error reading "100000": Result too large</computeroutput>
|
|
|
|
</screen>
|
|
|
|
Therefore, we provide a variant comparison file,
|
|
|
|
<filename>int2-too-large.out</filename>, that includes this
|
|
|
|
spelling of the error message. To silence the bogus
|
|
|
|
<quote>failure</quote> message on HPPA platforms, resultmap
|
|
|
|
includes
|
|
|
|
<programlisting>
|
|
|
|
int2/hppa=int2-too-large
|
|
|
|
</programlisting>
|
|
|
|
which will trigger on any machine for which config.guess's output
|
|
|
|
begins with <quote><literal>hppa</literal></quote>. Other lines
|
|
|
|
in resultmap select the variant comparison file for other
|
|
|
|
platforms where it's appropriate.
|
|
|
|
</para>
|
2000-01-09 21:54:36 +01:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
</sect1>
|
1998-12-29 03:24:47 +01:00
|
|
|
|
2000-05-02 22:02:03 +02:00
|
|
|
</chapter>
|
|
|
|
|
|
|
|
<!-- Keep this comment at the end of the file
|
|
|
|
Local variables:
|
|
|
|
mode:sgml
|
|
|
|
sgml-omittag:nil
|
|
|
|
sgml-shorttag:t
|
|
|
|
sgml-minimize-attributes:nil
|
|
|
|
sgml-always-quote-attributes:t
|
|
|
|
sgml-indent-step:1
|
|
|
|
sgml-indent-data:t
|
|
|
|
sgml-parent-document:nil
|
|
|
|
sgml-default-dtd-file:"./reference.ced"
|
|
|
|
sgml-exposed-tags:nil
|
|
|
|
sgml-local-catalogs:("/usr/lib/sgml/catalog")
|
|
|
|
sgml-local-ecat-files:nil
|
|
|
|
End:
|
|
|
|
-->
|