456 lines
18 KiB
Plaintext
456 lines
18 KiB
Plaintext
<!-- $PostgreSQL: pgsql/doc/src/sgml/regress.sgml,v 1.60 2008/09/05 12:11:18 petere Exp $ -->
|
|
|
|
<chapter id="regress">
|
|
<title id="regress-title">Regression Tests</title>
|
|
|
|
<indexterm zone="regress">
|
|
<primary>regression tests</primary>
|
|
</indexterm>
|
|
|
|
<indexterm zone="regress">
|
|
<primary>test</primary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
The regression tests are a comprehensive set of tests for the SQL
|
|
implementation in <productname>PostgreSQL</productname>. They test
|
|
standard SQL operations as well as the extended capabilities of
|
|
<productname>PostgreSQL</productname>.
|
|
</para>
|
|
|
|
<sect1 id="regress-run">
|
|
<title>Running the Tests</title>
|
|
|
|
<para>
|
|
The regression tests can be run against an already installed and
|
|
running server, or using a temporary installation within the build
|
|
tree. Furthermore, there is a <quote>parallel</quote> and a
|
|
<quote>sequential</quote> mode for running the tests. The
|
|
sequential method runs each test script in turn, whereas the
|
|
parallel method starts up multiple server processes to run groups
|
|
of tests in parallel. Parallel testing gives confidence that
|
|
interprocess communication and locking are working correctly. For
|
|
historical reasons, the sequential test is usually run against an
|
|
existing installation and the parallel method against a temporary
|
|
installation, but there are no technical reasons for this.
|
|
</para>
|
|
|
|
<para>
|
|
To run the regression tests after building but before installation,
|
|
type
|
|
<screen>
|
|
gmake check
|
|
</screen>
|
|
in the top-level directory. (Or you can change to
|
|
<filename>src/test/regress</filename> and run the command there.)
|
|
This will first build several auxiliary files, such as
|
|
some sample user-defined trigger functions, and then run the test driver
|
|
script. At the end you should see something like
|
|
<screen>
|
|
<computeroutput>
|
|
=======================
|
|
All 115 tests passed.
|
|
=======================
|
|
</computeroutput>
|
|
</screen>
|
|
or otherwise a note about which tests failed. See <xref
|
|
linkend="regress-evaluation"> below before assuming that a
|
|
<quote>failure</> represents a serious problem.
|
|
</para>
|
|
|
|
<para>
|
|
Because this test method runs a temporary server, it will not work
|
|
when you are the root user (since the server will not start as root).
|
|
If you already did the build as root, you do not have to start all
|
|
over. Instead, make the regression test directory writable by
|
|
some other user, log in as that user, and restart the tests.
|
|
For example
|
|
<screen>
|
|
<prompt>root# </prompt><userinput>chmod -R a+w src/test/regress</userinput>
|
|
<prompt>root# </prompt><userinput>su - joeuser</userinput>
|
|
<prompt>joeuser$ </prompt><userinput>cd <replaceable>top-level build directory</></userinput>
|
|
<prompt>joeuser$ </prompt><userinput>gmake check</userinput>
|
|
</screen>
|
|
(The only possible <quote>security risk</quote> here is that other
|
|
users might be able to alter the regression test results behind
|
|
your back. Use common sense when managing user permissions.)
|
|
</para>
|
|
<para>
|
|
Alternatively, run the tests after installation.
|
|
</para>
|
|
|
|
<para>
|
|
If you have configured <productname>PostgreSQL</productname> to install
|
|
into a location where an older <productname>PostgreSQL</productname>
|
|
installation already exists, and you perform <literal>gmake check</>
|
|
before installing the new version, you might find that the tests fail
|
|
because the new programs try to use the already-installed shared
|
|
libraries. (Typical symptoms are complaints about undefined symbols.)
|
|
If you wish to run the tests before overwriting the old installation,
|
|
you'll need to build with <literal>configure --disable-rpath</>.
|
|
It is not recommended that you use this option for the final installation,
|
|
however.
|
|
</para>
|
|
|
|
<para>
|
|
The parallel regression test starts quite a few processes under your
|
|
user ID. Presently, the maximum concurrency is twenty parallel test
|
|
scripts, which means forty processes: there's a server process and a
|
|
<application>psql</> process for each test script.
|
|
So if your system enforces a per-user limit on the number of processes,
|
|
make sure this limit is at least fifty or so, else you might get
|
|
random-seeming failures in the parallel test. If you are not in
|
|
a position to raise the limit, you can cut down the degree of parallelism
|
|
by setting the <literal>MAX_CONNECTIONS</> parameter. For example,
|
|
<screen>
|
|
gmake MAX_CONNECTIONS=10 check
|
|
</screen>
|
|
runs no more than ten tests concurrently.
|
|
</para>
|
|
|
|
<para>
|
|
To run the tests after installation<![%standalone-ignore;[ (see <xref linkend="installation">)]]>,
|
|
initialize a data area and start the
|
|
server, <![%standalone-ignore;[as explained in <xref linkend="runtime">, ]]> then type
|
|
<screen>
|
|
gmake installcheck
|
|
</screen>
|
|
or for a parallel test
|
|
<screen>
|
|
gmake installcheck-parallel
|
|
</screen>
|
|
The tests will expect to contact the server at the local host and the
|
|
default port number, unless directed otherwise by <envar>PGHOST</envar> and
|
|
<envar>PGPORT</envar> environment variables.
|
|
</para>
|
|
|
|
<para>
|
|
The source distribution also contains regression tests for the optional
|
|
procedural languages and for some of the <filename>contrib</> modules.
|
|
At present, these tests can be used only against an already-installed
|
|
server. To run the tests for all procedural languages that have been
|
|
built and installed, change to the <filename>src/pl</> directory of the
|
|
build tree and type
|
|
<screen>
|
|
gmake installcheck
|
|
</screen>
|
|
You can also do this in any of the subdirectories of <filename>src/pl</>
|
|
to run tests for just one procedural language. To run the tests for all
|
|
<filename>contrib</> modules that have them, change to the
|
|
<filename>contrib</> directory of the build tree and type
|
|
<screen>
|
|
gmake installcheck
|
|
</screen>
|
|
The <filename>contrib</> modules must have been built and installed first.
|
|
You can also do this in a subdirectory of <filename>contrib</> to run
|
|
the tests for just one module.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="regress-evaluation">
|
|
<title>Test Evaluation</title>
|
|
|
|
<para>
|
|
Some properly installed and fully functional
|
|
<productname>PostgreSQL</productname> installations can
|
|
<quote>fail</quote> some of these regression tests due to
|
|
platform-specific artifacts such as varying floating-point representation
|
|
and message wording. The tests are currently evaluated using a simple
|
|
<command>diff</command> comparison against the outputs
|
|
generated on a reference system, so the results are sensitive to
|
|
small system differences. When a test is reported as
|
|
<quote>failed</quote>, always examine the differences between
|
|
expected and actual results; you might find that the
|
|
differences are not significant. Nonetheless, we still strive to
|
|
maintain accurate reference files across all supported platforms,
|
|
so it can be expected that all tests pass.
|
|
</para>
|
|
|
|
<para>
|
|
The actual outputs of the regression tests are in files in the
|
|
<filename>src/test/regress/results</filename> directory. The test
|
|
script uses <command>diff</command> to compare each output
|
|
file against the reference outputs stored in the
|
|
<filename>src/test/regress/expected</filename> directory. Any
|
|
differences are saved for your inspection in
|
|
<filename>src/test/regress/regression.diffs</filename>. (Or you
|
|
can run <command>diff</command> yourself, if you prefer.)
|
|
</para>
|
|
|
|
<para>
|
|
If for some reason a particular platform generates a <quote>failure</>
|
|
for a given test, but inspection of the output convinces you that
|
|
the result is valid, you can add a new comparison file to silence
|
|
the failure report in future test runs. See
|
|
<xref linkend="regress-variant"> for details.
|
|
</para>
|
|
|
|
<sect2>
|
|
<title>Error message differences</title>
|
|
|
|
<para>
|
|
Some of the regression tests involve intentional invalid input
|
|
values. Error messages can come from either the
|
|
<productname>PostgreSQL</productname> code or from the host
|
|
platform system routines. In the latter case, the messages can
|
|
vary between platforms, but should reflect similar
|
|
information. These differences in messages will result in a
|
|
<quote>failed</quote> regression test that can be validated by
|
|
inspection.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Locale differences</title>
|
|
|
|
<para>
|
|
If you run the tests against an already-installed server that was
|
|
initialized with a collation-order locale other than C, then
|
|
there might be differences due to sort order and follow-up
|
|
failures. The regression test suite is set up to handle this
|
|
problem by providing alternative result files that together are
|
|
known to handle a large number of locales.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Date and time differences</title>
|
|
|
|
<para>
|
|
Most of the date and time results are dependent on the time zone
|
|
environment. The reference files are generated for time zone
|
|
<literal>PST8PDT</literal> (Berkeley, California), and there will be
|
|
apparent failures if the tests are not run with that time zone setting.
|
|
The regression test driver sets environment variable
|
|
<envar>PGTZ</envar> to <literal>PST8PDT</literal>, which normally
|
|
ensures proper results.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Floating-point differences</title>
|
|
|
|
<para>
|
|
Some of the tests involve computing 64-bit floating-point numbers (<type>double
|
|
precision</type>) from table columns. Differences in
|
|
results involving mathematical functions of <type>double
|
|
precision</type> columns have been observed. The <literal>float8</> and
|
|
<literal>geometry</> tests are particularly prone to small differences
|
|
across platforms, or even with different compiler optimization options.
|
|
Human eyeball comparison is needed to determine the real
|
|
significance of these differences which are usually 10 places to
|
|
the right of the decimal point.
|
|
</para>
|
|
|
|
<para>
|
|
Some systems display minus zero as <literal>-0</>, while others
|
|
just show <literal>0</>.
|
|
</para>
|
|
|
|
<para>
|
|
Some systems signal errors from <function>pow()</function> and
|
|
<function>exp()</function> differently from the mechanism
|
|
expected by the current <productname>PostgreSQL</productname>
|
|
code.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Row ordering differences</title>
|
|
|
|
<para>
|
|
You might see differences in which the same rows are output in a
|
|
different order than what appears in the expected file. In most cases
|
|
this is not, strictly speaking, a bug. Most of the regression test
|
|
scripts are not so pedantic as to use an <literal>ORDER BY</> for every single
|
|
<literal>SELECT</>, and so their result row orderings are not well-defined
|
|
according to the letter of the SQL specification. In practice, since we are
|
|
looking at the same queries being executed on the same data by the same
|
|
software, we usually get the same result ordering on all platforms, and
|
|
so the lack of <literal>ORDER BY</> isn't a problem. Some queries do exhibit
|
|
cross-platform ordering differences, however. When testing against an
|
|
already-installed server, ordering differences can also be caused by
|
|
non-C locale settings or non-default parameter settings, such as custom values
|
|
of <varname>work_mem</> or the planner cost parameters.
|
|
</para>
|
|
|
|
<para>
|
|
Therefore, if you see an ordering difference, it's not something to
|
|
worry about, unless the query does have an <literal>ORDER BY</> that your
|
|
result is violating. But please report it anyway, so that we can add an
|
|
<literal>ORDER BY</> to that particular query and thereby eliminate the bogus
|
|
<quote>failure</quote> in future releases.
|
|
</para>
|
|
|
|
<para>
|
|
You might wonder why we don't order all the regression test queries explicitly
|
|
to get rid of this issue once and for all. The reason is that that would
|
|
make the regression tests less useful, not more, since they'd tend
|
|
to exercise query plan types that produce ordered results to the
|
|
exclusion of those that don't.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Insufficient stack depth</title>
|
|
|
|
<para>
|
|
If the <literal>errors</literal> test results in a server crash
|
|
at the <literal>select infinite_recurse()</> command, it means that
|
|
the platform's limit on process stack size is smaller than the
|
|
<![%standalone-ignore;[<xref linkend="guc-max-stack-depth">]]>
|
|
<![%standalone-include;[<literal>max_stack_depth</literal>]]>
|
|
parameter indicates. This
|
|
can be fixed by running the server under a higher stack
|
|
size limit (4MB is recommended with the default value of
|
|
<varname>max_stack_depth</>). If you are unable to do that, an
|
|
alternative is to reduce the value of <varname>max_stack_depth</>.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>The <quote>random</quote> test</title>
|
|
|
|
<para>
|
|
The <literal>random</literal> test script is intended to produce
|
|
random results. In rare cases, this causes the random regression
|
|
test to fail. Typing:
|
|
<programlisting>
|
|
diff results/random.out expected/random.out
|
|
</programlisting>
|
|
should produce only one or a few lines of differences. You need
|
|
not worry unless the random test fails repeatedly.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- We might want to move the following section into the developer's guide. -->
|
|
<sect1 id="regress-variant">
|
|
<title>Variant Comparison Files</title>
|
|
|
|
<para>
|
|
Since some of the tests inherently produce environment-dependent
|
|
results, we have provided ways to specify alternative <quote>expected</>
|
|
result files. Each regression test can have several comparison files
|
|
showing possible results on different platforms. There are two
|
|
independent mechanisms for determining which comparison file is used
|
|
for each test.
|
|
</para>
|
|
|
|
<para>
|
|
The first mechanism allows comparison files to be selected for
|
|
specific platforms. There is a mapping file,
|
|
<filename>src/test/regress/resultmap</filename>, that defines
|
|
which comparison file to use for each platform.
|
|
To eliminate bogus test <quote>failures</quote> for a particular platform,
|
|
you first choose or make a variant result file, and then add a line to the
|
|
<filename>resultmap</filename> file.
|
|
</para>
|
|
|
|
<para>
|
|
Each line in the mapping file is of the form
|
|
<synopsis>
|
|
testname:output:platformpattern=comparisonfilename
|
|
</synopsis>
|
|
The test name is just the name of the particular regression test
|
|
module. The output value indicates which output file to check. For the
|
|
standard regression tests, this is always <literal>out</literal>. The
|
|
value corresponds to the file extension of the output file.
|
|
The platform pattern is a pattern in the style of the Unix
|
|
tool <command>expr</> (that is, a regular expression with an implicit
|
|
<literal>^</literal> anchor at the start). It is matched against the
|
|
platform name as printed by <command>config.guess</command>.
|
|
The comparison file name is the base name of the substitute result
|
|
comparison file.
|
|
</para>
|
|
|
|
<para>
|
|
For example: some systems interpret very small floating-point values
|
|
as zero, rather than reporting an underflow error. This causes a
|
|
few differences in the <filename>float8</> regression test.
|
|
Therefore, we provide a variant comparison file,
|
|
<filename>float8-small-is-zero.out</filename>, which includes
|
|
the results to be expected on these systems. To silence the bogus
|
|
<quote>failure</quote> message on <systemitem>OpenBSD</systemitem>
|
|
platforms, <filename>resultmap</filename> includes:
|
|
<programlisting>
|
|
float8:out:i.86-.*-openbsd=float8-small-is-zero.out
|
|
</programlisting>
|
|
which will trigger on any machine for which the output of
|
|
<command>config.guess</command> matches <literal>i.86-.*-openbsd</literal>.
|
|
Other lines
|
|
in <filename>resultmap</> select the variant comparison file for other
|
|
platforms where it's appropriate.
|
|
</para>
|
|
|
|
<para>
|
|
The second selection mechanism for variant comparison files is
|
|
much more automatic: it simply uses the <quote>best match</> among
|
|
several supplied comparison files. The regression test driver
|
|
script considers both the standard comparison file for a test,
|
|
<literal><replaceable>testname</>.out</>, and variant files named
|
|
<literal><replaceable>testname</>_<replaceable>digit</>.out</>
|
|
(where the <replaceable>digit</> is any single digit
|
|
<literal>0</>-<literal>9</>). If any such file is an exact match,
|
|
the test is considered to pass; otherwise, the one that generates
|
|
the shortest diff is used to create the failure report. (If
|
|
<filename>resultmap</filename> includes an entry for the particular
|
|
test, then the base <replaceable>testname</> is the substitute
|
|
name given in <filename>resultmap</filename>.)
|
|
</para>
|
|
|
|
<para>
|
|
For example, for the <literal>char</literal> test, the comparison file
|
|
<filename>char.out</filename> contains results that are expected
|
|
in the <literal>C</> and <literal>POSIX</> locales, while
|
|
the file <filename>char_1.out</filename> contains results sorted as
|
|
they appear in many other locales.
|
|
</para>
|
|
|
|
<para>
|
|
The best-match mechanism was devised to cope with locale-dependent
|
|
results, but it can be used in any situation where the test results
|
|
cannot be predicted easily from the platform name alone. A limitation of
|
|
this mechanism is that the test driver cannot tell which variant is
|
|
actually <quote>correct</> for the current environment; it will just pick
|
|
the variant that seems to work best. Therefore it is safest to use this
|
|
mechanism only for variant results that you are willing to consider
|
|
equally valid in all contexts.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="regress-coverage">
|
|
<title>Test Coverage Examination</title>
|
|
|
|
<para>
|
|
The PostgreSQL source code can be compiled with coverage testing
|
|
instrumentation, so that it becomes possible to examine which
|
|
parts of the code are covered by the regression tests or any other
|
|
test suite that is run with the code. This is currently supported
|
|
when compiling with GCC and requires the <command>gcov</command>
|
|
and <command>lcov</command> programs.
|
|
</para>
|
|
|
|
<para>
|
|
A typical workflow would look like this:
|
|
<screen>
|
|
./configure --enable-coverage ... OTHER OPTIONS ...
|
|
gmake
|
|
gmake check # or other test suite
|
|
gmake coverage-html
|
|
</screen>
|
|
Then point your HTML browser
|
|
to <filename>coverage/index.html</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
To reset the execution counts between test runs, run
|
|
<screen>
|
|
gmake coverage-clean
|
|
</screen>
|
|
</para>
|
|
</sect1>
|
|
|
|
</chapter>
|