279 lines
14 KiB
Plaintext
279 lines
14 KiB
Plaintext
|
|
Regression Tests
|
|
|
|
The regression tests are a comprehensive set of tests for the SQL
|
|
implementation in PostgreSQL. They test standard SQL operations as
|
|
well as the extended capabilities of PostgreSQL.
|
|
_________________________________________________________________
|
|
|
|
Running the Tests
|
|
|
|
The regression tests can be run against an already installed and
|
|
running server, or using a temporary installation within the build
|
|
tree. Furthermore, there is a "parallel" and a "sequential" mode for
|
|
running the tests. The sequential method runs each test script in
|
|
turn, whereas the parallel method starts up multiple server processes
|
|
to run groups of tests in parallel. Parallel testing gives confidence
|
|
that interprocess communication and locking are working correctly. For
|
|
historical reasons, the sequential test is usually run against an
|
|
existing installation and the parallel method against a temporary
|
|
installation, but there are no technical reasons for this.
|
|
|
|
To run the regression tests after building but before installation,
|
|
type
|
|
gmake check
|
|
|
|
in the top-level directory. (Or you can change to "src/test/regress"
|
|
and run the command there.) This will first build several auxiliary
|
|
files, such as some sample user-defined trigger functions, and then
|
|
run the test driver script. At the end you should see something like
|
|
======================
|
|
All 98 tests passed.
|
|
======================
|
|
|
|
or otherwise a note about which tests failed. See the section called
|
|
Test Evaluation below before assuming that a "failure" represents a
|
|
serious problem.
|
|
|
|
Because this test method runs a temporary server, it will not work
|
|
when you are the root user (since the server will not start as root).
|
|
If you already did the build as root, you do not have to start all
|
|
over. Instead, make the regression test directory writable by some
|
|
other user, log in as that user, and restart the tests. For example
|
|
root# chmod -R a+w src/test/regress
|
|
root# chmod -R a+w contrib/spi
|
|
root# su - joeuser
|
|
joeuser$ cd top-level build directory
|
|
joeuser$ gmake check
|
|
|
|
(The only possible "security risk" here is that other users might be
|
|
able to alter the regression test results behind your back. Use common
|
|
sense when managing user permissions.)
|
|
|
|
Alternatively, run the tests after installation.
|
|
|
|
If you have configured PostgreSQL to install into a location where an
|
|
older PostgreSQL installation already exists, and you perform gmake
|
|
check before installing the new version, you may find that the tests
|
|
fail because the new programs try to use the already-installed shared
|
|
libraries. (Typical symptoms are complaints about undefined symbols.)
|
|
If you wish to run the tests before overwriting the old installation,
|
|
you'll need to build with configure --disable-rpath. It is not
|
|
recommended that you use this option for the final installation,
|
|
however.
|
|
|
|
The parallel regression test starts quite a few processes under your
|
|
user ID. Presently, the maximum concurrency is twenty parallel test
|
|
scripts, which means sixty processes: there's a server process, a
|
|
psql, and usually a shell parent process for the psql for each test
|
|
script. So if your system enforces a per-user limit on the number of
|
|
processes, make sure this limit is at least seventy-five or so, else
|
|
you may get random-seeming failures in the parallel test. If you are
|
|
not in a position to raise the limit, you can cut down the degree of
|
|
parallelism by setting the MAX_CONNECTIONS parameter. For example,
|
|
gmake MAX_CONNECTIONS=10 check
|
|
|
|
runs no more than ten tests concurrently.
|
|
|
|
On some systems, the default Bourne-compatible shell ("/bin/sh") gets
|
|
confused when it has to manage too many child processes in parallel.
|
|
This may cause the parallel test run to lock up or fail. In such
|
|
cases, specify a different Bourne-compatible shell on the command
|
|
line, for example:
|
|
gmake SHELL=/bin/ksh check
|
|
|
|
If no non-broken shell is available, you may be able to work around
|
|
the problem by limiting the number of connections, as shown above.
|
|
|
|
To run the tests after installation, initialize a data area and start
|
|
the server, then type
|
|
gmake installcheck
|
|
|
|
or for a parallel test
|
|
gmake installcheck-parallel
|
|
|
|
The tests will expect to contact the server at the local host and the
|
|
default port number, unless directed otherwise by PGHOST and PGPORT
|
|
environment variables.
|
|
_________________________________________________________________
|
|
|
|
Test Evaluation
|
|
|
|
Some properly installed and fully functional PostgreSQL installations
|
|
can "fail" some of these regression tests due to platform-specific
|
|
artifacts such as varying floating-point representation and message
|
|
wording. The tests are currently evaluated using a simple "diff"
|
|
comparison against the outputs generated on a reference system, so the
|
|
results are sensitive to small system differences. When a test is
|
|
reported as "failed", always examine the differences between expected
|
|
and actual results; you may well find that the differences are not
|
|
significant. Nonetheless, we still strive to maintain accurate
|
|
reference files across all supported platforms, so it can be expected
|
|
that all tests pass.
|
|
|
|
The actual outputs of the regression tests are in files in the
|
|
"src/test/regress/results" directory. The test script uses "diff" to
|
|
compare each output file against the reference outputs stored in the
|
|
"src/test/regress/expected" directory. Any differences are saved for
|
|
your inspection in "src/test/regress/regression.diffs". (Or you can
|
|
run "diff" yourself, if you prefer.)
|
|
|
|
If for some reason a particular platform generates a "failure" for a
|
|
given test, but inspection of the output convinces you that the result
|
|
is valid, you can add a new comparison file to silence the failure
|
|
report in future test runs. See the section called Variant Comparison
|
|
Files for details.
|
|
_________________________________________________________________
|
|
|
|
Error message differences
|
|
|
|
Some of the regression tests involve intentional invalid input values.
|
|
Error messages can come from either the PostgreSQL code or from the
|
|
host platform system routines. In the latter case, the messages may
|
|
vary between platforms, but should reflect similar information. These
|
|
differences in messages will result in a "failed" regression test that
|
|
can be validated by inspection.
|
|
_________________________________________________________________
|
|
|
|
Locale differences
|
|
|
|
If you run the tests against an already-installed server that was
|
|
initialized with a collation-order locale other than C, then there may
|
|
be differences due to sort order and follow-up failures. The
|
|
regression test suite is set up to handle this problem by providing
|
|
alternative result files that together are known to handle a large
|
|
number of locales.
|
|
_________________________________________________________________
|
|
|
|
Date and time differences
|
|
|
|
Most of the date and time results are dependent on the time zone
|
|
environment. The reference files are generated for time zone PST8PDT
|
|
(Berkeley, California), and there will be apparent failures if the
|
|
tests are not run with that time zone setting. The regression test
|
|
driver sets environment variable PGTZ to PST8PDT, which normally
|
|
ensures proper results.
|
|
_________________________________________________________________
|
|
|
|
Floating-point differences
|
|
|
|
Some of the tests involve computing 64-bit floating-point numbers
|
|
(double precision) from table columns. Differences in results
|
|
involving mathematical functions of double precision columns have been
|
|
observed. The float8 and geometry tests are particularly prone to
|
|
small differences across platforms, or even with different compiler
|
|
optimization options. Human eyeball comparison is needed to determine
|
|
the real significance of these differences which are usually 10 places
|
|
to the right of the decimal point.
|
|
|
|
Some systems display minus zero as -0, while others just show 0.
|
|
|
|
Some systems signal errors from pow() and exp() differently from the
|
|
mechanism expected by the current PostgreSQL code.
|
|
_________________________________________________________________
|
|
|
|
Row ordering differences
|
|
|
|
You might see differences in which the same rows are output in a
|
|
different order than what appears in the expected file. In most cases
|
|
this is not, strictly speaking, a bug. Most of the regression test
|
|
scripts are not so pedantic as to use an ORDER BY for every single
|
|
SELECT, and so their result row orderings are not well-defined
|
|
according to the letter of the SQL specification. In practice, since
|
|
we are looking at the same queries being executed on the same data by
|
|
the same software, we usually get the same result ordering on all
|
|
platforms, and so the lack of ORDER BY isn't a problem. Some queries
|
|
do exhibit cross-platform ordering differences, however. When testing
|
|
against an already-installed server, ordering differences can also be
|
|
caused by non-C locale settings or non-default parameter settings,
|
|
such as custom values of work_mem or the planner cost parameters.
|
|
|
|
Therefore, if you see an ordering difference, it's not something to
|
|
worry about, unless the query does have an ORDER BY that your result
|
|
is violating. But please report it anyway, so that we can add an ORDER
|
|
BY to that particular query and thereby eliminate the bogus "failure"
|
|
in future releases.
|
|
|
|
You might wonder why we don't order all the regression test queries
|
|
explicitly to get rid of this issue once and for all. The reason is
|
|
that that would make the regression tests less useful, not more, since
|
|
they'd tend to exercise query plan types that produce ordered results
|
|
to the exclusion of those that don't.
|
|
_________________________________________________________________
|
|
|
|
The "random" test
|
|
|
|
The random test script is intended to produce random results. In rare
|
|
cases, this causes the random regression test to fail. Typing
|
|
diff results/random.out expected/random.out
|
|
|
|
should produce only one or a few lines of differences. You need not
|
|
worry unless the random test fails repeatedly.
|
|
_________________________________________________________________
|
|
|
|
Variant Comparison Files
|
|
|
|
Since some of the tests inherently produce environment-dependent
|
|
results, we have provided ways to specify alternative "expected"
|
|
result files. Each regression test can have several comparison files
|
|
showing possible results on different platforms. There are two
|
|
independent mechanisms for determining which comparison file is used
|
|
for each test.
|
|
|
|
The first mechanism allows comparison files to be selected for
|
|
specific platforms. There is a mapping file,
|
|
"src/test/regress/resultmap", that defines which comparison file to
|
|
use for each platform. To eliminate bogus test "failures" for a
|
|
particular platform, you first choose or make a variant result file,
|
|
and then add a line to the "resultmap" file.
|
|
|
|
Each line in the mapping file is of the form
|
|
testname/platformpattern=comparisonfilename
|
|
|
|
The test name is just the name of the particular regression test
|
|
module. The platform pattern is a pattern in the style of the Unix
|
|
tool "expr" (that is, a regular expression with an implicit ^ anchor
|
|
at the start). It is matched against the platform name as printed by
|
|
"config.guess" followed by :gcc or :cc, depending on whether you use
|
|
the GNU compiler or the system's native compiler (on systems where
|
|
there is a difference). The comparison file name is the base name of
|
|
the substitute result comparison file.
|
|
|
|
For example: some systems interpret very small floating-point values
|
|
as zero, rather than reporting an underflow error. This causes a few
|
|
differences in the "float8" regression test. Therefore, we provide a
|
|
variant comparison file, "float8-small-is-zero.out", which includes
|
|
the results to be expected on these systems. To silence the bogus
|
|
"failure" message on OpenBSD platforms, "resultmap" includes
|
|
float8/i.86-.*-openbsd=float8-small-is-zero
|
|
|
|
which will trigger on any machine for which the output of
|
|
"config.guess" matches i.86-.*-openbsd. Other lines in "resultmap"
|
|
select the variant comparison file for other platforms where it's
|
|
appropriate.
|
|
|
|
The second selection mechanism for variant comparison files is much
|
|
more automatic: it simply uses the "best match" among several supplied
|
|
comparison files. The regression test driver script considers both the
|
|
standard comparison file for a test, testname.out, and variant files
|
|
named testname_digit.out (where the "digit" is any single digit 0-9).
|
|
If any such file is an exact match, the test is considered to pass;
|
|
otherwise, the one that generates the shortest diff is used to create
|
|
the failure report. (If "resultmap" includes an entry for the
|
|
particular test, then the base "testname" is the substitute name given
|
|
in "resultmap".)
|
|
|
|
For example, for the char test, the comparison file "char.out"
|
|
contains results that are expected in the C and POSIX locales, while
|
|
the file "char_1.out" contains results sorted as they appear in many
|
|
other locales.
|
|
|
|
The best-match mechanism was devised to cope with locale-dependent
|
|
results, but it can be used in any situation where the test results
|
|
cannot be predicted easily from the platform name alone. A limitation
|
|
of this mechanism is that the test driver cannot tell which variant is
|
|
actually "correct" for the current environment; it will just pick the
|
|
variant that seems to work best. Therefore it is safest to use this
|
|
mechanism only for variant results that you are willing to consider
|
|
equally valid in all contexts.
|