mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-10-03 09:06:53 +02:00
066fce7bf9
Include section on error message differences.
227 lines
9.9 KiB
Plaintext
227 lines
9.9 KiB
Plaintext
|
|
Introduction
|
|
|
|
The PostgreSQL regression tests are a comprehensive set of tests for the
|
|
SQL implementation embedded in PostgreSQL developed by Jolly Chen and
|
|
Andrew Yu. It tests standard SQL operations as well as the extensibility
|
|
capabilities of PostgreSQL.
|
|
|
|
These tests have recently been revised by Marc Fournier and Thomas Lockhart
|
|
to become current for PostgreSQL v6.1. The tests are now packaged as
|
|
functional units and should be easier to run and easier to interpret.
|
|
|
|
Some properly installed and fully functional PostgreSQL installations
|
|
can fail some of these regression tests due to artifacts of floating point
|
|
representation and time zone support. The current tests are evaluated
|
|
using a simple "diff" algorithm, and are sensitive to small system
|
|
differences. For apparently failed tests, examining the differences
|
|
may reveal that the differences are not significant.
|
|
|
|
Preparation
|
|
|
|
The regression test is invoked by the 'make' command which compiles
|
|
a 'c' program with PostgreSQL extension functions into a shared library
|
|
in the current directory. Localized shell scripts are also created in
|
|
the current directory. The output file templates are massaged into the
|
|
./expected/*.out files. The localization replaces macros in the source
|
|
files with absolute pathnames and user names.
|
|
|
|
The postmaster should be invoked with the system time zone set for
|
|
Berkeley, California. On many systems, this can be accomplished by
|
|
setting the TZ environment variable before starting the postmaster
|
|
(for csh/bash; use set/export for some other shells):
|
|
|
|
setenv TZ PST8PDT
|
|
date
|
|
/usr/local/pgsql/bin/postmaster -s
|
|
|
|
The "date" command above should have returned the current system time
|
|
in the PST8PDT time zone. If the PST8PDT database is not available, then
|
|
your system may have returned the time in GMT. If the PST8PDT time zone
|
|
is not available, you can set the time zone rules explicitly:
|
|
|
|
setenv TZ PST8PDT7,M04.01.0,M10.05.03
|
|
|
|
Directory Layout
|
|
|
|
input/ .... .source files that are converted using 'make all' into
|
|
some of the .sql files in the 'sql' subdirectory
|
|
|
|
output/ ... .source files that are converted using 'make all' into
|
|
.out files in the 'expected' subdirectory
|
|
|
|
sql/ ...... .sql files used to perform the regression tests
|
|
|
|
expected/ . .out files that represent what we *expect* the results to
|
|
look like
|
|
|
|
results/ .. .out files that represent what the results *actually* look
|
|
like. Also used as temporary storage for table copy testing.
|
|
|
|
Running the regression test
|
|
|
|
If you have prevously invoked the regression test, clean up the
|
|
working directory with:
|
|
|
|
make clean
|
|
|
|
The regression test is invoked with the command:
|
|
|
|
make all runtest
|
|
|
|
Normally, the regression test should be run as the pg_superuser since
|
|
the 'src/test/regress' directory and sub-directories are owned by the
|
|
pg_superuser. If you run the regression test as another user the
|
|
'src/test/regress' directory tree should be writeable to that user.
|
|
|
|
Comparing expected/actual output
|
|
|
|
The results are in files in the ./results directory. These results
|
|
can be compared with results in the ./expected directory using 'diff'.
|
|
The files might not compare exactly. The following paragraphs attempt
|
|
to explain the differences.
|
|
|
|
Error message differences
|
|
|
|
Some of the regression tests involve intentional invalid input values.
|
|
Error messages can come from either the Postgres code or from the host
|
|
platform system routines. In the latter case, the messages may vary
|
|
between platforms, but should reflect similar information. These
|
|
differences in messages will result in a "failed" regression test which
|
|
can be validated by inspection.
|
|
|
|
OID differences
|
|
|
|
There are several places where PostgreSQL OID (object identifiers) appear
|
|
in 'regress.out'. OID's are unique 32-bit integers which are generated
|
|
by the PostgreSQL backend whenever a table row is inserted or updated.
|
|
If you run the regression test on a non-virgin database or run it multiple
|
|
times, the OID's reported will have different values.
|
|
|
|
The following SQL statements in 'misc.out' have shown this behavior:
|
|
|
|
QUERY: SELECT user_relns() AS user_relns ORDER BY user_relns;
|
|
|
|
The 'a,523676' row is composed from an OID.
|
|
|
|
DATE/TIME differences
|
|
|
|
On many supported platforms, you can force PostgreSQL to believe that it
|
|
is running in the same time zone as Berkeley, California. See details in
|
|
the section on how to run the regression tests.
|
|
|
|
If you do not explicitly set your time zone environment to PST8PDT, then
|
|
most of the date and time results will reflect your local time zone and
|
|
will fail the regression testing.
|
|
|
|
There appears to be some systems which do not accept the recommended syntax
|
|
for explicitly setting the local time zone rules. Some systems using the
|
|
public domain time zone package exhibit minor problems with pre-1970 PDT
|
|
times, representing them in PST instead.
|
|
|
|
FLOATING POINT differences
|
|
|
|
Some of the tests involve computing 64-bit (FLOAT8) number from table
|
|
columns. Differences in results involving mathematical functions of
|
|
FLOAT8 columns have been observed. These differences occur where
|
|
different operating systems are used on the same platform ie:
|
|
BSDI and SOLARIS on Intel/86, and where the same operating system is
|
|
used used on different platforms, ie: SOLARIS on SPARC and Intel/86.
|
|
|
|
Human eyeball comparison is needed to determine the real significance
|
|
of these differences which are usually 10 places to the right of
|
|
the decimal point.
|
|
|
|
Some systems signal errors from pow() and exp() differently from
|
|
the mechanism expected by the current Postgres code.
|
|
|
|
POLYGON differences
|
|
|
|
Several of the tests involve operations on geographic date about the
|
|
Oakland/Berkley CA street map. The map data is expressed as polygons
|
|
whose vertices are represented as pairs of FLOAT8 numbers (decimal
|
|
latitude and longitude). Initially, some tables are created and
|
|
loaded with geographic data, then some views are created which join
|
|
two tables using the polygon intersection operator (##), then a select
|
|
is done on the view.
|
|
|
|
When comparing the results from different platforms, differences occur
|
|
in the 2nd or 3rd place to the right of the decimal point. The SQL
|
|
statements where these problems occur are the folowing:
|
|
|
|
QUERY: SELECT * from street;
|
|
QUERY: SELECT * from iexit;
|
|
|
|
Random differences
|
|
|
|
There is at least one test case in random.out which is intended to produce
|
|
random results. This causes random to fail the regression testing.
|
|
Typing "diff results/random.out expected/random.out" should produce only
|
|
one or a few lines of differences for this reason, but other floating
|
|
point differences on dissimilar architectures might cause many more
|
|
differences. See the release notes below.
|
|
|
|
The 'expected' files
|
|
|
|
The ./expected/*.out files were adapted from the original monolithic
|
|
'expected.input' file provided by Jolly Chen et al. Newer versions of these
|
|
files generated on various development machines have been substituted after
|
|
careful (?) inspection. Many of the development machines are running a
|
|
Unix OS variant (FreeBSD, Linux, etc) on Ix86 hardware.
|
|
|
|
The original 'expected.input' file was created on a SPARC Solaris 2.4
|
|
system using the 'postgres5-1.02a5.tar.gz' source tree. It was compared
|
|
with a file created on an I386 Solaris 2.4 system and the differences
|
|
were only in the floating point polygons in the 3rd digit to the right
|
|
of the decimal point. (see below)
|
|
|
|
The original 'sample.regress.out' file was from the postgres-1.01 release
|
|
constructed by Jolly Chen and is included here for reference. It may
|
|
have been created on a DEC ALPHA machine as the 'Makefile.global'
|
|
in the postgres-1.01 release has PORTNAME=alpha.
|
|
|
|
Current release notes (Thomas.Lockhart@jpl.nasa.gov)
|
|
|
|
The regression tests have been adapted and extensively modified for the
|
|
v6.1 release of PostgreSQL.
|
|
|
|
Three new data types (datetime, timespan, and circle) have been added to
|
|
the native set of PostgreSQL types. Points, boxes, paths, and polygons
|
|
have had their output formats made consistant across the data types.
|
|
The polygon output in misc.out has only been spot-checked for correctness
|
|
relative to the original regression output.
|
|
|
|
PostgreSQL v6.1 introduces a new, alternate optimizer which uses "genetic"
|
|
algorithms. These algorithms introduce a random behavior in the ordering
|
|
of query results when the query contains multiple qualifiers or multiple
|
|
tables (giving the optimizer a choice on order of evaluation). Several
|
|
regression tests have been modified to explicitly order the results, and
|
|
hence are insensitive to optimizer choices. A few regression tests are
|
|
for data types which are inherently unordered (e.g. points and time
|
|
intervals) and tests involving those types are explicitly bracketed with
|
|
"set geqo to 'off'" and "reset geqo".
|
|
|
|
The interpretation of array specifiers (the curly braces around atomic
|
|
values) appears to have changed sometime after the original regression
|
|
tests were generated. The current ./expected/*.out files reflect this
|
|
new interpretation, which may not be correct!
|
|
|
|
The float8 regression test fails on at least some platforms. This is due
|
|
to differences in implementations of pow() and exp() and the signaling
|
|
mechanisms used for overflow and underflow conditions.
|
|
|
|
The "random" results in the random test should cause the "random" test
|
|
to be "failed", since the regression tests are evaluated using a simple
|
|
diff. However, "random" does not seem to produce random results on my
|
|
test machine (Linux/gcc/i686).
|
|
|
|
Sample timing results
|
|
|
|
Timing under Linux 2.0.27 seems to have a roughly 5% variation from run
|
|
to run, presumably due to the timing vagaries of multitasking systems.
|
|
|
|
Time System
|
|
06:12 Pentium Pro 180, 32MB, Linux 2.0.30, gcc 2.7.2 -O2 -m486
|
|
12:06 P-100, 48MB, Linux 2.0.29, gcc
|
|
39:58 Sparc IPC 32MB, Solaris 2.5, gcc 2.7.2.1 -O -g
|