Commit Graph

53 Commits

Author SHA1 Message Date
Andres Freund a28166df8c ci: Make compute resources for CI configurable
See prior commit for an explanation for the goal of the change and why it had
to be split into two commits.

Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 15-, where CI support was added
2023-08-23 15:15:28 -07:00
Andres Freund 19cc96503d ci: Prepare to make compute resources for CI configurable
cirrus-ci will soon restrict the amount of free resources every user gets (as
have many other CI providers). For most users of CI that should not be an
issue. But e.g. for cfbot it will be an issue.

To allow configuring different resources on a per-repository basis, introduce
infrastructure for overriding the task execution environment. Unfortunately
this is not entirely trivial, as yaml anchors have to be defined before their
use, and cirrus-ci only allows injecting additional contents at the end of
.cirrus.yml.

To deal with that, move the definition of the CI tasks to
.cirrus.tasks.yml. The main .cirrus.yml is loaded first, then, if defined, the
file referenced by the REPO_CI_CONFIG_GIT_URL variable, will be added,
followed by the contents of .cirrus.tasks.yml. That allows
REPO_CI_CONFIG_GIT_URL to override the yaml anchors defined in .cirrus.yml.

Unfortunately git's default merge / rebase strategy does not handle copied
files, just renamed ones. To avoid painful rebasing over this change, this
commit just renames .cirrus.yml to .cirrus.tasks.yml, without adding a new
.cirrus.yml. That's done in the followup commit, which moves the relevant
portion of .cirrus.tasks.yml to .cirrus.yml.  Until that is done,
REPO_CI_CONFIG_GIT_URL does not fully work.

The subsequent commit adds documentation for how to configure custom compute
resources to src/tools/ci/README

Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 15-, where CI support was added
2023-08-23 15:15:28 -07:00
Andres Freund b2c91d841f ci: Use VMs for SanityCheck and CompilerWarnings
The main reason for this change is to reduce different ways of executing
tasks, making it easier to use custom compute resources for cfbot. A secondary
benefit is that the tasks seem slightly faster this way, apparently the
increased startup overhead is outweighed by reduced runtime overhead.

Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 15-, where CI support was added
2023-08-23 12:26:55 -07:00
Andres Freund 119ee6ab1b ci: Move execution method of tasks into yaml templates
This is done in preparation for making the compute resources for CI
configurable. It also looks cleaner.

Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 15-, where CI support was added
2023-08-23 12:26:38 -07:00
Andres Freund 794e14e219 ci: Don't specify amount of memory
The number of CPUs is the cost-determining factor. Most instance types that
run tests have more memory/core than what we specified, there's no real
benefit in wasting that.

Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 15-, where CI support was added
2023-08-23 12:26:12 -07:00
Andres Freund a2a6249cf1 ci: macos: use cached macports install
A significant chunk of the time on the macos CI task is spent installing
packages using homebrew. The downloads of the packages are cached, but the
installation needs to happen every time. We can't cache the whole homebrew
installation, because it is too large due to pre-installed packages.

Speed this up by installing packages using macports and caching the
installation as .dmg. That's a lot faster than unpacking a tarball.

In addition, don't install llvm - it wasn't enabled when building, so it's
just a waste of time/space.

This substantially speeds up the mac CI time, both in the cold cache and in
the warm cache case (the latter from ~1m20s to ~5s).

It doesn't seem great to have diverging sources of packages for CI between
branches, so backpatch to 15 (where CI was added).

Discussion: https://postgr.es/m/20230805202539.r3umyamsnctysdc7@awork3.anarazel.de
Backpatch: 15-, where CI was added
2023-08-19 14:38:46 -07:00
Andres Freund 3d8d217450 ci: macos: Remove use of -Dsegsize_blocks=6
The option causes a measurable slowdown. Macos is, by far, the most expensive
platform for CI, therefore it doesn't make sense to run such a test there.

d3b111e320 used a small segment size for two tasks, one with autoconf, one
with meson. In hindsight that is a bit overkill, it's unlikely that the option
would silently break. Thus don't move the -Dsegsize_blocks=6, just remove
it. I did however change the autoconf test to use 6 instead of 8 blocks, as
long as we allow it, a non-power-of-two test seems like a good idea.

While at it, add a comment explaining why we use a small segment size for CI.

Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 16-, where d3b111e320 introduced the use of -Dsegsize_blocks=6
2023-08-12 15:08:07 -07:00
Andres Freund 17ebbdf7de ci: macos: Remove use of -DRANDOMIZE_ALLOCATED_MEMORY
RANDOMIZE_ALLOCATED_MEMORY causes a measurable slowdown. Macos is, by far, the
most expensive platform for CI, therefore it doesn't make sense to run such a
test there.

Ubsan and asan on linux should detect most of the the cases of uninitialized
memory, so it doesn't really seem worth using -DRANDOMIZE_ALLOCATED_MEMORY in
another instance type.

Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qmn3w@awork3.anarazel.de
Backpatch: 16-, where 89d16b635 added the use of -DRANDOMIZE_ALLOCATED_MEMORY
2023-08-12 15:08:07 -07:00
Peter Eisentraut 9945750ae0 cirrus/freebsd: define ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
See also: 54100f5c6052404f68de9ce7310ceb61f1c2918f

Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://www.postgresql.org/message-id/flat/20220528153741.GK19626@telsasoft.com
2023-07-03 10:35:48 +02:00
Daniel Gustafsson c5e4ec293e ci: Remove OpenSSL 3.1 workaround for missing system CA
Commit 0b5d1fb36 fixed the SSL test to properly handle the error message
from OpenSSL when the system CA pool is missing of invalid, so with that
we can remove the workaround in Cirrus for when Homebrew clears the cert
from OPENSSLDIR.

Author: Jacob Champion <jchampion@timescale.com>
Discussion: https://postgr.es/m/CAAWbhmj3JtKuRcHcMuf1eCJmoZxMXdzLaYpEGmzu9+kgRGpu3A@mail.gmail.com
2023-04-24 11:40:23 +02:00
Jeff Davis fcb21b3acd Build ICU support by default.
Discussion: https://postgr.es/m/82c4c816-06f6-d3e3-ba02-fca4a5cef065%40enterprisedb.com
Reviewed-by: Peter Eisentraut
2023-04-18 13:25:44 -07:00
Daniel Gustafsson 8eda731465 Allow to use system CA pool for certificate verification
This adds a new option to libpq's sslrootcert, "system", which will load
the system trusted CA roots for certificate verification. This is a more
convenient way to achieve this than pointing to the system CA roots
manually since the location can differ by installation and be locally
adjusted by env vars in OpenSSL.

When sslrootcert is set to system, sslmode is forced to be verify-full
as weaker modes aren't providing much security for public CAs.

Changing the location of the system roots by setting environment vars is
not supported by LibreSSL so the tests will use a heuristic to determine
if the system being tested is LibreSSL or OpenSSL.

The workaround in .cirrus.yml is required to handle a strange interaction
between homebrew and the openssl@3 formula; hopefully this can be removed
in the near future.

The original patch was written by Thomas Habets, which was later revived
by Jacob Champion.

Author: Jacob Champion <jchampion@timescale.com>
Author: Thomas Habets <thomas@habets.se>
Reviewed-by: Jelte Fennema <postgres@jeltef.nl>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Magnus Hagander <magnus@hagander.net>
Discussion: https://www.postgresql.org/message-id/flat/CA%2BkHd%2BcJwCUxVb-Gj_0ptr3_KZPwi3%2B67vK6HnLFBK9MzuYrLA%40mail.gmail.com
2023-04-05 23:22:17 +02:00
Daniel Gustafsson 7f5b19817e Support connection load balancing in libpq
This adds support for load balancing connections with libpq using a
connection parameter: load_balance_hosts=<string>. When setting the
param to random, hosts and addresses will be connected to in random
order. This then results in load balancing across these addresses and
hosts when multiple clients or frequent connection setups are used.

The randomization employed performs two levels of shuffling:

  1. The given hosts are randomly shuffled, before resolving them
     one-by-one.
  2. Once a host its addresses get resolved, the returned addresses
     are shuffled, before trying to connect to them one-by-one.

Author: Jelte Fennema <postgres@jeltef.nl>
Reviewed-by: Aleksander Alekseev <aleksander@timescale.com>
Reviewed-by: Michael Banck <mbanck@gmx.net>
Reviewed-by: Andrey Borodin <amborodin86@gmail.com>
Discussion: https://postgr.es/m/PR3PR83MB04768E2FF04818EEB2179949F7A69@PR3PR83MB0476.EURPRD83.prod.outlook.
2023-03-29 21:53:38 +02:00
Peter Eisentraut bf32ec2256 cirrus/ccache: Use G rather than GB suffix
The former being the documented spelling.

Author: Justin Pryzby <pryzbyj@telsasoft.com>
Discussion: https://www.postgresql.org/message-id/20230203142656.GA1653%40telsasoft.com
2023-03-13 07:23:45 +01:00
Peter Eisentraut 6a3002715e meson: Make auto the default of the ssl option
The 'ssl' option is of type 'combo', but we add a choice 'auto' that
simulates the behavior of a feature option.  This way, openssl is used
automatically by default if present, but we retain the ability to
potentially select another ssl library.

Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/ad65ffd1-a9a7-fda1-59c6-f7dc763c3051%40enterprisedb.com
2023-03-13 07:04:11 +01:00
Andres Freund 209f0f0e85 ci: freebsd: stop postgres instance on failure, to prevent log upload failures
Discussion: https://postgr.es/m/20230205233057.e2243nl44agf7c3i@alap3.anarazel.de
Discussion: https://postgr.es/m/20230206221422.ykidlne7pmywu4ay@awork3.anarazel.de
2023-02-06 15:41:38 -08:00
Andres Freund 98811323c8 ci: Use windows VMs instead of windows containers
So far we have used containers for testing windows on cirrus-ci. Unfortunately
they come with substantial overhead: First, the container images are pulled
onto the host on-demand. Due to the large size of windows containers, that
ends up taking nearly 4 minutes. Secondly, IO is slow, leading to CI runs
taking long.

Thus switch to windows VMs, improving windows CI times by well over 2x.

Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/211afb88-6df6-b74d-f1b7-84b5f21ad875@gmail.com
Backpatch: 15-, where CI was added
2023-02-02 21:31:24 -08:00
Thomas Munro 2e9f120b65 ci: Upgrade macOS version from 12 to 13.
Back-patch to 15, where in-tree CI began.

Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/1441145.1675300332%40sss.pgh.pa.us
2023-02-03 14:26:54 +13:00
Michael Paquier 33ab0a2a52 Fix typos in comments, code and documentation
While on it, newlines are removed from the end of two elog() strings.
The others are simple grammar mistakes.  One comment in pg_upgrade
referred incorrectly to sequences since a7e5457.

Author: Justin Pryzby
Discussion: https://postgr.es/m/20221230231257.GI1153@telsasoft.com
Backpatch-through: 11
2023-01-03 16:26:14 +09:00
Thomas Munro 14d63dd252 ci: Change macOS builds from Intel to ARM.
Cirrus is about to shut down its macOS-on-Intel support, so it's time to
move our CI testing over to ARM instances.  The Homebrew package manager
changed its default installation prefix for the new architecture, so a
couple of tests need tweaks to find binaries.

Back-patch to 15, where in-tree CI began.

Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20221122225744.GF11463%40telsasoft.com
2023-01-01 10:45:18 +13:00
Andres Freund d3b111e320 Add option to specify segment size in blocks
The tests don't have much coverage of segment related code, as we don't create
large enough tables. To make it easier to test these paths, add a new option
specifying the segment size in blocks.

Set the new option to 6 blocks in one of the CI tasks. Smaller numbers
currently fail one of the tests, for understandable reasons.

While at it, fix some segment size related issues in the meson build.

Author: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20221107171355.c23fzwanfzq2pmgt@awork3.anarazel.de
2022-12-07 19:32:59 -08:00
Andres Freund 3f0e786ccb meson: Add 'running' test setup, as a replacement for installcheck
To run all tests that support running against existing server:
$ meson test --setup running

To run just the main pg_regress tests against existing server:
$ meson test --setup running regress-running/regress

To ensure the 'running' setup continues to work, test it as part of the
freebsd CI task.

Discussion: https://postgr.es/m/CAH2-Wz=XDQcmLoo7RR_i6FKQdDmcyb9q5gStnfuuQXrOGhB2sQ@mail.gmail.com
2022-12-07 12:13:35 -08:00
Andres Freund bd82928625 ci: Use -fsanitize=undefined,alignment,address in linux tasks
We have coverage of the various sanitizers in the buildfarm. The sanitizers
however particularly interesting during the development of patches, where the
likelihood of bugs is even higher. There also have been complaints about only
seeing such failures on the buildfarm, rather than before commit.

This commit enables a reasonable set of sanitizers in CI. Use the linux task
for that, as it currently is one of the fastests tasks. Also several of the
sanitizers work best on linux.

The overhead of alignment sanitizer is low, undefined behaviour has moderate
overhead. Test alignment sanitizer in the meson task, as it does both 32 and
64 bit builds and is thus more likely to expose alignment bugs.

Address sanitizer in contrast somewhat expensive. Enable it in the autoconf
task, as the meson task tests both 32 and 64bit which would exacerbate the
cost.

Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20220323173537.ll7klrglnp4gn2um@alap3.anarazel.de
Discussion: https://postgr.es/m/20221121220903.kf5u7rokfzbmqskm@alap3.anarazel.de
2022-11-21 15:13:09 -08:00
Andres Freund 94a3e026cc ci: Introduce SanityCheck task that other tasks depend on
To avoid unnecessarily spinning up a lot of VMs / containers for entirely
broken commits, have a minimal task that all others depend on.

The concrete motivation for the change is to use sanitizers in the linux
tasks. As that makes the tests slower, the start of the CompilerWarnings would
be delayed even more. With this change the CompilerWarnings only depends on
the SanityCheck task.

This has the added advantage that now the CompilerWarnings task is not
prevented from running by (most) test failures (particularly annoying when
caused by a test that is flappy in HEAD).

Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20221002205201.injtofbx4ax4erww@awork3.anarazel.de
2022-11-21 15:13:09 -08:00
Andres Freund ec267fd5a5 ci: Clean up pre-meson cruft in windows task
We don't need CIRRUS_ESCAPING_PROCESSES anymore as the whole tests now run
within a single script: block. We don't need NO_TEMP_INSTALL anymore it was
addressing an issue specific to vcregress.pl.

Author: Justin Pryzby <pryzbyj@telsasoft.com>
Discussion: https://postgr.es/m/20221113235303.GA26337@telsasoft.com
2022-11-21 14:30:50 -08:00
Andres Freund 967db242c2 ci: Add task testing windows with mingw
For now the task has been set to be manually triggered, as we are already
limited by the amount of CI time available for windows, particularly on cfbot.

Author: Melih Mutlu <m.melihmutlu@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAGPVpCSKS9E0An4=e7ZDnme+y=WOcQFJYJegKO8kE9=gh8NJKQ@mail.gmail.com
2022-11-17 16:22:25 -08:00
Andres Freund e5555657ba meson: Add support for building with precompiled headers
This substantially speeds up building for windows, due to the vast amount of
headers included via windows.h. A cross build from linux targetting mingw goes
from

994.11user 136.43system 0:31.58elapsed 3579%CPU
to
422.41user 89.05system 0:14.35elapsed 3562%CPU

The wins on windows are similar-ish (but I don't have a system at hand just
now for actual numbers). Targetting other operating systems the wins are far
smaller (tested linux, macOS, FreeBSD).

For now precompiled headers are disabled by default, it's not clear how well
they work on all platforms. E.g. on FreeBSD gcc doesn't seem to have working
support, but clang does.

When doing a full build precompiled headers are only beneficial for targets
with multiple .c files, as meson builds a separate precompiled header for each
target (so that different compilation options take effect). This commit
therefore only changes target with at least two .c files to use precompiled
headers.

Because this commit adds b_pch=false to the default_options new build
directories will have precompiled headers disabled by default, however
existing build directories will continue use the default value of b_pch, which
is true.

Note that using precompiled headers with ccache requires setting
CCACHE_SLOPPINESS=pch_defines,time_macros to get hits.

Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/CA+hUKG+50eOUbN++ocDc0Qnp9Pvmou23DSXu=ZA6fepOcftKqA@mail.gmail.com
Discussion: https://postgr.es/m/c5736f70-bb6d-8d25-e35c-e3d886e4e905@enterprisedb.com
Discussion: https://postgr.es/m/20190826054000.GE7005%40paquier.xyz
2022-10-06 17:19:30 -07:00
Andres Freund 89d16b6352 ci: enable various runtime checks on FreeBSD and macOS
Increase likelihood of discovering problems during CI rather the buildfarm by
defining -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES
-DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST on FreeBSD, and
-DRANDOMIZE_ALLOCATED_MEMORY on macOS.

FreeBSD and macoS are currently not CI bottlenecks, so we can afford to
increase their test times a bit.

Author: Justin Pryzby <pryzbyj@telsasoft.com>
Discussion: https://postgr.es/m/20220910200542.GX31833@telsasoft.com
2022-10-01 17:39:53 -07:00
Andres Freund 2d2f083104 ci: macos: Reduce test concurrency
Test performance regresses noticably when using all cores. This is more
pronounced with meson than with autoconf, presumably because meson will
schedule the "full number" of tests more consistently.  8 seems to work
OK.

Discussion: https://postgr.es/m/20220927040208.l3shfcidovpzqxfh@awork3.anarazel.de
Backpatch: 15-, where CI was introduced
2022-10-01 16:55:16 -07:00
Andres Freund ccf36ea258 ci: Add 32bit build and test
It's easy enough to make changes that break on 32bit platforms and few people
test that locally. Add a test for that to CI. LLVM is disabled on 32bit
because installing it would bloat the image size further.

The debian w/ meson task is fast enough that we can afford to test both.

Use the occasion of a separate run of the tests to run them under LANG=C,
we've recently discovered there's not a lot of testing in the buildfarm for
the case.

Discussion: https://postgr.es/m/4033181.1664395633@sss.pgh.pa.us
2022-09-29 20:32:20 -07:00
Andres Freund 03bf971d2d Remove uses of register due to incompatibility with C++17 and up
The use in regexec.c could remain, since we only try to keep headers C++
clean. But there really doesn't seem to be a good reason to use register in
that spot.

Discussion: https://postgr.es/m/20220308185902.ibdqmasoaunzjrfc@alap3.anarazel.de
2022-09-24 12:08:37 -07:00
Andres Freund e3e6ee6964 ci: freebsd: Set extra_{lib,include}_dirs to /usr/local/...
Ommitted in e6b6ea025c, but necessary for libintl to be found. All other
dependencies can be found via pkg-config (which searches in /usr/local/...)
and thus worked even without adding the directories.
2022-09-22 20:42:06 -07:00
Andres Freund e6b6ea025c meson: ci: Convert some tasks to use meson
The Windows task is changed to use meson as there currently is no way to run
all tests in the old MSVC build system (only ninja is covered for now, we
don't have enough CI resources to test msbuild as well).

To maintain autoconf coverage, the Linux task is duplicated to test both meson
and autoconf builds (linux is currently the fastest task). FreeBSD and macOS
are also converted to meson, as it seems more important to have coverage for
meson than autoconf.

Author: Andres Freund <andres@anarazel.de>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Author: Justin Pryzby <pryzby@telsasoft.com>
2022-09-21 22:46:18 -07:00
Andres Freund 661ee7bfc6 ci: windows: set error mode to not include SEM_NOGPFAULTERRORBOX
Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That prevents
crash reporting from working unless binaries do SetErrorMode()
themselves. Furthermore, it appears that either python or, more likely, the C
runtime has a bug where SEM_NOGPFAULTERRORBOX can very occasionally *trigger*
a crash on process exit - which is hard to debug, given that it explicitly
prevents crash dumps from working...

Discussion: https://postgr.es/m/20220909235836.lz3igxtkcjb5w7zb%40awork3.anarazel.de
Backpatch: 15-, where CI was added
2022-09-21 17:16:34 -07:00
Thomas Munro a3a3c1ea87 ci: Increase requested memory size.
CI builds recently started failing with:

"Memory size for 4.0 vCPU instance should be between 3840MiB and
26624MiB, while 2048MiB is requested."

Ok then, let's ask for 4G instead of 2G.

This may be due to a change in the type of instance used to work around
an outage, per:

https://twitter.com/cirrus_labs/status/1572657320093712384
2022-09-22 11:36:19 +12:00
Andres Freund 2827f108d1 ci: remove minor version from freebsd image name
This way .cirrus.yml does not need to be changed just because freebsd releases
a new minor version.

Discussion: https://postgr.es/m/20220728095704.ryywoaz4dqqrwstc@alap3.anarazel.de
Backpatch: 15-, just like the CI support
2022-07-31 18:51:47 -07:00
Andres Freund e1c95e50e9 ci: switch to freebsd 13.1
freebsd 13.0 is out of support, switch to 13.1. It might be a good idea to
remove the minor version number from the image name, but there's not been a
response to that so far...

Discussion: https://postgr.es/m/20220728095704.ryywoaz4dqqrwstc@alap3.anarazel.de
Backpatch: 15-, just like the CI support
2022-07-31 12:31:36 -07:00
Michael Paquier d2a2ce4184 Make upgradecheck a no-op in MSVC's vcregress.pl
322becb has changed upgradecheck to use the TAP tests, discarding
pg_upgrade's tests in bincheck.  However, this is proving to be a bad
idea for the Windows buildfarm clients that use MSVC when TAP tests are
disabled as this causes a hard failure at the pg_upgrade step.

This commit disables upgradecheck, moving the execution of the tests of
pg_upgrade to bincheck, as per an initial suggestion from Andres
Freund, so as the buildfarm is able to live happily with those changes.

While on it, remove the routine that was used by upgradecheck to
create databases whose names are generated with a range of ASCII
characters as it is not used since 322becb.  upgradecheck is removed
from the CI script for Windows, as bincheck takes care of that now.

Per report from buildfarm member hamerkop (MSVC 2017 without a TAP
setup).

Reviewed-by: Justin Pryzby
Discussion: https://postgr.es/m/YkbnpriYEAagZ2wH@paquier.xyz
2022-04-02 12:06:11 +09:00
Andres Freund ff50baec65 ci: enable zstd where available.
Since zstd is now used in a bunch of places, enable it in CI. The windows
image unfortunately doesn't yet contain zstd, so it's not enabled there.

Discussion: https://postgr.es/m/20220330155017.lfnlzt3m42nk7kff@alap3.anarazel.de
2022-03-30 09:33:28 -07:00
Andres Freund 81b9f23c9c ci: test headerscheck, cpluspluscheck as part of CompilerWarnings task.
Discussion: https://postgr.es/m/20220323002024.f2g6tivduzrktgfa@alap3.anarazel.de
2022-03-23 11:33:57 -07:00
Andres Freund 50b1e8c51b ci: windows: Use CIRRUS_ESCAPING_PROCESSES, revert 770011e3f3.
cirrus-ci now defaults to killing processes still running at the end of a
script. Unfortunately we start postgres in the background, which seems
nontrivial to fix. Previously we worked around that in 770011e3f3 by using an
older agent version, but now that CIRRUS_ESCAPING_PROCESSES we should use that.

This reverts commit 770011e3f3 "ci: windows:
Work around cirrus-ci bug causing test failures.

Discussion: https://postgr.es/m/CA+hUKGKx7k14n2nAALSvv6M_AB6oHasNBA65X6Dvo8hwfi9y0A@mail.gmail.com
2022-03-19 11:42:22 -07:00
Andres Freund 4a288a37f9 ci: compile with -Og where applicable.
To improve performance of check-world, and improve debugging, without
significantly slower builds (they're cached anyway).

This makes freebsd check-world run in 8.5 minutes rather than 15 minutes.

Author: Justin Pryzby <pryzbyj@telsasoft.com>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220310220611.GH28503@telsasoft.com
2022-03-18 15:42:33 -07:00
Andres Freund 225fb558cd ci: include hints how to install OS packages.
This is useful for patches during development, but once a feature is merged,
new libraries should be added to the OS image files, rather than installed
during every CI run forever into the future.

Author: Justin Pryzby <pryzbyj@telsasoft.com>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220310220611.GH28503@telsasoft.com
2022-03-18 15:42:02 -07:00
Andres Freund 770011e3f3 ci: windows: Work around cirrus-ci bug causing test failures.
Will be reverted once fixed on cirrus's side.

See also https://github.com/cirruslabs/cirrus-ci-agent/issues/218

Discussion: https://postgr.es/m/CA+hUKGKx7k14n2nAALSvv6M_AB6oHasNBA65X6Dvo8hwfi9y0A@mail.gmail.com
2022-03-04 22:00:09 -08:00
Andres Freund f79c94f39d ci: fix copy-paste mistake in 16eb8231d1.
Reported-By: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20220213024704.GN31460@telsasoft.com
2022-02-12 19:38:06 -08:00
Andres Freund 0a156058aa ci: macos: align sysinfo_script to other tasks.
Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20220203035827.GG23027@telsasoft.com
2022-02-12 15:53:30 -08:00
Andres Freund 16eb8231d1 ci: Only use one artifact instruction for logs.
Having the different types of logs in artifacts instruction makes it quicker
to navigate between them. Either that didn't use to be possible, or I just did
it wrong. I saw that / how it works in a patch by Justin Pryzby.

Also remove a few unnecessary uses of **, suggested by Justin Pryzby.

Discussion: https://postgr.es/m/20220203195718.smqo5xg4ygp5qktq@alap3.anarazel.de
2022-02-12 15:52:36 -08:00
Andres Freund 79b79dd57a ci: s/CCACHE_SIZE/CCACHE_MAXSIZE/.
The wrong environment variable was used in one place.

Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/20220203035827.GG23027@telsasoft.com
2022-02-12 15:52:36 -08:00
Andres Freund f3feff8259 windows: Improve crash / assert / exception handling.
startup_hacks() called SetErrorMode() with the SEM_NOGPFAULTERRORBOX argument
to prevent GUI popups on error. While that likely was sufficient at some
point, there are other sources of error popups.

At the same time SEM_NOGPFAULTERRORBOX unfortunately also prevents
"just-in-time debuggers" from working reliably, i.e. the ability to attach to
a process on crash. This prevents collecting crash dumps as part of CI.

The error popups are particularly problematic when they occur during automated
testing, as they can cause the tests to hang, waiting for a button to be
clicked.

This commit improves the error handling setup in startup_hacks() to address
those problems. SEM_NOGPFAULTERRORBOX is not used anymore, instead various
other APIs are used to disable popups and to redirect output to stderr where
possible.

While this improves the situation for postgres.exe, it doesn't address similar
issues in all the other executables. There currently is no codepath that's
called early on for all frontend programs.

I've tested that this prevents GUI popups and allows JIT debugging in case of
crashes due to:
- abort()
- assert()
- C runtime errors
- unhandled exceptions
both in debug and non-debug mode, on Win10 with MSVC 2019 and with MinGW.

Now that crash reports are generated on windows, collect them in windows CI.

Discussion: https://postgr.es/m/20211005193033.tg4pqswgvu3hcolm@alap3.anarazel.de
2022-02-02 18:33:25 -08:00
Andres Freund 6dcc185266 ci: windows: run tests under timeout.
On windows ci/cfbot currently regularly hangs / times out. Presumably this is due
to the issues discussed in
https://postgr.es/m/CA%2BhUKG%2BG5DUNJfdE-qusq5pcj6omYTuWmmFuxCvs%3Dq1jNjkKKA%40mail.gmail.com
which lead to reverting 75674c7ec1 everywhere but master.

But it's hard to tell - because the entire test task times out, we don't get
to see debugging information.

This commit adds a timeout (using git's timeout binary) to all vcregress
invocations, which should address the problem for now. It might also be a good
idea to add timeouts to the other operating systems at a later time.

The diff is a bit larger than one might think necessary: Yaml doesn't like % -
from the windows command variable syntax - at the start of an unquoted
string...

Discussion: https://postgr.es/m/20220110005704.es4el6i2nxlxzwof@alap3.anarazel.de
2022-02-02 17:31:54 -08:00