gitea/tests/integration
wxiaoguang 6bc3079c00
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)

## Review without space diff

https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1

## Purpose of this PR

1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command

## The main idea of this PR

* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
    * Before: `AddArguments("-m").AddDynamicArguments(message)`
    * After: `AddOptionValues("-m", message)`
    * -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`

## FAQ

### Why these changes were not done in #21535 ?

#21535 is mainly a search&replace, it did its best to not change too
much logic.

Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.


### The naming of `AddOptionXxx`

According to git's manual, the `--xxx` part is called `option`.

### How can it guarantee that `internal.CmdArg` won't be not misused?

Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.

And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.

### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?

Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.

### Why there was a `CmdArgCheck` and why it's removed?

At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.


### Why many codes for `signArg == ""` is deleted?

Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-04 10:30:43 +08:00
..
migration-test Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
schemas verify nodeinfo response by schema (#22137) 2022-12-17 01:22:34 -05:00
README.md Kd/ci playwright go test (#20123) 2022-09-02 15:18:23 -04:00
README_ZH.md Kd/ci playwright go test (#20123) 2022-09-02 15:18:23 -04:00
admin_user_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_activitypub_person_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_admin_org_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_admin_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_branch_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_comment_attachment_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_comment_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_feed_user_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_fork_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_gpg_keys_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_helper_for_declarative_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_httpsig_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_attachment_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_label_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_milestone_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_reaction_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_stopwatch_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_subscription_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_issue_tracked_time_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_keys_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_nodeinfo_test.go verify nodeinfo response by schema (#22137) 2022-12-17 01:22:34 -05:00
api_notification_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_oauth2_apps_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_org_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_packages_composer_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_conan_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_conda_test.go Add Conda package registry (#22262) 2023-02-01 12:30:39 -06:00
api_packages_container_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_packages_generic_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_helm_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_maven_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_npm_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_nuget_test.go Remove deprecated packages & staticcheck fixes (#22012) 2022-12-02 17:06:23 -05:00
api_packages_pub_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_pypi_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_rubygems_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_packages_test.go Check quota limits for container uploads (#22450) 2023-01-29 11:34:29 -06:00
api_packages_vagrant_test.go Remove deprecated packages & staticcheck fixes (#22012) 2022-12-02 17:06:23 -05:00
api_private_serv_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_pull_commits_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_pull_review_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_pull_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_releases_test.go Add API endpoint to get latest release (#21267) 2023-01-26 10:33:47 -06:00
api_repo_archive_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_collaborator_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_edit_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_file_create_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_file_delete_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_file_get_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_file_helpers.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_repo_file_update_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_get_contents_list_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_get_contents_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_git_blobs_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_git_commits_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_git_hook_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_git_notes_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_git_ref_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_git_tags_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_git_trees_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_hook_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_languages_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_repo_lfs_locks_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_repo_lfs_migrate_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_lfs_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_raw_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_tags_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_teams_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_repo_topic_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_settings_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_team_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_team_user_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_token_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_user_email_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_user_follow_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_user_heatmap_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_user_info_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
api_user_org_perm_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_user_orgs_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_user_search_test.go Remove session in api tests (#21984) 2022-12-01 22:39:42 -05:00
api_user_star_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_user_watch_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
api_wiki_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
attachment_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
auth_ldap_test.go Fix group filter for ldap source sync (#22506) 2023-02-02 15:45:00 +08:00
benchmarks_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
branches_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
change_default_branch_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
cmd_keys_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
compare_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
cors_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
create_no_session_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
csrf_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
delete_user_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
download_test.go Remove deprecated packages & staticcheck fixes (#22012) 2022-12-02 17:06:23 -05:00
dump_restore_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
editor_test.go Supports wildcard protected branch (#20825) 2023-01-16 16:00:22 +08:00
empty_repo_test.go Improve testing for pgsql empty repository (#22223) 2022-12-23 12:34:51 -06:00
eventsource_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
explore_repos_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
git_clone_wiki_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
git_helper_for_declarative_test.go Refactor git command package to improve security and maintainability (#22678) 2023-02-04 10:30:43 +08:00
git_smart_http_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
git_test.go Refactor git command package to improve security and maintainability (#22678) 2023-02-04 10:30:43 +08:00
goget_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
gpg_git_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
html_helper.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
incoming_email_test.go Add support for incoming emails (#22056) 2023-01-14 23:57:10 +08:00
integration_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
issue_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
lfs_getobject_test.go Use context parameter in models/git (#22367) 2023-01-09 11:50:54 +08:00
lfs_local_endpoint_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
lfs_view_test.go Test views of LFS files (#22196) 2022-12-23 07:41:56 +08:00
links_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
markup_external_test.go Add a simple test for external renderer (#20033) 2022-12-12 20:45:21 +08:00
migrate_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
mirror_pull_test.go Rename almost all Ctx functions (#22071) 2022-12-10 10:46:31 +08:00
mirror_push_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
nonascii_branches_test.go Remove deprecated packages & staticcheck fixes (#22012) 2022-12-02 17:06:23 -05:00
oauth_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
org_count_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
org_team_invite_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
org_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
private-testing.key Kd/ci playwright go test (#20123) 2022-09-02 15:18:23 -04:00
privateactivity_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
pull_compare_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
pull_create_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
pull_merge_test.go Refactor git command package to improve security and maintainability (#22678) 2023-02-04 10:30:43 +08:00
pull_review_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
pull_status_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
pull_update_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
release_test.go Release and Tag List tweaks (#21712) 2022-12-06 21:15:46 +08:00
rename_branch_test.go Remove deprecated packages & staticcheck fixes (#22012) 2022-12-02 17:06:23 -05:00
repo_activity_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_branch_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_commits_search_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_commits_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
repo_fork_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_generate_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_migrate_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_search_test.go refactor some functions to support ctx as first parameter (#21878) 2022-12-03 10:48:26 +08:00
repo_tag_test.go Use context parameter in models/git (#22367) 2023-01-09 11:50:54 +08:00
repo_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_topic_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repo_watch_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repofiles_delete_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
repofiles_update_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
setting_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
signin_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
signout_test.go Remove test session cache to reduce possible concurrent problem (#22199) 2022-12-22 21:09:35 +08:00
signup_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
ssh_key_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
timetracking_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
user_avatar_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
user_test.go Support scoped access tokens (#20908) 2023-01-17 15:46:03 -06:00
version_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
view_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
webfinger_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
xss_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00

README.md

Integration tests

Integration tests can be run with make commands for the appropriate backends, namely:

make test-sqlite
make test-pgsql
make test-mysql
make test-mysql8
make test-mssql

Make sure to perform a clean build before running tests:

make clean build

Run all tests via local drone

drone exec --local --build-event "pull_request"

Run sqlite integration tests

Start tests

make test-sqlite

Run MySQL integration tests

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql

Run pgsql integration tests

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql

Run mssql integration tests

Setup a mssql database inside docker

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql

Running individual tests

Example command to run GPG test:

For SQLite:

make test-sqlite#GPG

For other databases(replace mssql to mysql, mysql8 or pgsql):

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG

Setting timeouts for declaring long-tests and long-flushes

We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.

You can either:

  • Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
  • Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite