chore: collect coverage using GOCOVERDIR (#9004)

- run all unit tests and integration tests in sequence in the same job
- upload the coverage data as an artifact to the run
- only run via dispatch because it is very long
- arguments are used to select which unit or integration tests to focus on

Refs https://go.dev/blog/integration-test-coverage

### Testing

- Run this workflow via workflow_dispatch (see https://codeberg.org/forgejo-integration/forgejo/actions/runs/11889)
- Download the coverage information (see https://codeberg.org/forgejo-integration/forgejo/actions/runs/11889/artifacts/coverage)
- Download the coverage information from an end-to-end cascade test (see https://code.forgejo.org/forgejo/end-to-end/actions/runs/3961)
- Unzip them in a the `coverage/data` sub-directory at the root of the Forgejo source tree
    ```sh
    $ ls coverage/data
    actions  coverage-actions.zip  coverage-federation.zip  coverage-tests.zip  coverage-upgrade.zip  federation  tests  upgrade
    ```
- Display the coverage percentage
```
$ make coverage-show-percentage
...
forgejo.org/services/wiki/wiki_path.go:96:					WebPathToGitPath						81.8%
forgejo.org/services/wiki/wiki_path.go:113:					GitPathToWebPath						90.0%
forgejo.org/services/wiki/wiki_path.go:129:					WebPathToUserTitle						71.4%
forgejo.org/services/wiki/wiki_path.go:140:					WebPathToURLPath						100.0%
forgejo.org/services/wiki/wiki_path.go:144:					WebPathFromRequest						100.0%
forgejo.org/services/wiki/wiki_path.go:151:					UserTitleToWebPath						80.0%
forgejo.org/services/wiki/wiki_path.go:163:					ToWikiPageMetaData						100.0%
forgejo.org/tests/integration/api_repo_file_helpers.go:18:			createFileInBranch						100.0%
...
total:										(statements)							63.9%
```

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9004
Reviewed-by: jerger <jerger@noreply.codeberg.org>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>

.forgejo/workflows/coverage.yml

@@ -0,0 +1,89 @@
+name: coverage
+
+on:
+  workflow_dispatch:
+    inputs:
+      repository:
+        description: 'repository'
+        type: string
+      ref:
+        description: 'ref'
+        type: string
+      unit-tests-env:
+        description: 'COVERAGE_TEST_PACKAGES=forgejo.org/modules/actions'
+        type: string
+      integration-tests-env:
+        description: 'COVERAGE_TEST_ARGS=-run=TestAPIListRepoComments'
+        type: string
+
+jobs:
+  all:
+    runs-on: docker
+    container:
+      image: 'data.forgejo.org/oci/ci:1'
+      options: --tmpfs /tmp:exec,noatime
+    services:
+      elasticsearch:
+        image: data.forgejo.org/oci/bitnami/elasticsearch:7
+        options: --tmpfs /bitnami/elasticsearch/data
+        env:
+          discovery.type: single-node
+          ES_JAVA_OPTS: "-Xms512m -Xmx512m"
+      minio:
+        image: data.forgejo.org/oci/bitnami/minio:2024.8.17
+        options: >-
+          --hostname gitea.minio --tmpfs /bitnami/minio/data:noatime
+        env:
+          MINIO_DOMAIN: minio
+          MINIO_ROOT_USER: 123456
+          MINIO_ROOT_PASSWORD: 12345678
+      mysql:
+        image: 'data.forgejo.org/oci/bitnami/mysql:8.4'
+        env:
+          ALLOW_EMPTY_PASSWORD: yes
+          MYSQL_DATABASE: testgitea
+          #
+          # See also https://codeberg.org/forgejo/forgejo/issues/976
+          #
+          MYSQL_EXTRA_FLAGS: --innodb-adaptive-flushing=OFF --innodb-buffer-pool-size=4G --innodb-log-buffer-size=128M --innodb-flush-log-at-trx-commit=0 --innodb-flush-log-at-timeout=30 --innodb-flush-method=nosync --innodb-fsync-threshold=1000000000 --disable-log-bin
+        options: --tmpfs /bitnami/mysql/data:noatime
+      ldap:
+        image: data.forgejo.org/oci/test-openldap:latest
+      pgsql:
+        image: data.forgejo.org/oci/bitnami/postgresql:16
+        env:
+          POSTGRESQL_DATABASE: test
+          POSTGRESQL_PASSWORD: postgres
+          POSTGRESQL_FSYNC: off
+          POSTGRESQL_EXTRA_FLAGS: -c full_page_writes=off
+        options: --tmpfs /bitnami/postgresql
+      cacher:
+        image: registry.redict.io/redict:7.3.0-scratch
+        options: --tmpfs /data:noatime
+    steps:
+      - uses: https://data.forgejo.org/actions/checkout@v4
+        with:
+          repository: ${{ inputs.repository }}
+          ref: ${{ inputs.ref }}
+      - uses: ./.forgejo/workflows-composite/setup-env
+      - name: install git >= 2.42
+        uses: ./.forgejo/workflows-composite/apt-install-from
+        with:
+          packages: git
+      - uses: ./.forgejo/workflows-composite/build-backend
+      - run: |
+          su forgejo -c '${{ inputs.unit-tests-env }} make coverage-run'
+          su forgejo -c '${{ inputs.integration-tests-env }} make coverage-run-pgsql'
+          su forgejo -c '${{ inputs.integration-tests-env }} make coverage-run-mysql'
+          su forgejo -c '${{ inputs.integration-tests-env }} make coverage-run-sqlite'
+          su forgejo -c 'make coverage-merge'
+        timeout-minutes: 180
+        env:
+          TEST_ELASTICSEARCH_URL: http://elasticsearch:9200
+          TEST_MINIO_ENDPOINT: minio:9000
+          TEST_LDAP: 1
+          TEST_REDIS_SERVER: cacher:6379
+      - uses: https://code.forgejo.org/forgejo/upload-artifact@v4
+        with:
+          name: coverage
+          path: ${{ forge.workspace }}/coverage/merged

Makefile

@@ -238,6 +238,9 @@ help:
 	@echo " - test-frontend-coverage           test frontend files and display code coverage"
 	@echo " - test-backend                     test backend files"
 	@echo " - test-remote-cacher               test backend files that use a remote cache"
+	@echo " - coverage-run*                    test and collect coverages in the coverage/data directory"
+	@echo " - coverage-show-html               display coverage-run results in an HTML page"
+	@echo " - coverage-show-percent            display coverage-run per package coverage percentage"
 	@echo " - test-e2e-sqlite[\#name.test.e2e] test end to end using playwright and sqlite"
 	@echo " - webpack                          build webpack files"
 	@echo " - svg                              build svg files"
@@ -556,16 +559,35 @@ test\#%:
 	@echo "Running go test with $(GOTESTFLAGS) -tags '$(TEST_TAGS)'..."
 	@$(GOTEST) $(GOTESTFLAGS) -tags='$(TEST_TAGS)' -run $(subst .,/,$*) $(GO_TEST_PACKAGES)
 
-.PHONY: coverage
+coverage-merge:
-coverage:
+	rm -fr coverage/merged ; mkdir -p coverage/merged
-	grep '^\(mode: .*\)\|\(.*:[0-9]\+\.[0-9]\+,[0-9]\+\.[0-9]\+ [0-9]\+ [0-9]\+\)$$' coverage.out > coverage-bodged.out
+	$(GO) tool covdata merge -i `find coverage/data -name 'covmeta.*' | sed -e 's|/covmeta.*|,|' | tr -d '\n' | sed -e 's/,$$//'` -o coverage/merged
-	grep '^\(mode: .*\)\|\(.*:[0-9]\+\.[0-9]\+,[0-9]\+\.[0-9]\+ [0-9]\+ [0-9]\+\)$$' integration.coverage.out > integration.coverage-bodged.out
-	$(GO) run build/gocovmerge.go integration.coverage-bodged.out coverage-bodged.out > coverage.all
 
-.PHONY: unit-test-coverage
+coverage-convert: coverage-merge
-unit-test-coverage:
+	$(GO) tool covdata textfmt -i=coverage/merged -o=coverage/textfmt.out
-	@echo "Running unit-test-coverage $(GOTESTFLAGS) -tags '$(TEST_TAGS)'..."
+
-	@$(GOTEST) $(GOTESTFLAGS) -timeout=20m -tags='$(TEST_TAGS)' -cover -coverprofile coverage.out $(GO_TEST_PACKAGES) && echo "\n==>\033[32m Ok\033[m\n" || exit 1
+coverage-show-html: coverage-convert
+	( cd coverage ; $(GO) tool cover -html=textfmt.out -o coverage.html )
+	xdg-open coverage/coverage.html
+
+coverage-show-percentage: coverage-convert
+	go tool cover -func=coverage/textfmt.out
+
+coverage-run:
+	contrib/coverage-helper.sh test_packages $(COVERAGE_TEST_PACKAGES)
+
+coverage-run-%: generate-ini-%
+  #
+  # Migration tests go first
+  #
+	$(MAKE) GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/$*.ini COVERAGE_TEST_ARGS= COVERAGE_TEST_PACKAGES=forgejo.org/tests/integration/migration-test coverage-run
+	for pkg in $(MIGRATION_PACKAGES); do \
+		$(MAKE) GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/$*.ini COVERAGE_TEST_DATABASE=$* COVERAGE_TEST_ARGS= COVERAGE_TEST_PACKAGES=$$pkg coverage-run ; \
+	done
+  #
+  # All other integration tests follow
+  #
+	$(MAKE) GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/$*.ini COVERAGE_TEST_DATABASE=$* COVERAGE_TEST_PACKAGES=forgejo.org/tests/integration coverage-run
 
 .PHONY: tidy
 tidy:
@@ -685,7 +707,7 @@ test-e2e-mysql\#%: playwright e2e.mysql.test generate-ini-mysql
 	GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/mysql.ini $(GOTESTCOMPILEDRUNPREFIX) ./e2e.mysql.test $(GOTESTCOMPILEDRUNSUFFIX) -test.run TestE2e/$*
 
 .PHONY: test-e2e-pgsql
-test-e2e-pgsql: playwright e2e.pgsql.test generate-ini-pgsql
+GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/sqlite.initest-e2e-pgsql: playwright e2e.pgsql.test generate-ini-pgsql
 	GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/pgsql.ini $(GOTESTCOMPILEDRUNPREFIX) ./e2e.pgsql.test $(GOTESTCOMPILEDRUNSUFFIX) -test.run TestE2e
 
 .PHONY: test-e2e-pgsql\#%
@@ -708,14 +730,6 @@ bench-mysql: integrations.mysql.test generate-ini-mysql
 bench-pgsql: integrations.pgsql.test generate-ini-pgsql
 	GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/pgsql.ini ./integrations.pgsql.test -test.cpuprofile=cpu.out -test.run DontRunTests -test.bench .
 
-.PHONY: integration-test-coverage
-integration-test-coverage: integrations.cover.test generate-ini-mysql
-	GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/mysql.ini ./integrations.cover.test -test.coverprofile=integration.coverage.out
-
-.PHONY: integration-test-coverage-sqlite
-integration-test-coverage-sqlite: integrations.cover.sqlite.test generate-ini-sqlite
-	GITEA_ROOT="$(CURDIR)" GITEA_CONF=tests/sqlite.ini ./integrations.cover.sqlite.test -test.coverprofile=integration.coverage.out
-
 integrations.mysql.test: git-check $(GO_SOURCES)
 	$(GOTEST) $(GOTESTFLAGS) -c forgejo.org/tests/integration -o integrations.mysql.test
 
@@ -725,12 +739,6 @@ integrations.pgsql.test: git-check $(GO_SOURCES)
 integrations.sqlite.test: git-check $(GO_SOURCES)
 	$(GOTEST) $(GOTESTFLAGS) -c forgejo.org/tests/integration -o integrations.sqlite.test -tags '$(TEST_TAGS)'
 
-integrations.cover.test: git-check $(GO_SOURCES)
-	$(GOTEST) $(GOTESTFLAGS) -c forgejo.org/tests/integration -coverpkg $(shell echo $(GO_TEST_PACKAGES) | tr ' ' ',') -o integrations.cover.test
-
-integrations.cover.sqlite.test: git-check $(GO_SOURCES)
-	$(GOTEST) $(GOTESTFLAGS) -c forgejo.org/tests/integration -coverpkg $(shell echo $(GO_TEST_PACKAGES) | tr ' ' ',') -o integrations.cover.sqlite.test -tags '$(TEST_TAGS)'
-
 .PHONY: migrations.mysql.test
 migrations.mysql.test: $(GO_SOURCES) generate-ini-mysql
 	$(GOTEST) $(GOTESTFLAGS) -c forgejo.org/tests/integration/migration-test -o migrations.mysql.test

contrib/coverage-helper.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+set -e
+#set -x
+PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}:  '
+
+#
+# Those must be explicitly required and are excluded from the full list of packages because they
+# would interfere with the testing fixtures.
+#
+excluded+='forgejo.org/models/migrations|'                # must be run before database specific tests
+excluded+='forgejo.org/models/forgejo_migrations|'        # must be run before database specific tests
+excluded+='forgejo.org/tests/integration/migration-test|' # must be run before database specific tests
+excluded+='forgejo.org/tests|'                            # only tests, no coverage to get there
+excluded+='forgejo.org/tests/e2e|'                        # JavaScript is not in scope here and if it adds coverage it should not be counted
+excluded+='FAKETERMINATOR'                                # do not modify
+
+: ${COVERAGEDIR:=$(pwd)/coverage/data}
+: ${GO:=$(go env GOROOT)/bin/go}
+
+DEFAULT_TEST_PACKAGES=$($GO list ./... | grep -E -v "$excluded")
+
+COVERED_PACKAGES=$($GO list ./...)
+COVERED_PACKAGES=$(echo $COVERED_PACKAGES | sed -e 's/ /,/g')
+
+function run_test() {
+  local package="$1"
+  if echo "$package" | grep --quiet --fixed-string ".."; then
+    echo "$package contains a suspicious .."
+    return 1
+  fi
+
+  local coverage="$COVERAGEDIR/$COVERAGE_TEST_DATABASE/$package"
+  rm -fr $coverage
+  mkdir -p $coverage
+
+  #
+  # -race cannot be used because it requires -covermode atomic which is
+  # different from the end-to-end tests and would cause issues wen merging
+  #
+  $GO test -timeout=20m -tags='sqlite sqlite_unlock_notify' -cover $package -coverpkg $COVERED_PACKAGES $COVERAGE_TEST_ARGS -args -test.gocoverdir=$coverage |& grep -v 'warning: no packages being tested depend on matches for pattern'
+}
+
+function test_packages() {
+  for package in ${@:-$DEFAULT_TEST_PACKAGES}; do
+    run_test $package
+  done
+}
+
+"$@"