diff options
| author | Will Chandler <wchandler@gitlab.com> | 2022-05-28 00:25:39 +0300 |
|---|---|---|
| committer | Will Chandler <wchandler@gitlab.com> | 2022-05-31 19:58:59 +0300 |
| commit | eaf3df45133d71ebb5493cadc19e0917af47bb6f (patch) | |
| tree | 7142f7a714ead73f95d0d64ae8b78940d4d2cb42 | |
| parent | e3aa6272e47af528b7599f93db2b02c76c55718e (diff) | |
ci: Capture panic stack traceswc-capture-panic
The `short` formatting option for `gotestsum` will usually suppress
panics and their stack traces, making it difficult to understand why a
panicking test failed.
Using `standard-verbose` will display the full trace, but this also
gives us the full output of `go test -v`, removing the readability
improvements that `gotestsum` provides.
To work around this, we use the `--jsonfile` option to write out the
full output of `go test -json`, then search for panics in an
`after_script` section with results in a pre-collapsed section.
To avoid unused log files piling up in local testing, by default the
full job output is written to `/dev/null`, with this setting overriden
in the CI config.
| -rw-r--r-- | .gitlab-ci.yml | 9 | ||||
| -rw-r--r-- | Makefile | 4 |
2 files changed, 12 insertions, 1 deletions
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6f832a963..916e6dfae 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -92,12 +92,21 @@ include: POSTGRES_HOST_AUTH_METHOD: trust TEST_REPORT: _unprivileged/go-tests-report.xml TEST_COVERAGE_DIR: _unprivileged + TEST_FULL_OUTPUT: /tmp/test-output.log before_script: &test_before_script - go version - while ! psql -h $PGHOST -U $PGUSER -c 'SELECT 1' > /dev/null; do echo "awaiting Postgres service to be ready..." && sleep 1 ; done && echo "Postgres service is ready!" # Create a directory for the unprivileged user that we're running tests as. # This is required so that we can still store test reports successfully. - install --directory --owner=${TEST_UID} --group=${TEST_UID} _unprivileged + after_script: + - | + # Checking for panics in ${TEST_FULL_OUTPUT} + if [ "${CI_JOB_STATUS}" = "failed" ] && grep 'Output":"panic' "${TEST_FULL_OUTPUT}" > /dev/null; then + echo -e "\e[0Ksection_start:`date +%s`:panic_stack_traces[collapsed=true]\r\e[0K\e[0;31mPanic stack traces\e[0m" + ruby -e "require 'json'; f = File.read(ENV['TEST_FULL_OUTPUT']); f.lines.each do |l| out = JSON.parse(l); puts out['Output']; end" | awk '/^panic/ || /goroutine/,/^\s*$/' + echo -e "\e[0Ksection_end:`date +%s`:panic_stack_traces\r\e[0K" + fi artifacts: paths: - ruby/tmp/gitaly-rspec-test.log @@ -222,6 +222,8 @@ TEST_OPTIONS ?= -count=1 ## Specify the output format used to print tests ["standard-verbose", "standard-quiet", "short"] TEST_FORMAT ?= short TEST_REPORT ?= ${BUILD_DIR}/reports/go-tests-report.xml +# Full output of `go test -json` +TEST_FULL_OUTPUT ?= /dev/null ## Specify the output directory for test coverage reports. TEST_COVERAGE_DIR ?= ${BUILD_DIR}/cover ## Directory where all runtime test data is being created. @@ -244,7 +246,7 @@ find_go_sources = $(shell find ${SOURCE_DIR} -type d \( -name ruby -o -nam # TEST_PACKAGES: packages which shall be tested run_go_tests = PATH='${SOURCE_DIR}/internal/testhelper/testdata/home/bin:${PATH}' \ TEST_TMP_DIR='${TEST_TMP_DIR}' \ - ${GOTESTSUM} --format ${TEST_FORMAT} --junitfile ${TEST_REPORT} -- -ldflags '${GO_LDFLAGS}' -tags '${SERVER_BUILD_TAGS},${GIT2GO_BUILD_TAGS}' ${TEST_OPTIONS} ${TEST_PACKAGES} + ${GOTESTSUM} --format ${TEST_FORMAT} --junitfile ${TEST_REPORT} --jsonfile ${TEST_FULL_OUTPUT} -- -ldflags '${GO_LDFLAGS}' -tags '${SERVER_BUILD_TAGS},${GIT2GO_BUILD_TAGS}' ${TEST_OPTIONS} ${TEST_PACKAGES} unexport GOROOT export GOCACHE ?= ${BUILD_DIR}/cache |