-
Notifications
You must be signed in to change notification settings - Fork 198
Improve E2E test debuggability in CI #4077
Description
Summary
When E2E tests fail in CI, we lack the information needed to diagnose root causes. A recent api-clients failure returned HTTP 500 instead of 201 during workload creation, but we couldn't determine why because:
- Response bodies aren't logged on failure — tests assert status codes but don't dump the response body when assertions fail
thv servelogs aren't captured — the API server's stdout/stderr isn't saved as a CI artifact- Ginkgo report files aren't generated — the upload step consistently finds no files at
test/e2e/ginkgo-report.xml
Proposed improvements
1. Log response bodies on assertion failure
Test helpers like createWorkload, registerClient, etc. should log the response body when the status code doesn't match expectations. Example pattern:
resp := createWorkload(apiServer, workloadReq)
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
Expect(resp.StatusCode).To(Equal(http.StatusCreated),
"Unexpected status code. Response body: %s", string(body))2. Capture thv serve logs as CI artifacts
The E2E test runner or helpers should redirect thv serve stderr/stdout to a log file, and the CI workflow should upload it alongside test results.
3. Fix Ginkgo report generation
The workflow expects test/e2e/ginkgo-report.xml and test/e2e/junit-report.xml but they're never created. Either:
- Pass
--junit-report=junit-report.xmltoginkgoinrun_tests.sh - Or remove the upload step if reports aren't needed
4. Consider adding --vv output on failure
When a test fails, having verbose Ginkgo output (--vv) would show the full step timeline, making it easier to trace what happened.
Context
- Example failure: https://github.com/stacklok/toolhive/actions/runs/22908568453/job/66475961041
- The test
should successfully register client with default groupgot HTTP 500 fromcreateWorkloadafter ~60 seconds (likely a container image pull timeout), but we couldn't confirm without server logs or the error response body.
Generated with Claude Code