Skip to content

ci: fix go module cache in Tekton pipelines or build a nightly base image #2533

@chmouel

Description

@chmouel

Problem

The Go cache in our internal Tekton CI pipelines does not appear to be working
effectively. The cache is configured in .tekton/go.yaml to use:

  • Cache source: oci://image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/cache-go:{{hash}}
  • Cache path: $(workspaces.source.path)/go-build-cache
  • Env vars: GOCACHE and GOMODCACHE pointing into the cache path

Despite this, Go modules appear to be re-downloaded on every pipeline run,
suggesting the cache fetch (cache-fetch StepAction) is not restoring correctly
or the hash key is changing too often.

Impact

  • Slower CI builds — every go.yaml pipeline run downloads all dependencies
  • Extra bandwidth usage on the OpenShift cluster
  • Slower PR feedback loop

Investigation needed

  • Verify the cache-fetch result (fetched=true/false) in recent pipeline runs
  • Check whether the OCI image cache-go:{{hash}} is being pushed successfully
    by cache-upload
  • Confirm GOCACHE / GOMODCACHE env vars are picked up by all steps
    (linting step uses a different image golangci/golangci-lint)
  • Check if the hash pattern ["**go.mod", "**go.sum"] is stable across runs

Alternative / Complementary Solution

Build a nightly Docker base image refreshed every night (e.g., via a
scheduled Tekton or GitHub Actions pipeline) that includes:

  • All required tools pre-installed: golangci-lint, kubectl, yq, ko, etc.
  • Go module cache pre-warmed with all current dependencies
  • Pinned to a daily tag (e.g., nightly-YYYYMMDD) for reproducibility

This image would replace docker.io/golang:1.25 in .tekton/go.yaml and
docker.io/golangci/golangci-lint:v2.10.1 in the linting step, eliminating
the need for the OCI cache mechanism entirely for the common case.

Pros of nightly image approach:

  • Simpler: no cache fetch/upload plumbing required
  • Faster cold starts — tools and modules are baked in
  • Single image pull instead of cache restore + module download

Cons:

  • Image push pipeline needed (maintenance overhead)
  • Slightly stale dependencies between nightly refreshes (acceptable)

Relevant files

  • .tekton/go.yaml — Go testing pipeline with cache config
  • .tekton/tasks/cache-fetch.yaml — StepAction for cache restore
  • .tekton/tasks/cache-upload.yaml — StepAction for cache save
  • .tekton/generate-coverage-release.yaml — Nightly pipeline (same cache config)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions