ci: migrate paid x86_64 CI workflows to self-hosted runner pool (#1483) * ci: add Hetzner self-hosted runner smoke test workflow Adds a workflow_dispatch + branch-pushed smoke test that targets the new SRE-managed Hetzner runner pool (fsn1-runner-01, 8 instances at tier:small/medium/large labels). Does NOT run on main or PRs. Provisioning context: shieldedtech/shielded-iac PR #1264. Signed-off-by: Scott Buckel * ci: TEST — flip main build job to Hetzner runner + enable Earthly cache Targets the new SRE-managed Hetzner runner pool (fsn1-runner-01) for the heavy 'Build node and images' job. Also flips EarthBuild use-cache from false to true so the persistent Earthly satellite carries layer cache across runs. ONLY for this test PR — revert before any merge to main. The Hetzner pool is currently a single AX162-R; production CI shouldn't depend on it until we have at least 2 boxes for redundancy. PR for the runner provisioning: shieldedtech/shielded-iac#1264. Signed-off-by: Scott Buckel * ci: revert EarthBuild use-cache=true — wrong semantics for our setup use-cache=true tells EarthBuild's setup-action to use GitHub Actions cache for Earthly state, including TLS-secured buildkitd certs. On a fresh runner with no prior GHA cache, those certs don't exist and Earthly fails: TLS CA file /opt/actions-runner/.earthly/certs/ca_cert.pem is missing We don't want GHA cache anyway — the Hetzner runner has a persistent local Earthly satellite at tcp://127.0.0.1:8372, accessed via EARTHLY_BUILDKIT_HOST env var (set by the ansible runner role). Reverting use-cache to false. The satellite is still doing its job; just not via the EarthBuild action's GHA-cache mechanism. Signed-off-by: Scott Buckel * ci: drop 'hetzner' label requirement from runs-on Self-hosted runner pool is the only one — 'self-hosted' + a tier label is enough to uniquely target it. Removing the 'hetzner' tag from runs-on lets the labels evolve without coupling workflow YAMLs to the hosting provider. Signed-off-by: Scott Buckel * ci: override workspace .earthly/config.yml to use Hetzner satellite The repo's .earthly/config.yml sets buildkit_additional_args and cache_size_pct, which tell earthly to manage its OWN local buildkitd with TLS. That config takes precedence over EARTHLY_BUILDKIT_HOST, so the env var is silently ignored and earthly tries to spin up a local buildkit container — failing because no TLS certs exist at ~/.earthly/certs/. Add a workflow step that overwrites .earthly/config.yml after checkout with one that points at the local satellite over plain TCP. Test-only; revert before merge. Signed-off-by: Scott Buckel * ci: drop buildkit_host override, just disable earthly TLS Switching strategy: instead of pointing earthly at our separately-managed 'earthly-satellite' container (which earthly treats as a port conflict because it thinks loopback buildkit_host = its own to manage), let earthly manage its own buildkitd at the default address. The only thing we need to override on midnight-node's repo config is TLS — earthly defaults to TLS-enabled but no certs are provisioned on the runner. Requires the 'earthly-satellite' systemd service to be stopped on the Hetzner runner so port 8372 is free for earthly's own managed buildkitd. Signed-off-by: Scott Buckel * ci: retrigger build after live runner .env patch Signed-off-by: Scott Buckel * ci: redirect earthly TMPDIR away from runner's PrivateTmp namespace The actions-runner@N.service has PrivateTmp=true, so /tmp lives in a per-service mount namespace. Earthly's own buildkitd is a sibling docker container (started by dockerd, not a child of the runner), so it can't see anything in the runner's private /tmp. Earthly's attempt to mkdir /tmp/earthly-cacheNNN fails as a result. Set TMPDIR=${{ runner.temp }} for the build step so earthly's tempdir lands under /var/lib/actions-runner/N/_temp/, which is outside any PrivateTmp namespace and visible to dockerd. Signed-off-by: Scott Buckel * ci: blank ACTIONS_CACHE_URL for earthly build step The Hetzner runner's .env sets ACTIONS_CACHE_URL to a local falcondev-oss cache server (so actions/cache can hit local NVMe). Earthly v0.8.16 auto-detects ACTIONS_CACHE_URL and tries to push build cache to that endpoint, authenticating with GHA's ACTIONS_RUNTIME_TOKEN. The local server expects a different auth (URL_ACCESS_TOKEN), so the push fails with insufficient_scope and the whole build returns exit 1 even though images already pushed successfully. On GHA-hosted runners EarthBuild/actions-setup --use-cache=false prevents this, but on our self-hosted runner the systemd EnvironmentFile re-exposes ACTIONS_CACHE_URL after the action runs. Override to empty in the step's env block — surgically scoped, actions/cache elsewhere still uses the local server. Signed-off-by: Scott Buckel * ci: log in to Docker Hub before the earthly build step midnight-node's Earthfile FROMs several Docker Hub images (rust, alpine, python, earthly/dind, and most importantly midnightntwrk/midnight-node-ci). With --push, earthly writes inline cache back to those base image repos. Pulling works anonymously (with rate-limit risk), but the cache push to midnightntwrk/midnight-node-ci needs auth — which is what's causing "error writing layer blob: push access denied... insufficient_scope" at the end of each otherwise-successful build. Reusing the DOCKERHUB_MIDNIGHTNTWRK_{USER,TOKEN} secrets that other workflows (sbom-scan-image, release-image, build-publish-contract-precompiles) already use. Signed-off-by: Scott Buckel * ci: belt-and-suspenders clear EARTHLY_REMOTE_CACHE in build step Auth URL decoded: scope=repository:library/local:pull,push service=registry.docker.io Earthly is using the literal string 'local' as remote-cache target, which Docker resolves to docker.io/library/local (the protected official-images namespace). That value came from an earlier EARTHLY_REMOTE_CACHE=local entry in the runner's .env. The .env has been cleaned but the value can persist in a runner process that was already executing when the restart was issued. Force-clear in the step env so any stale runner state is overridden for this invocation. Signed-off-by: Scott Buckel * ci: route paid x64 jobs (Test Toolkit, Mainnet Sync, Local Env Tests) to Hetzner Three jobs that were on GitHub's paid larger runners now target the Hetzner self-hosted pool: - Test Toolkit: ubuntu-latest-16-core-x64 -> tier:large - Mainnet Sync: ubuntu-latest-8-core-x64 -> tier:medium - Local Environment Tests: ubuntu-latest-8-core-x64 -> tier:medium The arm64 build job stays on ubuntu-latest-8-core-arm64 — our Hetzner host is EPYC x86_64 and we don't have an arm64 fleet yet. Public-repo cost impact per CI run: ~$0.49 of paid-runner spend shifts to the Hetzner box that's already paid for. Signed-off-by: Scott Buckel * ci: retry earthly --push up to 3x on transient layer-blob timeouts GHCR occasionally returns 'timeout awaiting response headers' partway through a layer upload from the Hetzner runner (longer net path than GHA-hosted runners). Buildkit's response-header timeout in the underlying containerd resolver is hardcoded around 30s and isn't tunable via buildkitd.toml, so we wrap the earthly call in a retry loop. Cost of a retry is minimal: earthly's cache short-circuits all the already-pushed layers, leaving only the failing layer to re-upload. Signed-off-by: Scott Buckel * ci: skip free-disk-space.sh on self-hosted runners scripts/free-disk-space.sh refuses to run on self-hosted runners ("Error: This script is only for GitHub-hosted runners. Refusing to run on local machine."), so the step crashes on Hetzner-routed jobs. Add 'runner.environment != self-hosted' to the conditionals for all three invocations (pallet fixtures, toolkit tests, local env tests), making the workflow self-protecting against future runner moves too. Signed-off-by: Scott Buckel * ci: apply Hetzner-runner earthly fixes to Test Toolkit job The Run build (linux/amd64) job has a stack of treatments to make earthly work cleanly on our self-hosted runner: - Docker Hub login (for inline cache push back to docker.io/midnightntwrk/*) - workspace .earthly/config.yml gets tls_enabled: false appended - TMPDIR redirected to $RUNNER_TEMP (PrivateTmp safe) - EARTHLY_REMOTE_CACHE/ACTIONS_CACHE_URL force-cleared Test Toolkit is a separate job with its own checkout and earthly invocation, so none of those treatments applied. Result: earthly saw a mismatched settings hash on the existing earthly-buildkitd container, tried to restart it with new settings, and failed because TLS defaults back on and no certs exist. Mirroring the full treatment here. Local Environment Tests will need the same set; deferring that since it goes through a composite action rather than calling earthly directly. Signed-off-by: Scott Buckel * ci: stop collapsing earthly retry output so errors are visible by default The ::group::/::endgroup:: wrapping I added with the retry loop is hiding the actual error inside a collapsed section — only 'Init' shows on the log when an attempt fails, you have to click into the group to see what went wrong. Drop the markers, keep the visual separator lines, expand always. Signed-off-by: Scott Buckel * fix(sync-test): fail loudly with logs when postgres never becomes ready run-sync.sh's pg_isready loop silently fell through if postgres failed to start within 60s, then proceeded to the snapshot load. psql would error out with the unrelated-looking "connection to server on socket ... No such file or directory" — psql looking for an in-container socket that doesn't exist because postgres isn't running. Track the loop result and on failure dump container state + postgres logs inline (independent of PRINT_LOGS) so the actual root cause is visible in the CI log. Signed-off-by: Scott Buckel * ci: re-trigger after Hetzner host firewall fix (allow br-* docker bridges) Signed-off-by: Scott Buckel * fix(ci/local-env-tests): disable earthly TLS so settings hash matches other Hetzner jobs The Build node and Test Toolkit jobs already append `tls_enabled: false` to .earthly/config.yml before invoking earthly. The local-environment-tests composite action didn't, so its earthly run had a different settings hash than the buildkitd container already running on the Hetzner host. Earthly's maybeRestart() / Stop() path then `docker stop`ped that container — which cancelled any concurrent earthly build (e.g. Test Toolkit's WITH DOCKER RUN microdnf step, which surfaces as "context canceled" mid-install). Signed-off-by: Scott Buckel * ci: re-trigger after host-level earthly tls_enabled=false applied Signed-off-by: Scott Buckel * fix(ci/local-env-tests): use block-scalar form for shell run YAML treats `: ` inside an unquoted scalar as a mapping indicator, so `run: echo " tls_enabled: false" >> ...` parsed as broken nested mapping ("Mapping values are not allowed in this context"). The shell- level double quotes don't matter to the YAML parser. Use the `run: |` block-scalar form that the workflow file already uses for the same line. Signed-off-by: Scott Buckel * chore(earthly): move buildkit memory cap to host-level config buildkit_additional_args -m sizes the earthly-buildkitd container against the host's available memory, so it's a property of the host, not the build. Lives now in the Hetzner runner host's ~/.earthly/config.yml (managed by shielded-iac/hetzner ansible) — non-Hetzner environments (laptops, github-hosted runners) fall back to earthly's default, which is fine for their sizing. Signed-off-by: Scott Buckel * ci: re-trigger after Hetzner runner/buildkit memory re-sizing Signed-off-by: Scott Buckel * feat(earthly): use a self-hosted-specific config on Hetzner runners Adds .earthly/config.selfhosted.yml with values shaped for the Hetzner host (-m 180Gb for buildkit, tls_enabled: false because no certs on the runner); .envrc switches EARTHLY_CONFIG to it when RUNNER_ENVIRONMENT=self-hosted. Devs and github-hosted runners fall through to the default ./config.yml unchanged, so local builds are unaffected. Workflow-level `echo "tls_enabled: false" >> .earthly/config.yml` appends become inert on self-hosted (they edit a file earthly won't read) but remain in place for now — separate cleanup. Signed-off-by: Scott Buckel * fix(ci/local-env-tests): source .envrc before earthly so it picks the right config The composite action's earthly invocations were calling `earthly +...` directly, bypassing .envrc. Without .envrc, EARTHLY_CONFIG isn't set, earthly falls back to ~/.earthly/config.yml (host-level default) instead of .earthly/config.selfhosted.yml. That gives a different settings hash than the other Hetzner jobs (Build node, Test Toolkit), so when local-env runs concurrently it triggers earthly's maybeRestart() → Stop() on the shared earthly-buildkitd container — cancelling any in-flight build, which is exactly what hit Test Toolkit again with "WITH DOCKER images: context canceled". Signed-off-by: Scott Buckel * ci: re-trigger to gather another local-env data point Signed-off-by: Scott Buckel * fix(local-env): tear down stale state before bringing the stack up in CI The +start-local-env-with-indexer-ci earthly target ran `docker compose up --detach` directly, no prior `down --volumes`. On ephemeral runners (github-hosted) that was harmless — the runner VM was recycled between jobs anyway. On a shared persistent host (self-hosted Hetzner) the named volumes (local-env_midnight-node-N-data etc.) survived between runs, and the new run booted validators against stale state from a previous (possibly failed) run. chain-indexer then surfaces the mismatch as "unsupported protocol version" when the genesis/runtime expectations diverge from what's in the persisted db. The non-CI sibling target +start-local-env-with-indexer already does this stop-before-start; the CI variant just hadn't been mirrored. Signed-off-by: Scott Buckel * ci: serialize local-environment-tests via concurrency group The local-env stack binds many fixed ports on 0.0.0.0 (kupo:1442, ogmios:1337, indexer-api:8088, nats:4222, postgres:5432, cardano-node:30000/32000). Multiple jobs landing on the same Hetzner runner host would collide on every one of those, and the second job's stack would fail to come up. Queue rather than run-concurrent until the stack is refactored to use internal-only networking (the mainnet-sync-test pattern). Signed-off-by: Scott Buckel * fix(test/toolkit-multi-dest-e2e): use RUNNER_TEMP so tempdir survives PrivateTmp The self-hosted Hetzner runner systemd unit has `PrivateTmp=true`, which gives the runner agent + step shells their own per-service /tmp namespace. dockerd lives outside that namespace, so when the test does `docker run -v $tempdir:/out` with tempdir under /tmp, the bind-mount silently falls back to an empty dir in dockerd's view of /tmp. The toolkit container then writes its `tx_*.mn` files there, but the host script reads from the runner's /tmp and finds nothing — surfacing as "Generated 0 transaction files" and a downstream "No such file or directory" when the next step tries to read them. Use $RUNNER_TEMP if set (GHA provides it under /var/lib/actions-runner/N/_temp, which is outside PrivateTmp and visible to both the runner and dockerd). Falls back to /tmp on dev laptops where PrivateTmp isn't a factor. Signed-off-by: Scott Buckel * ci: drop hosting-provider verbiage; delete one-off smoke test workflow Sanitize references to the specific cloud provider running our self-hosted runners across: - continuous-integration.yml (job name + comments + concurrency group) - local-environment-tests composite action (comments) - .envrc / Earthfile / .earthly/config.selfhosted.yml / toolkit-multi-dest-e2e.sh Delete .github/workflows/test-hetzner-runner.yml — that workflow was a one-off smoke test for runner reachability during initial provisioning, no longer needed now that the regular CI exercises the pool end-to-end. Signed-off-by: Scott Buckel * ci: route remaining paid x64 workflows to self-hosted pool Final audit: every workflow whose runs-on was a paid larger-runner SKU (ubuntu-latest-{8,16}-core-x64) now targets [self-hosted, tier:*]. The only paid runners remaining are the two arm64 build jobs — GitHub doesn't offer a free arm64 tier and we don't have arm64 self-hosted capacity yet. Moved: - main.yml AMD Build and Publish: 16-core-x64 -> tier:large - continuous-integration-test.yml +test: 16-core-x64 -> tier:large - fork-network.yml: 16-core-x64 -> tier:large - rebuild-chainspec-bot.yml: 8-core-x64 -> tier:medium - rebuild-metadata-bot.yml: 8-core-x64 -> tier:medium Stayed paid (no alternative): - continuous-integration.yml run-arm64 - main.yml ARM Build and Publish Signed-off-by: Scott Buckel * ci: skip free-disk-space.sh on self-hosted in remaining workflows + fix actionlint dup permissions Three workflows still tried to run free-disk-space.sh on self-hosted runners (it refuses with exit 1 there): continuous-integration-test.yml, main.yml AMD Build and Publish, fork-network.yml. Add 'runner.environment != self-hosted' to the existing conditions. The two ubuntu-latest workflows (continuous-integration-checks.yml and nightly-build-check.yml) keep their plain free-disk-space.sh call — they run on free GitHub-hosted runners where the script works as intended. Also: actionlint.yml had a duplicated 'permissions: {}' block (lines 3 and 18) — invalid YAML, blocking actionlint itself from running. Remove the redundant top-level one. Signed-off-by: Scott Buckel * ci: recycle earthly-buildkitd before push + register tier:* labels with actionlint Two unrelated CI failures fixed in one commit. 1) GHCR push fails with 'failed to fetch anonymous token: 403 Forbidden'. The earthly-buildkitd container persists across jobs on the self-hosted host. It bakes registry auth in at startup, so fresh `docker login` calls from subsequent jobs don't propagate. Force-recycle the container before earthly --push so it starts with current host auth. The earthly-cache docker volume survives the restart -- no cache loss. 2) actionlint flags 'unknown runner label' for tier:large/medium/small. Add them to .github/actionlint.yaml under self-hosted-runner.labels so actionlint recognizes them as legitimate self-hosted labels. Signed-off-by: Scott Buckel * ci: copy docker auth from $DOCKER_CONFIG to ~/.docker/config.json before earthly push docker/login-action writes the auth to a per-step temp dir at $DOCKER_CONFIG/config.json. Earthly reads ~/.docker/config.json (HOME-based), so it never sees the auth and falls back to anonymous token at push time, which GHCR rejects with 403. Copy the per-step config to the HOME path before earthly invocation so earthly's buildkit-side push has GHCR creds. Also dump DOCKER_CONFIG / HOME / .auths keys so the next failure (if any) tells us exactly what state earthly is seeing. Signed-off-by: Scott Buckel * ci: revert earthly-buildkitd recycle step — it kills concurrent jobs The recycle step (`docker rm -f earthly-buildkitd`) tears down the single shared buildkit container that ALL concurrent earthly jobs on the self-hosted host depend on. When Run build (in continuous-integration.yml) ran the recycle while +test (in continuous-integration-test.yml) was mid-build inside that same buildkit, +test crashed with: WITH DOCKER RUN failed: ... command [docker exec -i earthly-buildkitd buildctl dial-stdio] has exited with exit status 137 Exit 137 = SIGKILL. Buildkit was killed by the sibling recycle. The auth issue the recycle was added to fix (GHCR anonymous-token 403) is already addressed by the immediately-following 'Make docker auth visible to earthly' step, which copies $DOCKER_CONFIG/config.json to $HOME/.docker/config.json — earthly reads that and forwards auth to buildkit per-invocation via the buildkit gRPC API, no container restart required. Signed-off-by: Scott Buckel * chore: empty commit to re-trigger CI Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Scott Buckel --------- Signed-off-by: Scott Buckel Co-authored-by: Squirrel Co-authored-by: justinfrevert <81839854+justinfrevert@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context)

Prepare a PR to upgrade midnight-node pointing to update stable2512 PC branch #1264

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions