Skip to content

bug(release): release workflow bypasses scripts/build-binaries.sh — v0.2.13 and v0.3.0 binaries are broken #986

@Wirasm

Description

@Wirasm

Summary

Two consecutive releases (v0.2.13 and v0.3.0) have shipped broken binaries. The v0.3.0 binary crashes on archon version with Failed to read version: package.json not found (bad installation?). This is a P0 regression — no user can actually use the released binaries.

The root cause is a gap between two build paths: scripts/build-binaries.sh (used for local development) and .github/workflows/release.yml (used for release builds). The two paths diverge at a load-bearing step: writing build-time constants to packages/paths/src/bundled-build.ts before invoking bun build --compile.

Timeline

  1. PRs fix(paths): skip pino-pretty transport in compiled binaries #962 and fix(workflows): detect compiled binaries via execPath fallback #963 (from @leex279): fixed the same class of bug using runtime detection (import.meta.dir prefix + process.execPath basename heuristics). This worked regardless of how the binary was built — detection happened at process startup.
  2. Issue fix(build): use build-time constants for binary detection and pretty stream logger (replaces #962/#963) #979 / PR fix(build): use build-time constants for binary detection and pretty stream logger #982: replaced the runtime detection with build-time constants (BUNDLED_IS_BINARY, BUNDLED_VERSION, BUNDLED_GIT_COMMIT) written into packages/paths/src/bundled-build.ts by scripts/build-binaries.sh before compilation. This is architecturally cleaner (one source of truth, no runtime fragility, matches the existing BUNDLED_VERSION pattern) — BUT it requires the build script to actually run.
  3. The release workflow does NOT run scripts/build-binaries.sh. It runs bun build --compile inline:
    if [[ "${{ matrix.target }}" == *windows* ]]; then
      bun build --compile --minify --target=${{ matrix.target }} --outfile=dist/${{ matrix.binary }} packages/cli/src/cli.ts
    else
      bun build --compile --minify --bytecode --target=${{ matrix.target }} --outfile=dist/${{ matrix.binary }} packages/cli/src/cli.ts
    fi
  4. Result: bundled-build.ts is never rewritten before compile. Bun bakes the dev defaults (BUNDLED_IS_BINARY = false, BUNDLED_VERSION = 'dev', BUNDLED_GIT_COMMIT = 'unknown') into the released binary. isBinaryBuild() returns false at runtime. The version command falls into getDevVersion() which tries to read package.json from the /$bunfs/ virtual filesystem. Crash.

Why this wasn't caught

I verified #982 locally by running bash scripts/build-binaries.sh and testing the resulting binary. It worked because the script did rewrite bundled-build.ts. I did not verify .github/workflows/release.yml uses the same code path. It doesn't.

The lesson: "local smoke test passed" does not mean "release workflow works". These are two independent build paths and must both be tested before shipping a binary-touching change.

Proposed fix — the proper solution

The principle: the build-time constant rewrite must live in exactly one place, and both local dev and CI must call it via the same entry point. Inline duplication in release.yml works for v0.3.1 but leaves the drift risk in place. The proper fix is to refactor scripts/build-binaries.sh to be the canonical build entry point and have CI call it.

Step 1 — Refactor scripts/build-binaries.sh to support single-target mode

Current script builds all 4 targets in a loop. CI runs one target per matrix job. Script needs to accept env-var parameters so CI can invoke it with one target.

New behavior:

  • If TARGET and OUTFILE are both set → build only that one target (CI mode)
  • If neither is set → build all targets (local dev mode, backwards-compatible)
  • If only one is set → error out with clear message

Additional env vars to accept:

  • VERSION (existing, from package.json grep by default)
  • GIT_COMMIT (existing, from git rev-parse --short HEAD by default)

Script logic:

#!/usr/bin/env bash
set -euo pipefail

VERSION="${VERSION:-$(grep '"version"' package.json | head -1 | cut -d'"' -f4)}"
GIT_COMMIT="${GIT_COMMIT:-$(git rev-parse --short HEAD 2>/dev/null || echo 'unknown')}"
OUTFILE="${OUTFILE:-}"
TARGET="${TARGET:-}"

echo "Building Archon CLI v${VERSION} (commit: ${GIT_COMMIT})"

# Restore bundled-build.ts on exit (success or failure)
BUNDLED_BUILD_FILE="packages/paths/src/bundled-build.ts"
trap 'git checkout -- "$BUNDLED_BUILD_FILE" 2>/dev/null || true' EXIT

# Rewrite build-time constants
echo "Updating bundled build constants (is_binary=true, version=${VERSION})..."
cat > "$BUNDLED_BUILD_FILE" << EOF
/**
 * Build-time constants embedded into compiled binaries.
 *
 * This file is rewritten by scripts/build-binaries.sh before bun build --compile
 * and restored afterwards via an EXIT trap. Do NOT edit these values by hand
 * outside the build script — the dev defaults live in the committed copy.
 */

export const BUNDLED_IS_BINARY = true;
export const BUNDLED_VERSION = '${VERSION}';
export const BUNDLED_GIT_COMMIT = '${GIT_COMMIT}';
EOF

# Determine which targets to build
if [ -n "$TARGET" ] && [ -n "$OUTFILE" ]; then
  # Single-target mode (CI)
  TARGETS=("$TARGET:$OUTFILE")
elif [ -n "$TARGET" ] || [ -n "$OUTFILE" ]; then
  echo "ERROR: TARGET and OUTFILE must be set together (CI mode) or both unset (local mode)" >&2
  exit 1
else
  # Multi-target mode (local dev)
  DIST_DIR="dist/binaries"
  mkdir -p "$DIST_DIR"
  TARGETS=(
    "bun-darwin-arm64:${DIST_DIR}/archon-darwin-arm64"
    "bun-darwin-x64:${DIST_DIR}/archon-darwin-x64"
    "bun-linux-x64:${DIST_DIR}/archon-linux-x64"
    "bun-linux-arm64:${DIST_DIR}/archon-linux-arm64"
  )
fi

# Minimum expected size — Bun compiled binaries are typically 50MB+
MIN_BINARY_SIZE=1000000

for target_pair in "${TARGETS[@]}"; do
  IFS=':' read -r target outfile <<< "$target_pair"
  echo "Building $target$outfile"

  # Bytecode compile is supported on Linux and macOS but not Windows
  BYTECODE_FLAG=""
  if [[ "$target" != *windows* ]]; then
    BYTECODE_FLAG="--bytecode"
  fi

  # Always --minify for release parity
  bun build \
    --compile \
    --minify \
    $BYTECODE_FLAG \
    --target="$target" \
    --outfile="$outfile" \
    packages/cli/src/cli.ts

  if [ ! -f "$outfile" ]; then
    echo "ERROR: Build failed — $outfile not created" >&2
    exit 1
  fi

  size=$(stat -f%z "$outfile" 2>/dev/null || stat --printf="%s" "$outfile")
  if [ "$size" -lt "$MIN_BINARY_SIZE" ]; then
    echo "ERROR: Binary suspiciously small ($size bytes): $outfile" >&2
    exit 1
  fi

  echo "$outfile ($size bytes)"
done

echo "Build complete."

Step 2 — Update .github/workflows/release.yml to call the script

Replace the inline bun build --compile step with:

- name: Build binary
  env:
    VERSION: ${{ github.ref_name }}
    GIT_COMMIT: ${{ github.sha }}
    TARGET: ${{ matrix.target }}
    OUTFILE: dist/${{ matrix.binary }}
  run: |
    # Strip 'v' prefix from tag (e.g. v0.3.1 → 0.3.1)
    VERSION="${VERSION#v}"
    # Short commit (first 8 chars of SHA)
    GIT_COMMIT="${GIT_COMMIT::8}"
    mkdir -p dist
    VERSION="$VERSION" GIT_COMMIT="$GIT_COMMIT" TARGET="$TARGET" OUTFILE="$OUTFILE" bash scripts/build-binaries.sh

Single step, delegates everything to the script. The env block keeps it readable.

Step 3 — Add a post-build smoke test inside the release workflow

This would have caught both v0.2.13 and v0.3.0 before publishing. Add a step after the build that runs archon version on the freshly-built Linux x64 binary (which can execute on the Linux CI runner):

- name: Smoke-test built binary
  if: matrix.target == 'bun-linux-x64' && runner.os == 'Linux'
  run: |
    chmod +x dist/${{ matrix.binary }}
    VERSION_OUTPUT=$(./dist/${{ matrix.binary }} version 2>&1)
    echo "$VERSION_OUTPUT"

    # Must not error with "Failed to read version" or similar
    if echo "$VERSION_OUTPUT" | grep -qE "Failed to read version|package\.json not found|bad installation"; then
      echo "::error::Binary is broken — version command cannot read embedded version"
      echo "::error::This means BUNDLED_IS_BINARY was not set to true at build time."
      exit 1
    fi

    # Must report 'Build: binary', not 'Build: source'
    if ! echo "$VERSION_OUTPUT" | grep -q "Build: binary"; then
      echo "::error::Binary reports wrong build type"
      echo "::error::Expected 'Build: binary' in version output"
      exit 1
    fi

    # Must report a non-default version
    if echo "$VERSION_OUTPUT" | grep -q "v${{ github.ref_name }}"; then
      echo "::notice::Binary correctly reports version"
    else
      echo "::error::Binary does not report the tag version"
      exit 1
    fi

Limitations:

  • Only runs for bun-linux-x64 because the CI runner is Linux x64 — it can't execute Mac or Windows binaries. But if one target is broken due to the build-time-constants gap, all targets typically break the same way. One smoke test catches the class of bug.
  • Adds ~5 seconds to the workflow. Worth it to prevent broken releases.

Step 4 — Update the test-release skill to document the build env vars

The skill file at .claude/skills/test-release/SKILL.md currently describes testing a released binary. Add a "Local build" section that explains how to reproduce the CI build locally for pre-release QA:

## Local build for pre-release QA

To build a binary locally with the exact same flags and constants as CI uses:

\`\`\`bash
# Multi-target mode (builds all 4 platforms)
VERSION=0.3.1 GIT_COMMIT=abc12345 bash scripts/build-binaries.sh

# Single-target mode (matches CI matrix job)
VERSION=0.3.1 \
GIT_COMMIT=abc12345 \
TARGET=bun-darwin-arm64 \
OUTFILE=dist/test-archon-darwin-arm64 \
bash scripts/build-binaries.sh

# Verify the binary
./dist/binaries/archon-darwin-arm64 version
# Expected: Archon CLI v0.3.1, Build: binary, Git commit: abc12345
\`\`\`

Use this **before tagging a release** to catch build-time-constant issues
locally. Then run the full release workflow in CI with confidence that
the same code path has already been exercised.

Files to change

File Change
scripts/build-binaries.sh Refactor for single-target mode via TARGET/OUTFILE env vars; add Windows bytecode skip; add --minify by default; verify EXIT trap still fires
.github/workflows/release.yml Replace inline bun build --compile step with bash scripts/build-binaries.sh invocation; pass env vars from matrix
.github/workflows/release.yml (new step) Add post-build smoke test for bun-linux-x64 target
.claude/skills/test-release/SKILL.md Add "Local build for pre-release QA" section documenting the env vars

Validation steps (to run during implementation)

Local verification (before committing):

  1. Backwards compatibility: run bash scripts/build-binaries.sh with no args, confirm it still builds all 4 targets in dist/binaries/ identically to current behavior.
  2. Single-target mode: run VERSION=0.3.1-test GIT_COMMIT=test1234 TARGET=bun-darwin-arm64 OUTFILE=/tmp/test-single-target bash scripts/build-binaries.sh. Confirm the binary exists and runs.
  3. Build-time constants: run /tmp/test-single-target version and verify it reports v0.3.1-test, Build: binary, Git commit: test1234.
  4. EXIT trap restore: confirm git status packages/paths/src/bundled-build.ts shows the file unchanged (trap restored dev defaults).
  5. Error handling: run with only TARGET set (not OUTFILE) and confirm the script errors with a clear message.

CI verification (after merging to dev):

  1. Open a PR to dev with the changes. The existing docker-build and test jobs should pass unchanged (they don't touch the binary build).
  2. After merging, tag a test release (v0.3.1-rc1) on a test branch or use workflow_dispatch to trigger the release workflow manually.
  3. Confirm the new smoke-test step catches or passes the build correctly.
  4. Confirm gh release view v0.3.1-rc1 shows all 5 binaries + checksums.
  5. Download archon-darwin-arm64 from the release and verify ./archon-darwin-arm64 version reports the correct version.

Post-release verification (after v0.3.1 ships):

  1. Run /test-release curl-mac 0.3.1 — should pass all smoke tests.
  2. Run /test-release brew 0.3.1will still fail until the coleam00/homebrew-archon tap formula is synced (separate issue, see "Related" below).

Related issues

Priority

P0 — two consecutive releases have shipped broken binaries. The next release must ship working binaries. This is the blocking fix.

Credit

This regression is on me. When I argued for #982 over #962/#963, I said the build-time constant pattern was principled and matched the existing BUNDLED_VERSION precedent. I verified it locally by running bash scripts/build-binaries.sh directly and testing the output. I did not check whether .github/workflows/release.yml called the same script. It doesn't — it calls bun build --compile inline, skipping the constant rewrite entirely.

Thomas's runtime detection approach in #962/#963 would have worked against the current release workflow because it didn't depend on the build script running. My refactor did depend on it, and I shipped it without verifying the dependency. The v0.2.13 binary was broken before (same class of bug, different cause — pino-pretty transport), and the v0.3.0 binary is broken now because my fix assumed a code path that doesn't exist in CI.

The proper solution fixes the root design flaw (two drifted build paths) rather than patching the workflow inline. That way the next contributor who adds a build-time constant (e.g., BUNDLED_POSTHOG_KEY for telemetry #980) only has to modify one file and can be confident both local and CI builds pick it up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical - Do first, blocking or urgentbugSomething is broken

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions