Pull request overview

This PR introduces an automated CI-to-docs pipeline that measures Erigon sync disk usage in QA workflows, writes the results into a JSON data file, and renders the “Current Disk Usage” column on the hardware requirements docs page from that data.

Changes:

• Add disk-usage measurement + artifact upload steps to the full and minimal “sync from scratch” QA workflows.
• Add a new update-disk-sizes workflow plus a Python script to collect artifacts and update docs/site/src/data/disk-sizes.json.
• Update the hardware requirements MDX to render disk usage from disk-sizes.json instead of hard-coded values.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Overall: nice direction. Removes manual edits, leans on existing QA sync runs, draft-PR human gate before publishing. The earlier Copilot rounds already fixed the bigger ones (base-branch derivation, accumulating history on the auto branch). A few issues remain — flagging the two that will actually misbehave in production first.

Blocking — these will cause runtime failures

1. workflow_run won't fire — workflow file isn't on the default branch

GitHub evaluates workflow_run triggers against the workflow file on the default branch (here, main). I checked and update-disk-sizes.yml does not exist on origin/main, only on this PR branch. After this merges to release/3.4, neither the weekly full sync nor the nightly minimal-node sync will invoke the new workflow. Only the workflow_dispatch manual path will work.

Fix: also land the workflow file on main (either retarget the base, or open a parallel PR to main ahead of merge). The data file + MDX changes should remain release/3.4-only since that's where the site deploys from.

2. Heredoc terminator is indented — auto-PR body will be mangled

At .github/workflows/update-disk-sizes.yml:161 the EOF line has 16 spaces of indentation. After YAML's | block strips the 10-space common indent for the run: block, bash sees EOF (6 leading spaces). <<'EOF' requires the terminator at column 0, so bash never terminates the heredoc and consumes through the closing ) of $(…). I reproduced this locally with the same indentation pattern:

----- BEGIN BODY -----
      ## Automated disk size update
      …
      EOF
      
----- END BODY -----

Two visible consequences in the auto-created PR:

• Every body line gets 6 leading spaces, so GFM renders the whole description as an indented code block.
• A literal EOF line appears in the body.

The gh pr create call still exits 0, so this doesn't hard-fail the job — it just produces a broken PR description.

Fix: dedent both the heredoc body and the EOF terminator to the leftmost shell column. With the surrounding run: | at 8 spaces and content baseline at 10 spaces, the EOF line needs to start at 10 spaces in the YAML so it lands at column 0 in bash. Or sidestep heredocs with a single --body $'…\n…\n…'. Note: <<-EOF only strips leading tabs, not spaces — don't rely on it as a fix.

Notable

3. Update branch name doesn't include the base branch

.github/workflows/update-disk-sizes.yml:79,98 uses docs/auto/disk-sizes regardless of BASE_BRANCH. Once the workflow is on main, scheduled Sunday runs (ref=main) and release-branch push events (ref=release/3.4) will both produce branches with this name. Each force-pushes over the other. The first opened PR's base is pinned at creation; a later run from a different base ends up pushing content rooted at a different base branch. The concurrency.group keying on head_branch only protects against simultaneous runs, not sequential ones.

Suggest: BRANCH="docs/auto/disk-sizes-${BASE_BRANCH//\//-}" so you get docs/auto/disk-sizes-release-3.4 and docs/auto/disk-sizes-main as separate streams.

4. MDX accessor will hard-fail the docs build if a JSON key goes missing

docs/site/docs/get-started/hardware-requirements.mdx:31-35,42-44 uses diskSizes.networks.mainnet.archive.display directly. Today the seed JSON has every key, and the Python updater only mutates existing keys, so this is fine. But if anyone ever drops a chain from disk-sizes.json (e.g. when Polygon support is fully removed), the next docs build dies with an undefined-property error. Cheap defense: a small helper or optional chaining + fallback.

Minor / nitpicks

5. Action version inconsistency

New steps use actions/upload-artifact@v6 vs @v7 already used in the same file (qa-sync-from-scratch-minimal-node.yml uses @v7 at lines 78/100/109/116). The collector uses actions/download-artifact@v4 vs @v8 elsewhere in the repo. Either align or pin via SHA like release.yml does.

6. du -sb quoting + set -euo pipefail

In the new measure step, an unset/missing ERIGON_DATA_DIR would silently produce an empty BYTES var, an empty artifact file, and a Python "could not parse" log on the next run. Cheap insurance: set -euo pipefail at the top of the run block and quote the variable as du -sb "$ERIGON_DATA_DIR".

7. format_bytes boundary precision

docs/site/scripts/update-disk-sizes.py:24-28 — anything ≥ 1 GB but < 1 TB renders as integer-rounded GB. 999.5 GB → "1000 GB" instead of "1.00 TB". Either lower the TB threshold to ~950 GB or use .2f for the GB branch too. Not user-visible until volumes cross the boundary, which mainnet will do within a year.

8. gh pr list --head $BRANCH could match across base branches

If item 3 is fixed by including the base in the branch name, this becomes moot. Otherwise the existence check could match a PR opened against the wrong base.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Re-review (post-fix state)

All 8 issues from my prior review — verified fixed

The added unit tests (test_update_disk_sizes.py) and the _strip_jsx_expr fallback handler for ?? 'val' in generate-llms.py are nice additions; both look correct.

New — Notable

A. #21100's workflow file is not identical to this PR's, despite #21100's body claiming otherwise

Diffing the two update-disk-sizes.yml versions:

< on release/3.4 (this PR)              > on main (#21100)
---
  branches:                               branches:
    - release/3.*                           - release/3.*
-   - main                                  (omitted)
  uses: actions/download-artifact@v7    > uses: actions/download-artifact@v8
                                        > - name: Verify artifacts are present
                                        >   run: |
                                        >     if [ -z "$(ls -A ./artifacts ...)" ]; then
                                        >       echo "::error::No disk-usage artifacts..."
                                        >       exit 1
                                        >     fi
  uses: actions/setup-python@v5         > uses: actions/setup-python@v6

Implications:

• Since workflow_run is evaluated against the file on main, the version that actually fires is #21100's. The release/3.4 copy is dead weight for the trigger path (only reachable via workflow_dispatch from that ref).
• The branches: filter on main excludes main itself. If QA - Sync from scratch ever runs on main (scheduled cron / dispatch), the auto-update won't fire. Probably intentional (docs deploy from release/3.4) but worth confirming.
• Two divergent copies is a maintenance trap. Suggest one of:
  • Drop update-disk-sizes.yml from this PR entirely, keep only the QA-workflow + docs changes here. Land the workflow file via #21100 only.
  • Or sync the two so they're actually identical (v8 download-artifact, verify step, setup-python v6, matching branches filter).
• The "identical to the version in #21030 after all review fixes" claim in #21100's body should be corrected either way.

B. git push --force could clobber human edits to the auto-branch

update-disk-sizes.yml:127 force-pushes the auto-branch on every run. Because concurrency.group serializes runs, the only realistic case where origin advances between fetch and push is a human pushing a manual fix to the open auto-PR (e.g. to nudge a value before merge). The next nightly run will silently wipe that commit.

A plain git push origin "$BRANCH" would be sufficient: each run is exactly one commit ahead of origin/$BRANCH after the fetch+commit, so it'll always be a fast-forward in the happy path, and bail out (rather than clobber) if a human pushed concurrently. Recommend dropping --force.

Minor / nits

C. Manual workflow_dispatch hardcodes BASE_BRANCH=release/3.4

update-disk-sizes.yml:112. Will need a code change when release/3.5 ships. Cheap fix: add a base_branch workflow_dispatch input alongside run_id and prune_mode, default release/3.4.

D. gh pr list --head $BRANCH — variable unquoted

update-disk-sizes.yml:136. ${BASE_BRANCH} is properly quoted on the same line; $BRANCH should be too. Same on line 127 (git push --force origin $BRANCH).

E. set -euo pipefail not added to all multi-line run: blocks

The "Measure disk usage" steps have it (good). The auto-branch-prep, commit-and-push, and create-or-update-PR blocks do not. Worth aligning since you've already adopted the pattern in this PR.

F. Seed JSON's ci_last_updated: "2025-09-01" with all source: "manual"

Cosmetic — the top-level ci_last_updated reads as a CI marker but no CI run has actually occurred at that date. Either drop the field from the seed or rename to last_updated. Not blocking.

G. llms-full.txt will show — until manually regenerated

The PR acknowledges this (Option A → follow-up Option B). Fine.

H. Polygon table left static with September 2025 note

Reasonable given Erigon 3.1.* is the last Polygon-supported series.

Verdict

The blocking items from the first round are all genuinely fixed. The companion-PR strategy works.

Before merge:

1. The #21100-vs-this divergence resolved one way or the other (item A) — the only thing I'd block on.
2. --force dropped from the push (item B) — strong recommend, not block.

C–H are nits that can land in a follow-up. CI is green on the latest commit.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Re-review (post 6e779bc7)

Prior items — all fixed

branches: is kept on release/3.* only (no main) consistently across this PR and #21100 — matches the docs-deploy-from-release-branch model. The heredoc fix from the first round survives — EOF at line 180 lands at column 0 in shell after YAML strips the 10-space common indent. CI green.

Two non-blocking nits (follow-up fine)

1. format_bytes 1000-MB boundary — format_bytes(999_900_000) returns "1000 MB" because round(999.9) == 1000, instead of promoting to GB. Not user-visible for Erigon datadirs (always TB-or-GB scale) but worth a one-liner guard, e.g.:

   mb = round(b / 1_000_000)
   if mb >= 1000:
       return f"{b / 1_000_000_000:.2f} GB"
   return f"{mb} MB"

2. Script exits 0 on no-match — update-disk-sizes.py returns success when no artifact filenames match disk-usage-*-<mode>.txt. The workflow's Verify artifacts are present step catches the empty-directory case before the script runs, so this only fires if artifacts exist but none match the producer/consumer label contract — i.e. a silent divergence. sys.exit(1) on not updated would surface that loudly. Not critical.

Companion PR

#21100 needs to land alongside this for workflow_run to actually fire (still REVIEW_REQUIRED, CI green).

LGTM — approving.