Cache the cibuildwheel C extension build with ccache on Linux and macOS#233
Merged
Conversation
Wrap the C compiler inside the cibuildwheel matrix with ccache so that _helpers_c.c is not rebuilt from scratch for every Python version on every run. hendrikmuhs/ccache-action sets up the cache on the runner; for Linux the manylinux/musllinux container picks ccache up via the existing before-all install plus a bind-mounted CCACHE_DIR and a PATH prepend in CIBW_ENVIRONMENT_LINUX; macOS uses the action's compiler symlinks directly. QEMU jobs are skipped because the host cache contains foreign-arch objects. Windows is intentionally left out: setuptools' MSVCCompiler invokes cl.exe without honoring CC/CXX, so sccache wiring there needs a cl.exe shim and is deferred.
The previous version assumed hendrikmuhs/ccache-action exports
CCACHE_DIR via GITHUB_ENV. It does not (the action configures
ccache itself via `ccache --set-config cache_dir=...`), so the
CIBW_CONTAINER_ENGINE override expanded to `--volume=:/host_ccache`
and docker rejected the empty source path.
Drop the bespoke container-engine override and rely on
cibuildwheel's default `/:/host` host mount; pull the cache dir
from `ccache --get-config cache_dir` and reference it inside the
container at `/host\${cache_dir}`.
Member
|
This looks interesting. How reusable the patch is? I've been having plans of extracting |
Member
Author
|
Probably pretty reusable. ccache is sad state on windows though. Will likely want it to smoke for a few weeks before making it reuable |
5 tasks
Member
Author
|
seems to be working fine on my fork testing. going to merge it. not a big deal to revert |
This was referenced May 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Wrap the C compiler inside the
cibuildwheelmatrix withccacheso that_helpers_c.cis not rebuilt from scratchfor every Python version on every CI run.
hendrikmuhs/ccache-actionprovisions the cache on the runner.On Linux,
cibuildwheelalready bind-mounts the runnerfilesystem at
/hostinside the manylinux/musllinux container,so no custom container-engine override is needed; the
in-container
ccache(installed alongsidelibffi-develinthe existing
before-all) is wired up by prepending theccache compiler symlink dirs to
PATHand pointingCCACHE_DIRat/host\${cache_dir}viaCIBW_ENVIRONMENT_LINUX. The cache dir itself is read fromccache --get-config cache_dirbecause the action does notexport it to
GITHUB_ENV.macOS picks up the action's compiler symlinks directly through
PATH. QEMU jobs are skipped (host cache holds foreign-archobjects).
Windows is intentionally left out for now: setuptools'
MSVCCompilerinvokescl.exedirectly and ignoresCC/CXX, so ansccacheintegration there needs acl.exeshim earlier inPATHand is deferred to a separatepiece of work.
Expected effect on warm cache: the per-Python-version compile of
_helpers_c.cdrops from roughly 25 to 40 seconds to asub-second hit, bringing the slowest Linux and macOS wheel jobs
down by roughly half.
Are there changes in behavior for the user?
No, contributor tooling only.
Related issue number
N/A
Checklist
ruff/ruff format/yamllintrun via pre-commit, andmake doc-spellingis clean against the new fragment)CONTRIBUTORS.txt: N/A (no such file in this repo)CHANGES/folder (CHANGES/233.contrib.rst)Drafted with Claude Code (Opus 4.7); reviewed by @bdraco.
Agent run details (optional, for reviewers)
First push (commit
2ad0e88) crashed all four wheel jobs onLinux with
docker: invalid spec: :/host_ccache: empty section between colons. Root cause:hendrikmuhs/ccache-actiondoesnot export
CCACHE_DIRtoGITHUB_ENV(it configures ccachevia
ccache --set-config cache_dir=...), so the customCIBW_CONTAINER_ENGINEmount expanded to an empty source.Windows and macOS were cancelled by fail-fast.
Follow-up commit (
efe94c3) drops the custom container-engineoverride, relies on
cibuildwheel's default/:/hostmount,and reads the cache dir from
ccache --get-config cache_dir,referencing it at
/host\${cache_dir}inside the container.The "Wire ccache" step shrank by roughly half.
Local validation:
```
$ python3 -c "import yaml; yaml.safe_load(open('.github/workflows/reusable-build-wheel.yml'))"
$ python3 -c "import tomllib; tomllib.load(open('pyproject.toml','rb'))"
$ make doc-spelling
2 pre-existing misspellings only ("subdirectory" in CHANGES/207.contrib.rst, "postfix" in CHANGES.rst); CHANGES/233.contrib.rst is clean.
$ SKIP=actionlint-docker pre-commit run --all-files
all hooks pass (actionlint-docker skipped: Docker daemon not running locally; runs in CI).
```
Container behavior cannot be exercised locally; the live CI run
on this branch is the verification.