[CI] Improve ccache performance for Windows Python builds (make wheel build directory consistent across runs)#42212
Conversation
|
The job finished in 48 minutes with a 100% cache hit rate. |
|
https://btx.cloud.google.com/invocations/e2c298cf-c5f2-480b-9f33-4d36fa21b230/log For posterity: Great job, @murgatroid99! |
| python -m build --no-isolation --sdist || goto :error | ||
| python -m build --no-isolation --wheel || goto :error |
There was a problem hiding this comment.
We should add a comment explaining why we ended up breaking it into two steps - and explain that we can try reverting it to a single build command if after upgrading ccache/compilers/setuptools the hit rate remains around 100% on the second run.
But we can add a comment in a separate PR; I'm looking forward to getting this merged way too much 😄
… build directory consistent across runs) (grpc#42212) We see ccache cache misses because each python build runs in a different temporary directory, and some header file names in the cache entry's manifest appear to use the full file path, so they fail to match. This change resolves that by making the build directory consistent for each build type. ref: - b/501458064 - Previous change: grpc#42195 Closes grpc#42212 COPYBARA_INTEGRATE_REVIEW=grpc#42212 from murgatroid99:python_windows_ccache_basedir a29d1bb PiperOrigin-RevId: 904139620
|
Ack |
|
Ohh.. did you compare and inspect the tar.gz and .whl contents before and after this PR? While I worked on pyproject.toml migration, i remember I was initially building the sdist and wheels on 2 different lines like in this PR at first and that resulted in some unwanted files (files in third_party IIRC) getting included in the artifacts. So, just to be sure, can you manually compare the contents of the wheel and tar.gz before and after this PR and see if it's not getting affected. |
|
ok nvm, manually verified the artifact contents, and there are no extra files like I suspected. I probably resolved the problem at that time through some other way and misunderstood that building together was the fix. Sorry for the false alarm :) |
… build directory consistent across runs) (grpc#42212) We see ccache cache misses because each python build runs in a different temporary directory, and some header file names in the cache entry's manifest appear to use the full file path, so they fail to match. This change resolves that by making the build directory consistent for each build type. ref: - b/501458064 - Previous change: grpc#42195 Closes grpc#42212 COPYBARA_INTEGRATE_REVIEW=grpc#42212 from murgatroid99:python_windows_ccache_basedir a29d1bb PiperOrigin-RevId: 904139620
We see ccache cache misses because each python build runs in a different temporary directory, and some header file names in the cache entry's manifest appear to use the full file path, so they fail to match. This change resolves that by making the build directory consistent for each build type.
ref: