Add cross-process AOT autograd cache hit test#177397
Add cross-process AOT autograd cache hit test#177397frgossen wants to merge 5 commits intogh/frgossen/9/basefrom
Conversation
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/177397
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 7a1e4f8 with merge base 6a461fe ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx kadeng chauhang amjames Lucaskabela jataylo [ghstack-poisoned]
|
Adding @zhxchen17 as well since he does quite a bit of work on AOT Autograd caching |
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx kadeng chauhang amjames Lucaskabela jataylo [ghstack-poisoned]
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx kadeng chauhang amjames Lucaskabela jataylo [ghstack-poisoned]
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx kadeng chauhang amjames Lucaskabela jataylo [ghstack-poisoned]
This PR needs a
|
|
Starting merge as part of PR stack under #177428 |
|
Starting merge as part of PR stack under #177428 |
2 similar comments
|
Starting merge as part of PR stack under #177428 |
|
Starting merge as part of PR stack under #177428 |
|
@pytorchbot merge -i |
Merge startedYour change will be merged while ignoring the following 1 checks: pull / linux-jammy-cpu-py3.10-gcc11-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge) Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
pre_grad_custom_pass was the only custom pass config without UUID-based cache key integration. It was excluded from config serialization but not handled specially via UUID extraction, so its effect was only captured indirectly through the resulting FX graph. This meant two different passes producing the same graph could incorrectly share a cache entry. Align pre_grad_custom_pass with post-grad and joint passes: change its type to CustomGraphPassType, add it to _cache_config_ignore_prefix (so the UUID is extracted explicitly), include it in FxGraphHashDetails, and validate it in _check_can_cache. Pull Request resolved: #177403 Approved by: https://github.com/aorenste, https://github.com/zou3519 ghstack dependencies: #177397
Pull Request resolved: #177428 Approved by: https://github.com/mlazos, https://github.com/zou3519 ghstack dependencies: #177397, #177403
…77429) Add a pre_grad_pass_timing config ("early", "late", or "default") that controls when pre-grad passes run relative to the AOT autograd cache lookup. - "early": passes run before cache lookup, so they execute on every compile (including cache hits) and the cache key reflects the already-transformed graph. - "late": passes run after cache lookup (only on cache miss); requires custom passes to provide a UUID for the cache key. - "default": automatically resolves to "late" when possible (no custom pass, or a custom pass with a UUID), and falls back to "early" when the custom pass has no UUID. Explicitly setting "late" with a UUID-less custom pass now raises a RuntimeError instead of silently bypassing the cache. The existing test_pre_grad_passes_called_on_cache_miss_only test is renamed and pinned to "late" timing, and new tests cover early timing, both default timing branches, and the error case. Pull Request resolved: #177429 Approved by: https://github.com/aorenste, https://github.com/zou3519 ghstack dependencies: #177397, #177403, #177428
Test that the AOT autograd cache persists across separate processes by running two subprocesses with a shared cache directory and verifying the second one gets a cache hit. Pull Request resolved: pytorch#177397 Approved by: https://github.com/aorenste, https://github.com/zou3519
…177403) pre_grad_custom_pass was the only custom pass config without UUID-based cache key integration. It was excluded from config serialization but not handled specially via UUID extraction, so its effect was only captured indirectly through the resulting FX graph. This meant two different passes producing the same graph could incorrectly share a cache entry. Align pre_grad_custom_pass with post-grad and joint passes: change its type to CustomGraphPassType, add it to _cache_config_ignore_prefix (so the UUID is extracted explicitly), include it in FxGraphHashDetails, and validate it in _check_can_cache. Pull Request resolved: pytorch#177403 Approved by: https://github.com/aorenste, https://github.com/zou3519 ghstack dependencies: pytorch#177397
Pull Request resolved: pytorch#177428 Approved by: https://github.com/mlazos, https://github.com/zou3519 ghstack dependencies: pytorch#177397, pytorch#177403
…torch#177429) Add a pre_grad_pass_timing config ("early", "late", or "default") that controls when pre-grad passes run relative to the AOT autograd cache lookup. - "early": passes run before cache lookup, so they execute on every compile (including cache hits) and the cache key reflects the already-transformed graph. - "late": passes run after cache lookup (only on cache miss); requires custom passes to provide a UUID for the cache key. - "default": automatically resolves to "late" when possible (no custom pass, or a custom pass with a UUID), and falls back to "early" when the custom pass has no UUID. Explicitly setting "late" with a UUID-less custom pass now raises a RuntimeError instead of silently bypassing the cache. The existing test_pre_grad_passes_called_on_cache_miss_only test is renamed and pinned to "late" timing, and new tests cover early timing, both default timing branches, and the error case. Pull Request resolved: pytorch#177429 Approved by: https://github.com/aorenste, https://github.com/zou3519 ghstack dependencies: pytorch#177397, pytorch#177403, pytorch#177428
Stack from ghstack (oldest at bottom):
Test that the AOT autograd cache persists across separate processes by
running two subprocesses with a shared cache directory and verifying the
second one gets a cache hit.
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @Lucaskabela @jataylo