Move backend-specific c10d files into per-backend subfolders (#187083)#187083
Open
d4l3k wants to merge 1 commit into
Open
Move backend-specific c10d files into per-backend subfolders (#187083)#187083d4l3k wants to merge 1 commit into
d4l3k wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/187083
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 73 PendingAs of commit dd9f5b7 with merge base 083e261 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@d4l3k has exported this pull request. If you are a Meta employee, you can view the originating Diff in D108332288. |
d4l3k
added a commit
to d4l3k/pytorch
that referenced
this pull request
Jun 11, 2026
…#187083) Summary: Pull Request resolved: pytorch#187083 Reorganizes `torch/csrc/distributed/c10d` by moving non-public, backend-specific implementation files and the TCPStore backend files into per-backend subfolders, while leaving the public-facing classes at the top level (the `ProcessGroupGloo`/`NCCL`/`MPI`/`UCC` backends and the `Store`/`TCPStore`/`FileStore`/`HashStore`/`PrefixStore` classes all stay put). The moves are: `store/` gets `TCPStoreBackend.{cpp,hpp}` and `TCPStoreLibUvBackend.cpp`; `gloo/` gets `ProcessGroupGlooCuda.cpp`, `ProcessGroupGlooDetail.hpp`, and `GlooDeviceFactory.{cpp,hpp}`; `ucc/` gets `UCCTracing.{cpp,hpp}` and `UCCUtils.{cpp,hpp}`; `nccl/` gets `NCCLXStub.hpp`. `NCCLUtils.{cpp,hpp}` was deliberately kept at the top level even though it is backend-specific: it is included by several call sites outside `caffe2` (in `gen_ai`, `ads_mkl`, and `fbgemm_gpu`), so relocating it would be a wider, riskier change better done on its own. As a result the new `nccl/` folder currently holds only `NCCLXStub.hpp`. All include sites were updated, covering both the canonical `torch/csrc/distributed/c10d/...` include form and the legacy short `c10d/...` form (used by `fb/GlooDeviceFactory.cpp`). Build wiring was updated in `build_variables.bzl` -- the canonical source list consumed by CMake (via `append_filelist` in `cmake/Codegen.cmake`), OSS Bazel, and OSS Buck -- and in the internal `fb/fbcode/target_definitions.bzl` for `ProcessGroupGlooCuda.cpp`. Headers are picked up by recursive globs, so no header-list edits were needed. This is a pure file move: contents are unchanged apart from the relocated `#include` paths, so correctness is established by a clean build rather than by behavioral tests. Authored with the assistance of an AI coding assistant (Claude Code). Test Plan: Confirmed no references to the old paths remain anywhere in `fbcode`, then ran the fbcode lint and build tooling: ``` arc f arc lint arc lint --take AUTODEPS --apply-patches buck2 build fbcode//caffe2:_libtorch fbcode//caffe2:_libtorch_cuda ``` `arc f` and `arc lint` reported no issues; AUTODEPS produced no dependency changes (the moves stayed within existing Buck targets); both the CPU (`_libtorch`) and CUDA (`_libtorch_cuda`) libraries built successfully (exit 0). Differential Revision: D108332288
d4l3k
added a commit
to d4l3k/pytorch
that referenced
this pull request
Jun 11, 2026
…#187083) Summary: Pull Request resolved: pytorch#187083 Reorganizes `torch/csrc/distributed/c10d` by moving non-public, backend-specific implementation files and the TCPStore backend files into per-backend subfolders, while leaving the public-facing classes at the top level (the `ProcessGroupGloo`/`NCCL`/`MPI`/`UCC` backends and the `Store`/`TCPStore`/`FileStore`/`HashStore`/`PrefixStore` classes all stay put). The moves are: `store/` gets `TCPStoreBackend.{cpp,hpp}` and `TCPStoreLibUvBackend.cpp`; `gloo/` gets `ProcessGroupGlooCuda.cpp`, `ProcessGroupGlooDetail.hpp`, and `GlooDeviceFactory.{cpp,hpp}`; `ucc/` gets `UCCTracing.{cpp,hpp}` and `UCCUtils.{cpp,hpp}`; `nccl/` gets `NCCLXStub.hpp`. `NCCLUtils.{cpp,hpp}` was deliberately kept at the top level even though it is backend-specific: it is included by several call sites outside `caffe2` (in `gen_ai`, `ads_mkl`, and `fbgemm_gpu`), so relocating it would be a wider, riskier change better done on its own. As a result the new `nccl/` folder currently holds only `NCCLXStub.hpp`. All include sites were updated, covering both the canonical `torch/csrc/distributed/c10d/...` include form and the legacy short `c10d/...` form (used by `fb/GlooDeviceFactory.cpp`). Build wiring was updated in `build_variables.bzl` -- the canonical source list consumed by CMake (via `append_filelist` in `cmake/Codegen.cmake`), OSS Bazel, and OSS Buck -- and in the internal `fb/fbcode/target_definitions.bzl` for `ProcessGroupGlooCuda.cpp`. Headers are picked up by recursive globs, so no header-list edits were needed. This is a pure file move: contents are unchanged apart from the relocated `#include` paths, so correctness is established by a clean build rather than by behavioral tests. Authored with the assistance of an AI coding assistant (Claude Code). Test Plan: Confirmed no references to the old paths remain anywhere in `fbcode`, then ran the fbcode lint and build tooling: ``` arc f arc lint arc lint --take AUTODEPS --apply-patches buck2 build fbcode//caffe2:_libtorch fbcode//caffe2:_libtorch_cuda ``` `arc f` and `arc lint` reported no issues; AUTODEPS produced no dependency changes (the moves stayed within existing Buck targets); both the CPU (`_libtorch`) and CUDA (`_libtorch_cuda`) libraries built successfully (exit 0). Differential Revision: D108332288
…#187083) Summary: Pull Request resolved: pytorch#187083 Reorganizes `torch/csrc/distributed/c10d` by moving non-public, backend-specific implementation files and the TCPStore backend files into per-backend subfolders, while leaving the public-facing classes at the top level (the `ProcessGroupGloo`/`NCCL`/`MPI`/`UCC` backends and the `Store`/`TCPStore`/`FileStore`/`HashStore`/`PrefixStore` classes all stay put). The moves are: `store/` gets `TCPStoreBackend.{cpp,hpp}` and `TCPStoreLibUvBackend.cpp`; `gloo/` gets `ProcessGroupGlooCuda.cpp`, `ProcessGroupGlooDetail.hpp`, and `GlooDeviceFactory.{cpp,hpp}`; `ucc/` gets `UCCTracing.{cpp,hpp}` and `UCCUtils.{cpp,hpp}`; `nccl/` gets `NCCLXStub.hpp`. `NCCLUtils.{cpp,hpp}` was deliberately kept at the top level even though it is backend-specific: it is included by several call sites outside `caffe2` (in `gen_ai`, `ads_mkl`, and `fbgemm_gpu`), so relocating it would be a wider, riskier change better done on its own. As a result the new `nccl/` folder currently holds only `NCCLXStub.hpp`. All include sites were updated, covering both the canonical `torch/csrc/distributed/c10d/...` include form and the legacy short `c10d/...` form (used by `fb/GlooDeviceFactory.cpp`). Build wiring was updated in `build_variables.bzl` -- the canonical source list consumed by CMake (via `append_filelist` in `cmake/Codegen.cmake`), OSS Bazel, and OSS Buck -- and in the internal `fb/fbcode/target_definitions.bzl` for `ProcessGroupGlooCuda.cpp`. Headers are picked up by recursive globs, so no header-list edits were needed. This is a pure file move: contents are unchanged apart from the relocated `#include` paths, so correctness is established by a clean build rather than by behavioral tests. Authored with the assistance of an AI coding assistant (Claude Code). Test Plan: Confirmed no references to the old paths remain anywhere in `fbcode`, then ran the fbcode lint and build tooling: ``` arc f arc lint arc lint --take AUTODEPS --apply-patches buck2 build fbcode//caffe2:_libtorch fbcode//caffe2:_libtorch_cuda ``` `arc f` and `arc lint` reported no issues; AUTODEPS produced no dependency changes (the moves stayed within existing Buck targets); both the CPU (`_libtorch`) and CUDA (`_libtorch_cuda`) libraries built successfully (exit 0). Reviewed By: kapilsh Differential Revision: D108332288
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Reorganizes
torch/csrc/distributed/c10dby moving non-public, backend-specific implementation files and the TCPStore backend files into per-backend subfolders, while leaving the public-facing classes at the top level (theProcessGroupGloo/NCCL/MPI/UCCbackends and theStore/TCPStore/FileStore/HashStore/PrefixStoreclasses all stay put).The moves are:
store/getsTCPStoreBackend.{cpp,hpp}andTCPStoreLibUvBackend.cpp;gloo/getsProcessGroupGlooCuda.cpp,ProcessGroupGlooDetail.hpp, andGlooDeviceFactory.{cpp,hpp};ucc/getsUCCTracing.{cpp,hpp}andUCCUtils.{cpp,hpp};nccl/getsNCCLXStub.hpp.NCCLUtils.{cpp,hpp}was deliberately kept at the top level even though it is backend-specific: it is included by several call sites outsidecaffe2(ingen_ai,ads_mkl, andfbgemm_gpu), so relocating it would be a wider, riskier change better done on its own. As a result the newnccl/folder currently holds onlyNCCLXStub.hpp.All include sites were updated, covering both the canonical
torch/csrc/distributed/c10d/...include form and the legacy shortc10d/...form (used byfb/GlooDeviceFactory.cpp). Build wiring was updated inbuild_variables.bzl-- the canonical source list consumed by CMake (viaappend_filelistincmake/Codegen.cmake), OSS Bazel, and OSS Buck -- and in the internalfb/fbcode/target_definitions.bzlforProcessGroupGlooCuda.cpp. Headers are picked up by recursive globs, so no header-list edits were needed.This is a pure file move: contents are unchanged apart from the relocated
#includepaths, so correctness is established by a clean build rather than by behavioral tests.Authored with the assistance of an AI coding assistant (Claude Code).
Test Plan:
Confirmed no references to the old paths remain anywhere in
fbcode, then ran the fbcode lint and build tooling:arc fandarc lintreported no issues; AUTODEPS produced no dependency changes (the moves stayed within existing Buck targets); both the CPU (_libtorch) and CUDA (_libtorch_cuda) libraries built successfully (exit 0).Reviewed By: kapilsh
Differential Revision: D108332288