🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119571

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit c4499e7 with merge base a0207c8 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@huydhn Can you help us trigger the rest of the pipelines in the CI.

@lezcano @nikitaved @IvanYashchuk
Can you please review this PR.

The linalg changes lgtm. I don't think I'm the best reviewer for the rest of the PR.

@jerryzh168 @salilsdesai @kimishpatel @digantdesai @jianyuh
Hello All, can you please review this PR.

Hello @albanD @seemethere . This contribution PR size is little more than 2000 and we have all pipelines passing except the - Lint / pr-sanity-checks (pull_request).

It's good for this contribution to go in as one rather than multiple patches as it is easy for reviewers to understand and approve and also to make sure it doesn't break the CI pipelines as this vec backend requires changes in multiple files.

Can you please help us bypass this one check for this pull request.

@jerryzh168 @salilsdesai @kimishpatel @digantdesai @jianyuh Hello All, can you please review this PR.

I definitely can. Unfortunately it will go very slow due to other things on my plate.

This is an important contribution FWIW

this seems to be low risk, @kimishpatel should we just stamp? if it doesn't affect code size etc.

@mcr229 can you import this and build size bot to check build size impact of this. I havent looked enough to be assured of that.

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Hi @maajidkhann , curious to know how much perf improvement you observed across those accelerated ops, any data you can share? thank you.

This is awesome. Thanks for this PR. I will also try to review this.

Successfully rebased vec_backend_for_sve onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout vec_backend_for_sve && git pull --rebase)

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Tried to rebase and push PR #119571, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

Merge failed

Reason: Approvers from one of the following sets are needed:

• superuser (pytorch/metamates)
• Core Reviewers (mruberry, lezcano, Skylion007, ngimel, peterbell10, ...)
• Core Maintainers (soumith, gchanan, ezyang, dzhulgakov, malfet, ...)

Looks like there's just 1 unrelated failure for this PR.
@maajidkhann this PR still fails all aarch64 tests, aren't it? Before merging change that essentially alters behavior for every op, I thin it's important to see clear signal, isn't it?

Hi @malfet
All aarch64 tests CI are passed now.

New failure occurs due to linker issue on macos SIMULATOR: periodic / ios-build-test / build (default, 1, 1, macos-14-xlarge, SIMULATOR, arm64, 1, 0, 1) (gh)

Undefined symbols for architecture arm64:
  "torch::jit::mobile::Module::find_method(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const", referenced from:
      -[TestAppTests runModel:] in TestLiteInterpreter.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

This failure is produce with every PR with ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR

Also, this PR needs your approval (Core Maintainers) before merging.

@pytorchbot merge -f "Lint + ARM CI + MacOS builds are green"

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

@maajidkhann I'm interested in this PR, but I would like to know if there would be any benefits based on this PR when using CUDA as the backend?
I would greatly appreciate if you could provide me with some information.

@maajidkhann I'm interested in this PR, but I would like to know if there would be any benefits based on this PR when using CUDA as the backend? I would greatly appreciate if you could provide me with some information.

Since the code in this PR is extending SVE support for ARM CPUs, it would primarily benefit CPU workloads running on ARM hardware and has SVE hardware support. If the user is leveraging CUDA as the backend for GPU acceleration, the PR would likely have no direct impact.

CUDA workloads are accelerated by NVIDIA GPUs, and optimizations specific to ARM CPU vector extensions (like SVE) would not affect these GPU workloads.

Possible Indirect Benefits:
While the PR itself is unlikely to affect CUDA directly, there could be indirect benefits. For example, if the user is running hybrid workloads that offload some tasks to the GPU while others remain on the CPU, the ARM CPU-side improvements (like SVE) could help with tasks that remain on the CPU, potentially improving the overall system throughput. However, this would depend on the specific workload and the balance between CPU and GPU operations.

@maajidkhann I'm interested in this PR, but I would like to know if there would be any benefits based on this PR when using CUDA as the backend? I would greatly appreciate if you could provide me with some information.我对此 PR 感兴趣，但我想知道使用 CUDA 作为后端时基于此 PR 是否有任何好处？如果您能为我提供一些信息，我将不胜感激。

Since the code in this PR is extending SVE support for ARM CPUs, it would primarily benefit CPU workloads running on ARM hardware and has SVE hardware support. If the user is leveraging CUDA as the backend for GPU acceleration, the PR would likely have no direct impact.由于此 PR 中的代码正在扩展对 ARM CPU 的 SVE 支持，因此它主要有利于在 ARM 硬件上运行并具有 SVE 硬件支持的 CPU 工作负载。如果用户利用 CUDA 作为 GPU 加速的后端，PR 可能不会产生直接影响。

CUDA workloads are accelerated by NVIDIA GPUs, and optimizations specific to ARM CPU vector extensions (like SVE) would not affect these GPU workloads.CUDA 工作负载由 NVIDIA GPU 加速，并且特定于 ARM CPU 矢量扩展（如 SVE）的优化不会影响这些 GPU 工作负载。

Possible Indirect Benefits:可能的间接好处： While the PR itself is unlikely to affect CUDA directly, there could be indirect benefits. For example, if the user is running hybrid workloads that offload some tasks to the GPU while others remain on the CPU, the ARM CPU-side improvements (like SVE) could help with tasks that remain on the CPU, potentially improving the overall system throughput. However, this would depend on the specific workload and the balance between CPU and GPU operations.虽然 PR 本身不太可能直接影响 CUDA，但可能会带来间接的好处。例如，如果用户运行混合工作负载，将某些任务卸载到 GPU，而其他任务保留在 CPU 上，则 ARM CPU 端改进（如 SVE）可以帮助处理保留在 CPU 上的任务，从而有可能提高整体系统吞吐量。但是，这取决于具体的工作负载以及 CPU 和 GPU 操作之间的平衡。

Thank you very much, this has addressed most of my questions.

However, I still have one doubt: According to the blog lets-talk-about-the-pytorch-dispatcher and native_functions.yaml, not all operators have specifically registered implementations for CUDA, means some operators use a generic kernel implementation (CPU) across all devices. So could these operators potentially benefit from this change as well, in spite of I use CUDA as backend ?

Unfortunately, I've tried reading PyTorch's documentation and source code, but haven't reached a very clear conclusion on this point.

@maajidkhann I'm interested in this PR, but I would like to know if there would be any benefits based on this PR when using CUDA as the backend? I would greatly appreciate if you could provide me with some information.我对此 PR 感兴趣，但我想知道使用 CUDA 作为后端时基于此 PR 是否有任何好处？如果您能为我提供一些信息，我将不胜感激。

Since the code in this PR is extending SVE support for ARM CPUs, it would primarily benefit CPU workloads running on ARM hardware and has SVE hardware support. If the user is leveraging CUDA as the backend for GPU acceleration, the PR would likely have no direct impact.由于此 PR 中的代码正在扩展对 ARM CPU 的 SVE 支持，因此它主要有利于在 ARM 硬件上运行并具有 SVE 硬件支持的 CPU 工作负载。如果用户利用 CUDA 作为 GPU 加速的后端，PR 可能不会产生直接影响。
CUDA workloads are accelerated by NVIDIA GPUs, and optimizations specific to ARM CPU vector extensions (like SVE) would not affect these GPU workloads.CUDA 工作负载由 NVIDIA GPU 加速，并且特定于 ARM CPU 矢量扩展（如 SVE）的优化不会影响这些 GPU 工作负载。
Possible Indirect Benefits:可能的间接好处： While the PR itself is unlikely to affect CUDA directly, there could be indirect benefits. For example, if the user is running hybrid workloads that offload some tasks to the GPU while others remain on the CPU, the ARM CPU-side improvements (like SVE) could help with tasks that remain on the CPU, potentially improving the overall system throughput. However, this would depend on the specific workload and the balance between CPU and GPU operations.虽然 PR 本身不太可能直接影响 CUDA，但可能会带来间接的好处。例如，如果用户运行混合工作负载，将某些任务卸载到 GPU，而其他任务保留在 CPU 上，则 ARM CPU 端改进（如 SVE）可以帮助处理保留在 CPU 上的任务，从而有可能提高整体系统吞吐量。但是，这取决于具体的工作负载以及 CPU 和 GPU 操作之间的平衡。

Thank you very much, this has addressed most of my questions.

However, I still have one doubt: According to the blog lets-talk-about-the-pytorch-dispatcher and native_functions.yaml, not all operators have specifically registered implementations for CUDA, means some operators use a generic kernel implementation (CPU) across all devices. So could these operators potentially benefit from this change as well, in spite of I use CUDA as backend ?

Unfortunately, I've tried reading PyTorch's documentation and source code, but haven't reached a very clear conclusion on this point.

Yes, I think, potentially, certain operators that fall back on the CPU implementation could benefit from the SVE optimizations introduced by our PR. Here's why:

Fallback to CPU: If the dispatcher falls back to a CPU implementation for an operator on a system using CUDA (because no CUDA-specific kernel exists), then the CPU version of the operator will be executed. In this case, your improvements to the ATen CPU backend (with SVE support for ARM) will help optimize that CPU-side execution.

@maajidkhann I'm interested in this PR, but I would like to know if there would be any benefits based on this PR when using CUDA as the backend? I would greatly appreciate if you could provide me with some information.我对此 PR 感兴趣，但我想知道使用 CUDA 作为后端时基于此 PR 是否有任何好处？如果您能为我提供一些信息，我将不胜感激。

Since the code in this PR is extending SVE support for ARM CPUs, it would primarily benefit CPU workloads running on ARM hardware and has SVE hardware support. If the user is leveraging CUDA as the backend for GPU acceleration, the PR would likely have no direct impact.由于此 PR 中的代码正在扩展对 ARM CPU 的 SVE 支持，因此它主要有利于在 ARM 硬件上运行并具有 SVE 硬件支持的 CPU 工作负载。如果用户利用 CUDA 作为 GPU 加速的后端，PR 可能不会产生直接影响。
CUDA workloads are accelerated by NVIDIA GPUs, and optimizations specific to ARM CPU vector extensions (like SVE) would not affect these GPU workloads.CUDA 工作负载由 NVIDIA GPU 加速，并且特定于 ARM CPU 矢量扩展（如 SVE）的优化不会影响这些 GPU 工作负载。
Possible Indirect Benefits:可能的间接好处： While the PR itself is unlikely to affect CUDA directly, there could be indirect benefits. For example, if the user is running hybrid workloads that offload some tasks to the GPU while others remain on the CPU, the ARM CPU-side improvements (like SVE) could help with tasks that remain on the CPU, potentially improving the overall system throughput. However, this would depend on the specific workload and the balance between CPU and GPU operations.虽然 PR 本身不太可能直接影响 CUDA，但可能会带来间接的好处。例如，如果用户运行混合工作负载，将某些任务卸载到 GPU，而其他任务保留在 CPU 上，则 ARM CPU 端改进（如 SVE）可以帮助处理保留在 CPU 上的任务，从而有可能提高整体系统吞吐量。但是，这取决于具体的工作负载以及 CPU 和 GPU 操作之间的平衡。

Thank you very much, this has addressed most of my questions.
However, I still have one doubt: According to the blog lets-talk-about-the-pytorch-dispatcher and native_functions.yaml, not all operators have specifically registered implementations for CUDA, means some operators use a generic kernel implementation (CPU) across all devices. So could these operators potentially benefit from this change as well, in spite of I use CUDA as backend ?
Unfortunately, I've tried reading PyTorch's documentation and source code, but haven't reached a very clear conclusion on this point.

Yes, I think, potentially, certain operators that fall back on the CPU implementation could benefit from the SVE optimizations introduced by our PR. Here's why:

Fallback to CPU: If the dispatcher falls back to a CPU implementation for an operator on a system using CUDA (because no CUDA-specific kernel exists), then the CPU version of the operator will be executed. In this case, your improvements to the ATen CPU backend (with SVE support for ARM) will help optimize that CPU-side execution.

got it! thanks for your response