[Linalg] Add cusolver syevjBatched path for torch.linalg.eigh when cuda >= 11.3 U1#62003
Closed
xwang233 wants to merge 9 commits intopytorch:masterfrom
Closed
[Linalg] Add cusolver syevjBatched path for torch.linalg.eigh when cuda >= 11.3 U1#62003xwang233 wants to merge 9 commits intopytorch:masterfrom
xwang233 wants to merge 9 commits intopytorch:masterfrom
Conversation
Contributor
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit 39b7245 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
…-batched-cuda11.3.1
…-batched-cuda11.3.1
IvanYashchuk
approved these changes
Jul 23, 2021
Collaborator
IvanYashchuk
left a comment
There was a problem hiding this comment.
It's great that the bug is fixed!
| #endif | ||
|
|
||
| // cusolverDn<T>syevjBatched may have numerical issue before cuda 11.3.1 release, | ||
| // (which is cusolver version 11102 in the header), so we only use cusolver potrf batched |
Collaborator
There was a problem hiding this comment.
Suggested change
| // (which is cusolver version 11102 in the header), so we only use cusolver potrf batched | |
| // (which is cusolver version 11102 in the header), so we only use cusolver syevj batched |
Collaborator
Author
There was a problem hiding this comment.
Thanks for finding this! 😂
Co-authored-by: Ivan Yashchuk <IvanYashchuk@users.noreply.github.com>
ngimel
reviewed
Jul 28, 2021
| } | ||
|
|
||
| void linalg_eigh_cusolver(const Tensor& eigenvalues, const Tensor& eigenvectors, const Tensor& infos, bool upper, bool compute_eigenvectors) { | ||
| // syevj is better than syevd for float32 dtype and matrix sizes 32x32 - 512x512 |
Collaborator
There was a problem hiding this comment.
this comment is misplaced (it refers to line 1256 now). Can you also add another comment with link to the data about why we are using syevjBatched for heuristics in line 1254?
…-batched-cuda11.3.1
Collaborator
|
fyi @ngimel is merging this one |
Collaborator
Author
|
Thank you! |
ngimel
approved these changes
Jul 30, 2021
Contributor
|
@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Contributor
laurentdupin
pushed a commit
to laurentdupin/pytorch
that referenced
this pull request
Apr 25, 2026
…da >= 11.3 U1 (pytorch#62003) Summary: This PR adds the `cusolverDn<T>SyevjBatched` fuction to the backend of `torch.linalg.eigh` (eigenvalue solver for Hermitian matrix). Using the heuristics from pytorch#53040 (comment) and my local tests, the `syevj_batched` path is only used when `batch_size > 1` and `matrix_size <= 32`. This would give us huge performance boost in those cases. Since there were known numerical issues on cusolver `syevj_batched` before cuda 11.3 update 1, this PR only enables the dispatch when cuda version is no less than that. See also pytorch#42666 pytorch#47953 pytorch#53040 Pull Request resolved: pytorch#62003 Reviewed By: heitorschueroff Differential Revision: D30006316 Pulled By: ngimel fbshipit-source-id: 3a65c5fc9adbbe776524f8957df5442c3d3aeb8e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the
cusolverDn<T>SyevjBatchedfuction to the backend oftorch.linalg.eigh(eigenvalue solver for Hermitian matrix). Using the heuristics from #53040 (comment) and my local tests, thesyevj_batchedpath is only used whenbatch_size > 1andmatrix_size <= 32. This would give us huge performance boost in those cases.Since there were known numerical issues on cusolver
syevj_batchedbefore cuda 11.3 update 1, this PR only enables the dispatch when cuda version is no less than that.See also #42666 #47953 #53040