Skip to content

Add Support for Einsum Layer#24037

Merged
asmorkalov merged 29 commits intoopencv:4.xfrom
Abdurrahheem:ash/dev_einsum
Sep 22, 2023
Merged

Add Support for Einsum Layer#24037
asmorkalov merged 29 commits intoopencv:4.xfrom
Abdurrahheem:ash/dev_einsum

Conversation

@Abdurrahheem
Copy link
Copy Markdown
Contributor

@Abdurrahheem Abdurrahheem commented Jul 21, 2023 •
edited by dkurt
Loading

This PR adding support for Einsum Layer (in progress).

This PR is currently not to be merged but only reviewed. Test cases are located in #1090RP in OpenCV extra

resolves #23134
resolves #19067

DONE:

  • 2-5D GMM support added
  • Matrix transpose support added
  • Reduction type comupte 'ij->j'
  • 2nd shape computation - during forward

Next PRs:

  • Broadcasting reduction "...ii ->...i"
  • Add lazy shape deduction. "...ij, ...jk->...ik"
  • Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
  • Add support for CUDA backend
  • BatchWiseMultiply optimize

Later in 5.x version (requires support for 1D matrices):

  • Add 1D vector multiplication support
  • Inter product "i, i" (problems with 1D shapes)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@Abdurrahheem Abdurrahheem added category: dnn category: dnn (onnx) ONNX suport issues in DNN module labels Jul 21, 2023
@Abdurrahheem Abdurrahheem added this to the 4.9.0 milestone Jul 21, 2023
@Abdurrahheem Abdurrahheem self-assigned this Jul 21, 2023
dkurt
dkurt reviewed Jul 21, 2023
View reviewed changes
@dkurt dkurt added the pr: needs test New functionality requires minimal tests set label Jul 21, 2023
@fengyuentau fengyuentau mentioned this pull request Aug 14, 2023
Closed
5 tasks
@Abdurrahheem Abdurrahheem marked this pull request as ready for review August 25, 2023 17:40
dkurt
dkurt reviewed Aug 28, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
return true;
}

Mat batchwiseMatMul(
Copy link
Copy Markdown
Member

@dkurt dkurt Aug 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create an object of FullyConnected layer and use it for inference? For example, Permute layer inside ShuffleChannel

dkurt
dkurt reviewed Aug 28, 2023
View reviewed changes
fengyuentau
fengyuentau reviewed Aug 29, 2023
View reviewed changes
fengyuentau
fengyuentau reviewed Aug 29, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
part2 = part2.reshape(1, sizeof(shape2)/sizeof(shape2[0]), shape2);

Mat tmp_output;
cv::gemm(part1, part2, 1.0, cv::Mat(), 1.0, tmp_output);
Copy link
Copy Markdown
Member

@fengyuentau fengyuentau Aug 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use the fast_gemm which is introduced in #23897 once it is merged.

Copy link
Copy Markdown
Member

@dkurt dkurt Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or even entire Gemm layer, without using separate primitives.

fengyuentau
fengyuentau reviewed Aug 29, 2023
View reviewed changes
@Abdurrahheem Abdurrahheem mentioned this pull request Aug 30, 2023
Merged
dkurt
dkurt reviewed Sep 19, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 19, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 19, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 19, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 19, 2023
View reviewed changes
asmorkalov
asmorkalov approved these changes Sep 19, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@asmorkalov asmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
}
}

void LayerEinsumImpl::createOutputSubsctipt()
Copy link
Copy Markdown
Member

@dkurt dkurt Sep 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method seems useless. Both branches lead to an error.

dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
"Check input shapes/equation passed."
"Input shape of operand [%d]", inputIdx) +
cv::format(" is incompatible in the dimention [%zu].", static_cast<size_t>(dim_count)));
}
Copy link
Copy Markdown
Member

@dkurt dkurt Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CV_CheckEQ

dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
currentLeft = reduceSum(tensorToReduce, shapeToReduce);
}

} else {
Copy link
Copy Markdown
Member

@dkurt dkurt Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mix of code styles

dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
dkurt
dkurt reviewed Sep 20, 2023
View reviewed changes
modules/dnn/src/layers/einsum_layer.cpp
// assuming that we won't encounter any input with a rank greater than 10.
// In such cases, the value of num_subscript_indices_ would be greater than 10.
subscriptIndicesToLastInput.reserve(10);
subscriptIndicesToDimValue.reserve(10);
Copy link
Copy Markdown
Member

@dkurt dkurt Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this allocation?

dkurt
dkurt reviewed Sep 22, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@dkurt dkurt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge. The rest of syntax issues will be solved later. Missed functionality will be adjusted later.

fengyuentau
fengyuentau approved these changes Sep 22, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@fengyuentau fengyuentau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov asmorkalov merged commit 865e7ca into opencv:4.x Sep 22, 2023
@opencv-alalek opencv-alalek mentioned this pull request Sep 24, 2023
Closed
@opencv-alalek
Copy link
Copy Markdown
Contributor

opencv-alalek commented Sep 24, 2023

Several tests don't pass #24311

@Abdurrahheem Abdurrahheem mentioned this pull request Sep 26, 2023
Merged
12 tasks
@asmorkalov asmorkalov mentioned this pull request Sep 28, 2023
Merged
hanliutong pushed a commit to hanliutong/opencv that referenced this pull request Oct 7, 2023
@Abdurrahheem @hanliutong
Merge pull request opencv#24037 from Abdurrahheem:ash/dev_einsum
cd81901 
Add Support for Einsum Layer opencv#24037

### This PR adding support for [Einsum Layer](https://pytorch.org/docs/stable/generated/torch.einsum.html) (in progress). 

This PR is currently not to be merged but only reviewed. Test cases are located in [opencv#1090](opencv/opencv_extra#1090 in OpenCV extra

**DONE**: 
 - [x] 2-5D GMM support added
 - [x] Matrix transpose support added
 - [x] Reduction type comupte  'ij->j'
 - [x] 2nd shape computation - during forward 

**Next PRs**:
- [ ] Broadcasting reduction "...ii ->...i"
- [ ] Add lazy shape deduction. "...ij, ...jk->...ik"
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize

**Later in 5.x version (requires support for 1D matrices)**: 
- [ ] Add 1D vector multiplication support 
- [ ] Inter product "i, i" (problems with 1D shapes)

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
asmorkalov pushed a commit that referenced this pull request Oct 24, 2023
@Abdurrahheem
Merge pull request #24322 from Abdurrahheem:ash/dev_einsum_ellips
a3b3a58 
Ellipses supported added for Einsum Layer #24322

This PR added addresses issues not covered in #24037. Namely these are: 
Test case for this patch is in this PR [#1106](opencv/opencv_extra#1106) in opencv extra

Added: 
 - [x] Broadcasting reduction "...ii ->...I"
 - [x] Add lazy shape deduction. "...ij, ...jk->...ik"
 
 Features to add: 
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize
- [ ] Performance test

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
IskXCr pushed a commit to Haosonn/opencv that referenced this pull request Dec 20, 2023
@Abdurrahheem @IskXCr
Merge pull request opencv#24322 from Abdurrahheem:ash/dev_einsum_ellips
4cac695 
Ellipses supported added for Einsum Layer opencv#24322

This PR added addresses issues not covered in opencv#24037. Namely these are: 
Test case for this patch is in this PR [opencv#1106](opencv/opencv_extra#1106) in opencv extra

Added: 
 - [x] Broadcasting reduction "...ii ->...I"
 - [x] Add lazy shape deduction. "...ij, ...jk->...ik"
 
 Features to add: 
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize
- [ ] Performance test

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
thewoz pushed a commit to thewoz/opencv that referenced this pull request Jan 4, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#24037 from Abdurrahheem:ash/dev_einsum
a794eaf 
Add Support for Einsum Layer opencv#24037

### This PR adding support for [Einsum Layer](https://pytorch.org/docs/stable/generated/torch.einsum.html) (in progress). 

This PR is currently not to be merged but only reviewed. Test cases are located in [opencv#1090](opencv/opencv_extra#1090 in OpenCV extra

**DONE**: 
 - [x] 2-5D GMM support added
 - [x] Matrix transpose support added
 - [x] Reduction type comupte  'ij->j'
 - [x] 2nd shape computation - during forward 

**Next PRs**:
- [ ] Broadcasting reduction "...ii ->...i"
- [ ] Add lazy shape deduction. "...ij, ...jk->...ik"
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize

**Later in 5.x version (requires support for 1D matrices)**: 
- [ ] Add 1D vector multiplication support 
- [ ] Inter product "i, i" (problems with 1D shapes)

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
thewoz pushed a commit to thewoz/opencv that referenced this pull request Jan 4, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#24322 from Abdurrahheem:ash/dev_einsum_ellips
5c7874e 
Ellipses supported added for Einsum Layer opencv#24322

This PR added addresses issues not covered in opencv#24037. Namely these are: 
Test case for this patch is in this PR [opencv#1106](opencv/opencv_extra#1106) in opencv extra

Added: 
 - [x] Broadcasting reduction "...ii ->...I"
 - [x] Add lazy shape deduction. "...ij, ...jk->...ik"
 
 Features to add: 
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize
- [ ] Performance test

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
@Abdurrahheem Abdurrahheem mentioned this pull request Jan 8, 2024
Merged
6 tasks
thewoz pushed a commit to thewoz/opencv that referenced this pull request May 29, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#24037 from Abdurrahheem:ash/dev_einsum
07e0a00 
Add Support for Einsum Layer opencv#24037

### This PR adding support for [Einsum Layer](https://pytorch.org/docs/stable/generated/torch.einsum.html) (in progress). 

This PR is currently not to be merged but only reviewed. Test cases are located in [opencv#1090](opencv/opencv_extra#1090 in OpenCV extra

**DONE**: 
 - [x] 2-5D GMM support added
 - [x] Matrix transpose support added
 - [x] Reduction type comupte  'ij->j'
 - [x] 2nd shape computation - during forward 

**Next PRs**:
- [ ] Broadcasting reduction "...ii ->...i"
- [ ] Add lazy shape deduction. "...ij, ...jk->...ik"
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize

**Later in 5.x version (requires support for 1D matrices)**: 
- [ ] Add 1D vector multiplication support 
- [ ] Inter product "i, i" (problems with 1D shapes)

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
thewoz pushed a commit to thewoz/opencv that referenced this pull request May 29, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#24322 from Abdurrahheem:ash/dev_einsum_ellips
efeabe7 
Ellipses supported added for Einsum Layer opencv#24322

This PR added addresses issues not covered in opencv#24037. Namely these are: 
Test case for this patch is in this PR [opencv#1106](opencv/opencv_extra#1106) in opencv extra

Added: 
 - [x] Broadcasting reduction "...ii ->...I"
 - [x] Add lazy shape deduction. "...ij, ...jk->...ik"
 
 Features to add: 
- [ ] Add implicit output computation support. "bij,bjk ->" (output subscripts should be "bik")
- [ ] Add support for CUDA backend 
- [ ] BatchWiseMultiply optimize
- [ ] Performance test

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dkurt dkurt dkurt approved these changes

@opencv-alalek opencv-alalek opencv-alalek left review comments

@asmorkalov asmorkalov asmorkalov approved these changes

@fengyuentau fengyuentau fengyuentau approved these changes

@vpisarev vpisarev Awaiting requested review from vpisarev

Assignees

@Abdurrahheem Abdurrahheem

Labels

category: dnn (onnx) ONNX suport issues in DNN module category: dnn

Projects

None yet

Milestone

4.9.0

Development

Successfully merging this pull request may close these issues.

The Einsum Layer will be supported in the future Can't create layer "195" of type "Einsum" in function 'getLayerInstance'

5 participants

@Abdurrahheem @opencv-alalek @asmorkalov @fengyuentau @dkurt