Skip to content

Added LSTM and GRU tests for various batch and input length sizes#23501

Merged
asmorkalov merged 5 commits intoopencv:4.xfrom
Abdurrahheem:additional_lstm_tests
Apr 20, 2023
Merged

Added LSTM and GRU tests for various batch and input length sizes#23501
asmorkalov merged 5 commits intoopencv:4.xfrom
Abdurrahheem:additional_lstm_tests

Conversation

@Abdurrahheem
Copy link
Copy Markdown
Contributor

@Abdurrahheem Abdurrahheem commented Apr 17, 2023 •
edited by asmorkalov
Loading

Added tests with various sequence length and batch sizes

Test data: opencv/opencv_extra#1057

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@Abdurrahheem Abdurrahheem mentioned this pull request Apr 17, 2023
Merged
@Abdurrahheem Abdurrahheem self-assigned this Apr 17, 2023
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 18, 2023

@Abdurrahheem You need to rebase the PR to OpenCV Extra on top of 4.x. There is merge conflict there.

@asmorkalov asmorkalov requested a review from dkurt April 18, 2023 06:20
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 18, 2023 

[ RUN      ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/0, where GetParam() = CUDA/CUDA
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:75: Failure
Expected: (normL1) <= (l1), actual: 0.0991663 vs 1e-05
  |ref| = 0.62599176168441772
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:78: Failure
Expected: (normInf) <= (lInf), actual: 0.474885 vs 0.0001
  |ref| = 0.62599176168441772
[  FAILED  ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/0, where GetParam() = CUDA/CUDA (5 ms)
[ RUN      ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/1, where GetParam() = CUDA/CUDA_FP16
[       OK ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/1 (7 ms)
[ RUN      ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/2, where GetParam() = OCV/CPU
[       OK ] Test_ONNX_layers.LSTM_cell_batchsize_1_seqlen_50/2 (1 ms)
[ RUN      ] Test_ONNX_layers.LSTM_cell_batchsize_5_seqlen_5/0, where GetParam() = CUDA/CUDA
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:75: Failure
Expected: (normL1) <= (l1), actual: 0.15313 vs 1e-05
  |ref| = 0.98481488227844238
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:78: Failure
Expected: (normInf) <= (lInf), actual: 0.846282 vs 0.0001
  |ref| = 0.98481488227844238
[  FAILED  ] Test_ONNX_layers.LSTM_cell_batchsize_5_seqlen_5/0, where GetParam() = CUDA/CUDA (5 ms)

It looks like CUDA branch of LSTM is not complete. Please skip CuDNN backend in test like here: https://github.com/opencv/opencv/blob/4.x/modules/dnn/test/test_layers.cpp#L231, create ticket about LSTM CUDA issue and add reference to it to test code.

@opencv-alalek opencv-alalek added this to the 4.8.0 milestone Apr 18, 2023
@Abdurrahheem Abdurrahheem mentioned this pull request Apr 19, 2023
Open
4 tasks
@asmorkalov asmorkalov changed the title Added LSTM tests for various batch and input length sizes Added LSTM and GRU tests for various batch and input length sizes Apr 19, 2023
@asmorkalov asmorkalov requested review from fengyuentau and removed request for AleksandrPanov and asmorkalov April 19, 2023 13:13
asmorkalov
asmorkalov reviewed Apr 19, 2023
View reviewed changes
@asmorkalov asmorkalov marked this pull request as ready for review April 19, 2023 16:09
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 20, 2023

@dkurt @fengyuentau Could you take a look too.

fengyuentau
fengyuentau approved these changes Apr 20, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@fengyuentau fengyuentau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

asmorkalov pushed a commit to opencv/opencv_extra that referenced this pull request Apr 20, 2023
@Abdurrahheem
Merge pull request #1057 from Abdurrahheem:additional_lstm_tests
4f4eb3a 
Main PR opencv/opencv#23501

* added LSTM tests for various batch and seqeuence lenght sizes
* Added additional GRU tests for seq length and batch size
@asmorkalov asmorkalov merged commit b3a2444 into opencv:4.x Apr 20, 2023
@asmorkalov asmorkalov mentioned this pull request Apr 25, 2023
Closed
4 tasks
@asmorkalov asmorkalov mentioned this pull request May 31, 2023
Merged
thewoz pushed a commit to thewoz/opencv that referenced this pull request Jan 4, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#23501 from Abdurrahheem:additional_lstm_tests
5d2948d 
Added LSTM and GRU tests for various batch and input length sizes opencv#23501

Added tests with various sequence length and batch sizes
Test data: opencv/opencv_extra#1057

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
thewoz pushed a commit to thewoz/opencv that referenced this pull request May 29, 2024
@Abdurrahheem @thewoz
Merge pull request opencv#23501 from Abdurrahheem:additional_lstm_tests
b6c797a 
Added LSTM and GRU tests for various batch and input length sizes opencv#23501

Added tests with various sequence length and batch sizes
Test data: opencv/opencv_extra#1057

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asmorkalov asmorkalov asmorkalov approved these changes

@fengyuentau fengyuentau fengyuentau approved these changes

@dkurt dkurt Awaiting requested review from dkurt

Assignees

@asmorkalov asmorkalov

Labels

category: dnn test

Projects

None yet

Milestone

4.8.0

Development

Successfully merging this pull request may close these issues.

4 participants

@Abdurrahheem @asmorkalov @fengyuentau @opencv-alalek