Skip to content

dnn: avoid const layer forwarding in layer norm layer and attention layer#25238

Merged
asmorkalov merged 12 commits intoopencv:4.xfrom
fengyuentau:optimized_const
Mar 26, 2024
Merged

dnn: avoid const layer forwarding in layer norm layer and attention layer#25238
asmorkalov merged 12 commits intoopencv:4.xfrom
fengyuentau:optimized_const

Conversation

@fengyuentau
Copy link
Copy Markdown
Member

@fengyuentau fengyuentau commented Mar 20, 2024 •
edited
Loading

While profiling ViTs with dnn, I found ConstLayer can take a proportion of the inference time, which is weird. This comes from the data copy during the inference of ConstLayer. There is a chance that we can improve the efficiency of data copying but the easiest and most convenient way is to avoid ConstLayer. This PR change the way how we handle constants in layer normalization layer and attention layer, which is storing in the layer blobs instead of making constant layers for them.

Checklists:

  • Backend compatibility in layer normalization layer.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
force_builders=Linux OpenCL, Win64 OpenCL

@fengyuentau fengyuentau added this to the 4.10.0 milestone Mar 20, 2024
@fengyuentau fengyuentau requested a review from dkurt March 20, 2024 09:10
@dkurt
Copy link
Copy Markdown
Member

dkurt commented Mar 20, 2024

@fengyuentau, good catch, thanks! Once you can cover backends initialization, let's merge this.

@fengyuentau fengyuentau requested a review from vpisarev March 21, 2024 09:30
@fengyuentau
Copy link
Copy Markdown
Member Author

fengyuentau commented Mar 21, 2024

Backend intialization with constant blobs is done. Lets see whether all tests are passed or not.

asmorkalov
asmorkalov reviewed Mar 25, 2024
View reviewed changes
@fengyuentau fengyuentau changed the title dnn: store and use constants in blobs for layer normalization layer and attention layer dnn: avoid const layer forwarding in layer norm layer and attention layer Mar 26, 2024
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Mar 26, 2024

Some perf numbers for AMD Ryzen 7 2700X, 64Gb RAM:

Geometric mean (ms)

               Name of Test                  4.x-1  patched-1 patched-1 
                                                                  vs    
                                                                4.x-1   
                                                              (x-factor)
VIT_B_32::DNNTestNetwork::OCV/CPU           122.025  114.232     1.07   
VIT_B_32::DNNTestNetwork::OCV/OCL           310.753  240.440     1.29   
VIT_B_32::DNNTestNetwork::OCV/OCL_FP16      307.549  239.036     1.29   
VisionTransformer::Layer_Attention::OCV/CPU  4.689    4.760      0.98   
VitTrack::DNNTestNetwork::OCV/CPU            6.730    6.600      1.02   
VitTrack::DNNTestNetwork::OCV/OCL           11.914   11.575      1.03   
VitTrack::DNNTestNetwork::OCV/OCL_FP16      11.870   11.442      1.04

asmorkalov
asmorkalov approved these changes Mar 26, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@asmorkalov asmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov asmorkalov merged commit accf200 into opencv:4.x Mar 26, 2024
@fengyuentau fengyuentau deleted the optimized_const branch March 26, 2024 12:59
@asmorkalov asmorkalov mentioned this pull request Apr 1, 2024
Merged
klatism pushed a commit to klatism/opencv that referenced this pull request May 17, 2024
@fengyuentau @klatism
Merge pull request opencv#25238 from fengyuentau:optimized_const
f2d5aa6 
dnn: avoid const layer forwarding in layer norm layer and attention layer opencv#25238

While profiling ViTs with dnn, I found `ConstLayer` can take a proportion of the inference time, which is weird. This comes from the data copy during the inference of `ConstLayer`. There is a chance that we can improve the efficiency of data copying but the easiest and most convenient way is to avoid `ConstLayer`. This PR change the way how we handle constants in layer normalization layer and attention layer, which is storing in the layer blobs instead of making constant layers for them.

Checklists:

- [x] Backend compatibility in layer normalization layer.

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asmorkalov asmorkalov asmorkalov approved these changes

@dkurt dkurt dkurt approved these changes

@vpisarev vpisarev Awaiting requested review from vpisarev

Assignees

@dkurt dkurt

Labels

category: dnn

Projects

OpenCV 5.0
Status: Done

Milestone

4.10.0

Development

Successfully merging this pull request may close these issues.

3 participants

@fengyuentau @dkurt @asmorkalov