Refactor the baseclass related to transformer by jshilong · Pull Request #978 · open-mmlab/mmcv

jshilong · 2021-04-23T03:08:29Z

PR Message

This pr refactor the FFN and BaseTransformerLayer.

Add registry named DROPOUT_LAYERS and support to specify the DropPath or Dropout when adding the residue in the Attention or FFN.
Add registry named FEEDFORWARD_NETWORK.
Support to specify different probability for MultiheadAttention . with attn_drop and proj_drop
Move all FFN related arguments to a list of Dict named ffn_cfgs so that we can build the FFN more flexibly.
Change all nn.ModuleList to Moduleist.
Support batch_first arguments for MultiheadAttention.
Move MultiScaleDeformableAttention to mmcv/ops/multi_scale_deform_attn.py so that mmcv can be used without compiling ops.

BC-breaking

from mmcv.cnn.bricks.transformer import MultiScaleDeformableAttention should be changed to from mmcv.ops.multi_scale_deform_attn import MultiScaleDeformableAttention

codecov · 2021-04-23T03:30:54Z

Codecov Report

Merging #978 (65bcd3f) into master (4d42365) will increase coverage by 2.29%.
The diff coverage is 74.86%.

@@            Coverage Diff             @@
##           master     #978      +/-   ##
==========================================
+ Coverage   65.20%   67.50%   +2.29%     
==========================================
  Files         156      159       +3     
  Lines       10034    10186     +152     
  Branches     1816     1847      +31     
==========================================
+ Hits         6543     6876     +333     
+ Misses       3154     2948     -206     
- Partials      337      362      +25

Flag	Coverage Δ
unittests	`67.50% <74.86%> (+2.29%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcv/ops/multi_scale_deform_attn.py	`58.40% <53.75%> (-11.29%)`	⬇️
mmcv/cnn/bricks/transformer.py	`81.62% <86.48%> (+81.62%)`	⬆️
mmcv/cnn/bricks/__init__.py	`100.00% <100.00%> (ø)`
mmcv/cnn/bricks/drop.py	`100.00% <100.00%> (ø)`
mmcv/cnn/bricks/registry.py	`100.00% <100.00%> (ø)`
mmcv/ops/__init__.py	`100.00% <100.00%> (ø)`
mmcv/onnx/info.py	`68.75% <0.00%> (-6.25%)`	⬇️
mmcv/tensorrt/tensorrt_utils.py	`0.00% <0.00%> (-2.39%)`	⬇️
mmcv/fileio/parse.py	`100.00% <0.00%> (ø)`
mmcv/runner/hooks/evaluation.py	`88.15% <0.00%> (ø)`
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d42365...65bcd3f. Read the comment docs.

[Feature]Replace dropout with attn_drop and proj_drop in MultiheadAttention

CLAassistant · 2021-04-26T08:33:30Z

All committers have signed the CLA.

Support registering FFN of transformer

mmcv/cnn/bricks/drop.py

mmcv/cnn/bricks/transformer.py

nbei

Just suggestions.

mmcv/cnn/bricks/drop.py

mmcv/cnn/bricks/registry.py

mmcv/ops/multi_scale_deform_attn.py

mmcv/cnn/bricks/transformer.py

tests/test_cnn/test_transformer.py

zhangshilong added 2 commits April 14, 2021 16:15

minor changes

d3977cd

change to modulist

36418e1

jshilong added the WIP label Apr 23, 2021

jshilong changed the title ~~change to modulist~~ Refactor the baseclass releated to transformer Apr 23, 2021

HIT-cwh added 2 commits April 23, 2021 15:25

change to Sequential

806fed0

replace dropout with attn_drop and proj_drop in MultiheadAttention

12140f8

ZwwWayne requested a review from nbei April 24, 2021 11:26

ZwwWayne assigned nbei Apr 24, 2021

zhangshilong and others added 2 commits April 26, 2021 16:13

add operation_name for attn

b15616e

Merge pull request #1 from HIT-cwh/refactor_transformer

6bc3254

[Feature]Replace dropout with attn_drop and proj_drop in MultiheadAttention

zhangshilong and others added 12 commits April 30, 2021 12:51

add drop path and move all ffn args to ffncfgs

869e87f

add drop path and move all ffn args to ffncfgs

a022220

fix typo

8b5d9b4

fix a bug when use default value of ffn_cfgs

7ab36c0

fix ffns

a1659aa

add deprecate warning

f808e2a

pull master

47cbf8b

fix deprecate warning

282a2e4

change to pop kwargs

8b15261

support register FFN of transformer

be135f3

support batch first

0f11d0f

fix batch first wapper

f3635bd

ZwwWayne mentioned this pull request May 14, 2021

Iteration Plan v1.3.5 - May 2021 #1026

Closed

9 tasks

zhangshilong and others added 5 commits May 15, 2021 21:48

fix forward wapper

5df1df0

fix typo

5e5b47e

Merge pull request #2 from congee524/transformer

8eb66e5

Support registering FFN of transformer

fix lint

464f5fe

add unitest for transformer

5e966cb

add noqa 501

d410613

xvjiarui reviewed Jun 9, 2021

View reviewed changes

mmcv/cnn/bricks/drop.py Outdated Show resolved Hide resolved

xvjiarui reviewed Jun 9, 2021

View reviewed changes

mmcv/cnn/bricks/drop.py Show resolved Hide resolved

mmcv/cnn/bricks/transformer.py Outdated Show resolved Hide resolved

mmcv/cnn/bricks/transformer.py Show resolved Hide resolved

jshilong added 3 commits June 10, 2021 11:03

move bnc wapper to MultiheadAttention

2444084

move bnc wapper to MultiheadAttention

d33f54e

use dep warning

f5622d1

jshilong requested review from ZwwWayne, nbei and xvjiarui June 10, 2021 03:58

xvjiarui approved these changes Jun 10, 2021

View reviewed changes

nbei reviewed Jun 10, 2021

View reviewed changes

nbei assigned nbei and unassigned nbei Jun 10, 2021

jshilong added 5 commits June 10, 2021 13:49

resolve comments

08f3739

add unitest:

987a11c

rename residual to identity

41830d5

revert runner

db3f857

msda residual to identity

d59c7fd

ZwwWayne reviewed Jun 11, 2021

View reviewed changes

mmcv/cnn/bricks/transformer.py Outdated Show resolved Hide resolved

ZwwWayne reviewed Jun 11, 2021

View reviewed changes

tests/test_cnn/test_transformer.py Outdated Show resolved Hide resolved

jshilong added 4 commits June 11, 2021 11:35

rename inp_identity to identity

d4871b7

fix name

4cb38de

fix transformer

2fb7ae0

remove key in msda

8b46c09

jshilong requested review from ZwwWayne and nbei June 11, 2021 04:04

remove assert for key

65bcd3f

ZwwWayne approved these changes Jun 11, 2021

View reviewed changes

jshilong mentioned this pull request Jun 11, 2021

fix the import path of msda open-mmlab/mmdetection#5338

Merged

ZwwWayne merged commit e05fb56 into open-mmlab:master Jun 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor the baseclass related to transformer#978

Refactor the baseclass related to transformer#978
ZwwWayne merged 47 commits intoopen-mmlab:masterfrom
jshilong:refactor_tramsformer_base

jshilong commented Apr 23, 2021 •

edited

Loading

Uh oh!

codecov bot commented Apr 23, 2021 •

edited

Loading

Uh oh!

CLAassistant commented Apr 26, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nbei left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

jshilong commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Message

BC-breaking

Uh oh!

codecov bot commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CLAassistant commented Apr 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nbei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jshilong commented Apr 23, 2021 •

edited

Loading

codecov bot commented Apr 23, 2021 •

edited

Loading

CLAassistant commented Apr 26, 2021 •

edited

Loading