[feature] add `A Comprehensive Overhaul of Feature Distillation` by fpshuang · Pull Request #244 · open-mmlab/mmrazor

fpshuang · 2022-08-23T13:28:26Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Add OFD A Comprehensive Overhaul of Feature Distillation, a KD algo

Modification

add OFD components
a OFD experiment on CIFAR10
comprehensive UTs

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

codecov · 2022-08-23T13:38:58Z

Codecov Report

Merging #244 (7550f1c) into dev-1.x (1c0da58) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

❗ Current head 7550f1c differs from pull request most recent head 1c2ca44. Consider uploading reports for the commit 1c2ca44 to get more accurate results

@@            Coverage Diff             @@
##           dev-1.x    #244      +/-   ##
==========================================
- Coverage     0.43%   0.42%   -0.01%     
==========================================
  Files          167     168       +1     
  Lines         6700    6776      +76     
  Branches      1081    1102      +21     
==========================================
  Hits            29      29              
- Misses        6666    6742      +76     
  Partials         5       5

Flag	Coverage Δ
unittests	`0.42% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmrazor/models/algorithms/__init__.py	`0.00% <ø> (ø)`
mmrazor/models/algorithms/distill/__init__.py	`0.00% <ø> (ø)`
...models/algorithms/distill/configurable/__init__.py	`0.00% <0.00%> (ø)`
...till/configurable/overhaul_feature_distillation.py	`0.00% <0.00%> (ø)`
mmrazor/models/architectures/backbones/__init__.py	`0.00% <0.00%> (ø)`
...razor/models/architectures/backbones/wideresnet.py	`0.00% <0.00%> (ø)`
...mrazor/models/architectures/connectors/__init__.py	`0.00% <0.00%> (ø)`
...r/models/architectures/connectors/ofd_connector.py	`0.00% <0.00%> (ø)`
mmrazor/models/distillers/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/distillers/ofd_distiller.py	`0.00% <0.00%> (ø)`
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

wilxy · 2022-08-25T03:45:30Z

configs/distill/mmcls/ofd/README.md

@@ -0,0 +1,160 @@
+# A Comprehensive Overhaul of Feature Distillation
+


readme needs to update

wilxy · 2022-08-25T03:50:13Z

mmrazor/models/algorithms/distill/configurable/overhaul_feature_distillation.py

+
+
+@MODELS.register_module()
+class OverhaulFeatureDistillation(SingleTeacherDistill):


mmrazor/models/architectures/connectors/ofd_connector.py

wilxy · 2022-08-25T03:56:48Z

mmrazor/models/architectures/backbones/wideresnet.py

+
+
+@MODELS.register_module()
+class WideResNet(BaseModule):


Does this module requires UT?

wilxy · 2022-08-25T03:58:26Z

mmrazor/models/distillers/ofd_distiller.py

+
+
+@MODELS.register_module()
+class OFDDistiller(ConfigurableDistiller):


UT and docstring

wilxy · 2022-08-25T03:59:18Z

mmrazor/models/losses/ofd_loss.py

+
+
+@MODELS.register_module()
+class OFDLoss(nn.Module):


wilxy · 2022-08-25T04:01:56Z

It's ok to me when UT and docstring are added.

configs/distill/mmcls/ofd/README.md

…n-mmlab#244) * Add sort-data arg to test.py * Set is_sort_dataset to True. * Add a check for the possibility of sorting. * lint * Added mmdeploy.utils.dataset. * Add unit test Co-authored-by: SingleZombie <singlezombie@163.com>

huangpengsheng added 2 commits August 23, 2022 18:25

init

1459ee1

init

47e9e7f

fpshuang changed the title ~~Ofd~~ WIP: [feature] add A Comprehensive Overhaul of Feature Distillation Aug 23, 2022

fpshuang marked this pull request as draft August 23, 2022 13:28

fpshuang changed the base branch from master to dev-1.x August 23, 2022 13:29

huangpengsheng added 5 commits August 24, 2022 17:10

linting

c76cbd6

add README

4df2588

resolve conflicts

437d782

add wrn docstrings

81f5ba4

add vanilla wrn configs

6902f9e

fpshuang requested review from aptsunny, gaoyang07, pppppM, spynccat, sunnyxiaohu, wilxy and wutongshenqiu August 24, 2022 15:22

fpshuang marked this pull request as ready for review August 24, 2022 15:22

fix UT

a652661

wilxy reviewed Aug 25, 2022

View reviewed changes

huangpengsheng added 2 commits August 25, 2022 14:48

OFD DONE

bd66df4

update OFD readme

7550f1c

sunnyxiaohu reviewed Aug 25, 2022

View reviewed changes

configs/distill/mmcls/ofd/README.md Show resolved Hide resolved

fpshuang changed the title ~~WIP: [feature] add A Comprehensive Overhaul of Feature Distillation~~ [feature] add A Comprehensive Overhaul of Feature Distillation Aug 26, 2022

huangpengsheng added 4 commits August 26, 2022 11:55

update config path

b5728aa

rename vanilla model files

cc39d32

rename vanilla models config files

b660cdf

resolve conflicts

1c2ca44

sunnyxiaohu merged commit 5d9fcd8 into open-mmlab:dev-1.x Aug 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] add `A Comprehensive Overhaul of Feature Distillation`#244

[feature] add `A Comprehensive Overhaul of Feature Distillation`#244
sunnyxiaohu merged 14 commits intoopen-mmlab:dev-1.xfrom
fpshuang:ofd

fpshuang commented Aug 23, 2022

Uh oh!

codecov bot commented Aug 23, 2022 •

edited

Loading

Uh oh!

wilxy Aug 25, 2022

Uh oh!

wilxy Aug 25, 2022

Uh oh!

Uh oh!

wilxy Aug 25, 2022

Uh oh!

wilxy Aug 25, 2022

Uh oh!

wilxy Aug 25, 2022

Uh oh!

fpshuang Aug 25, 2022

Uh oh!

wilxy commented Aug 25, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,160 @@
		# A Comprehensive Overhaul of Feature Distillation



		@MODELS.register_module()
		class OverhaulFeatureDistillation(SingleTeacherDistill):



		@MODELS.register_module()
		class OFDDistiller(ConfigurableDistiller):

Conversation

fpshuang commented Aug 23, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

Uh oh!

codecov bot commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wilxy Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

wilxy Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wilxy Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

wilxy Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

wilxy Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

fpshuang Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

wilxy commented Aug 25, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Aug 23, 2022 •

edited

Loading