[Feature] Add connector components and FitNet by wilxy · Pull Request #207 · open-mmlab/mmrazor

wilxy · 2022-07-26T13:32:23Z

New Features

Add connector components. Connector is mainly used for distillation, it usually converts the channel number of input feature to align features of student and teacher. In this PR, BaseConnector and three general connectors (SingleConvConnector, ConvBNConnector and ConvBNReLUConnector) are added.
ConfigurableDistiller adds two connector related parameters ( student_connectors and teacher_connectors) and function build_connectors to build connectors.
Add FitNet distillation algorithm and its README (where the model and log paths are empty, waiting for the training results of the unified benchmark before the release).
Add L2Loss. It can be used as the following example.
from mmrazor.models import L2Loss
loss_cfg = dict(loss_weight=10, normalize=False)
l2loss = L2Loss(**loss_cfg)

Improvement

Add student_trainable parameter in SingleTeacherDistill to control whether the student model is trainable.
Add calculate_student_loss attribute in SingleTeacherDistill to control whether the original task loss to be calculated and used to update student model.
Add test_connectors and test_losses in tests/test_models.

codecov · 2022-07-26T13:46:10Z

Codecov Report

Merging #207 (d75d120) into dev-1.x (6987511) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           dev-1.x    #207      +/-   ##
==========================================
- Coverage     0.68%   0.67%   -0.02%     
==========================================
  Files          115     119       +4     
  Lines         4216    4321     +105     
  Branches       659     675      +16     
==========================================
  Hits            29      29              
- Misses        4182    4287     +105     
  Partials         5       5

Flag	Coverage Δ
unittests	`0.67% <0.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...hms/distill/configurable/single_teacher_distill.py	`0.00% <0.00%> (ø)`
mmrazor/models/architectures/__init__.py	`0.00% <0.00%> (ø)`
...mrazor/models/architectures/connectors/__init__.py	`0.00% <0.00%> (ø)`
.../models/architectures/connectors/base_connector.py	`0.00% <0.00%> (ø)`
...dels/architectures/connectors/general_connector.py	`0.00% <0.00%> (ø)`
mmrazor/models/distillers/base_distiller.py	`0.00% <0.00%> (ø)`
...mrazor/models/distillers/configurable_distiller.py	`0.00% <0.00%> (ø)`
mmrazor/models/losses/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/losses/l2_loss.py	`0.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6987511...d75d120. Read the comment docs.

sunnyxiaohu · 2022-07-27T02:03:31Z

mmrazor/models/connectors/base_connector.py

+class BaseConnector(nn.Module, metaclass=ABCMeta):
+    """Base class of connectors.
+
+    Connector is mainly used for distill, it usually converts the channel


distill -> distillation

sunnyxiaohu · 2022-07-27T02:07:40Z

mmrazor/models/connectors/base_connector.py

+import torch.nn as nn
+
+
+class BaseConnector(nn.Module, metaclass=ABCMeta):


It's better to inherit from BaseModule?
Then, self.init_parameters() could be rewritten by init_weights() and init_cfg

sunnyxiaohu · 2022-07-27T02:12:03Z

mmrazor/models/connectors/general_connector.py

+
+
+@MODELS.register_module()
+class BNConnector(BaseConnector):


-> ConvBNConnnector

sunnyxiaohu · 2022-07-27T02:12:54Z

mmrazor/models/connectors/general_connector.py

+        out_channel: int,
+    ) -> None:
+        super().__init__()
+        self.conv = nn.Conv2d(in_channel, out_channel, kernel_size=1, stride=1)


Use build_conv_layer to build a conv layer

sunnyxiaohu · 2022-07-27T02:13:23Z

mmrazor/models/connectors/general_connector.py

+        out_channel: int,
+    ) -> None:
+        super().__init__()
+        self.conv = nn.Conv2d(


Use build_conv_layer to build a conv layer
build_norm_layer to build a norm layer

sunnyxiaohu · 2022-07-27T02:14:05Z

mmrazor/models/connectors/general_connector.py

+
+
+@MODELS.register_module()
+class ReLUConnector(BaseConnector):


->ConvBNReLUConnector?

sunnyxiaohu · 2022-07-27T02:14:52Z

mmrazor/models/connectors/general_connector.py

+        out_channel: int,
+    ) -> None:
+        super().__init__()
+        self.conv = nn.Conv2d(in_channel, out_channel, kernel_size=1)


sunnyxiaohu · 2022-07-27T02:19:17Z

mmrazor/models/distillers/configurable_distiller.py

        distill_deliveries (dict, optional): Config for multiple deliveries. A
            distill algorithm may have more than one delivery. Defaults to
            None.
+        student_connectors (dict, optional): Config for multiple connectors. A


Explain the key mapping relations between connectors, distill_losses, and loss_forward_mappings in Note below.

sunnyxiaohu · 2022-07-27T02:28:48Z

mmrazor/models/distillers/configurable_distiller.py

+        distill_connecotrs = nn.ModuleDict()
+        if connectors:
+            for loss_name, connector_cfg in connectors.items():
+                assert loss_name in self.distill_losses


Add assert_str

sunnyxiaohu · 2022-07-27T02:28:52Z

mmrazor/models/distillers/configurable_distiller.py

+        if connectors:
+            for loss_name, connector_cfg in connectors.items():
+                assert loss_name in self.distill_losses
+                assert loss_name not in distill_connecotrs


…replace nn.bn to build_norm_layer.

sunnyxiaohu · 2022-07-27T08:45:34Z

mmrazor/models/connectors/general_connector.py

+            stride=1,
+            padding=0,
+            bias=False)
+        _, self.bn = build_norm_layer(dict(type='BN'), out_channel)


sunnyxiaohu · 2022-07-27T08:45:53Z

mmrazor/models/connectors/general_connector.py

+        super().__init__(init_cfg)
+        self.conv = build_conv_layer(
+            conv_cfg, in_channel, out_channel, kernel_size=1)
+        _, self.bn = build_norm_layer(dict(type='BN'), out_channel)


…ctor to realize automatically invocation.

pppppM · 2022-07-28T08:06:08Z

I have three comments about this PR:

calculate_student_loss should be an attribute of the distillation algorithm, not the distiller
The current mapping between connectors and features cannot handle the loss, which has multiple inputs and each input has a different connector.
mmrazor.models.connector can be moved to mmrazor.models.architectures

For comment 2, I think you can refer to the mapping method between recorder, loss, and feature

…or.models.connector to mmrazor.models.architectures. 3.Merge stu_connectors and tea_connectors into connectors, and call connectors by their connector_name.

pppppM

LGTM if the latest comments are fixed.

pppppM · 2022-07-28T10:06:46Z

configs/distill/mmcls/fitnet/fitnet_backbone_logits_resnet50_resnet18_8xb32_in1k.py

+                    from_student=True,
+                    recorder='bb_s4',
+                    record_idx=1,
+                    connector_name='loss_s4_sfeat'),


connector_name -> connector

pppppM · 2022-07-28T10:07:55Z

mmrazor/models/distillers/configurable_distiller.py

                   record_idx: int = 0,
-                   data_idx: Optional[int] = None) -> List:
+                   data_idx: Optional[int] = None,
+                   connector_name: Optional[str] = None) -> List:


connector_name -> connector

…at each connector must be in connectors.

* [WIP] Refactor v2.0 (open-mmlab#163) * Refactor backend wrapper * Refactor mmdet.inference * Fix * merge * refactor utils * Use deployer and deploy_model to manage pipeline * Resolve comments * Add a real inference api function * rename wrappers * Set execute to private method * Rename deployer deploy_model * Refactor task * remove type hint * lint * Resolve comments * resolve comments * lint * docstring * [Fix]: Fix bugs in details in refactor branch (open-mmlab#192) * [WIP] Refactor v2.0 (open-mmlab#163) * Refactor backend wrapper * Refactor mmdet.inference * Fix * merge * refactor utils * Use deployer and deploy_model to manage pipeline * Resolve comments * Add a real inference api function * rename wrappers * Set execute to private method * Rename deployer deploy_model * Refactor task * remove type hint * lint * Resolve comments * resolve comments * lint * docstring * Fix errors * lint * resolve comments * fix bugs * conflict * lint and typo * Resolve comment * refactor mmseg (open-mmlab#201) * support mmseg * fix docstring * fix docstring * [Refactor]: Get the count of backend files (open-mmlab#202) * Fix backend files * resolve comments * lint * Fix ncnn * [Refactor]: Refactor folders of mmdet (open-mmlab#200) * Move folders * lint * test object detection model * lint * reset changes * fix openvino * resolve comments * __init__.py * Fix path * [Refactor]: move mmseg (open-mmlab#206) * [Refactor]: Refactor mmedit (open-mmlab#205) * feature mmedit * edit2.0 * edit * refactor mmedit * fix __init__.py * fix __init__ * fix formai * fix comment * fix comment * Fix wrong func_name of ConvFCBBoxHead (open-mmlab#209) * [Refactor]: Refactor mmdet unit test (open-mmlab#207) * Move folders * lint * test object detection model * lint * WIP * remove print * finish unit test * Fix tests * resolve comments * Add mask test * lint * resolve comments * Refine cfg file * Move files * add files * Fix path * [Unittest]: Refine the unit tests in mmdet open-mmlab#214 * [Refactor] refactor mmocr to mmdeploy/codebase (open-mmlab#213) * refactor mmocr to mmdeploy/codebase * fix docstring of show_result * fix docstring of visualize * refine docstring * replace print with logging * refince codes * resolve comments * resolve comments * [Refactor]: mmseg tests (open-mmlab#210) * refactor mmseg tests * rename test_codebase * update * add model.py * fix * [Refactor] Refactor mmcls and the package (open-mmlab#217) * refactor mmcls * fix yapf * fix isort * refactor-mmcls-package * fix print to logging * fix docstrings according to others comments * fix comments * fix comments * fix allentdans comment in pr215 * remove mmocr init * [Refactor] Refactor mmedit tests (open-mmlab#212) * feature mmedit * edit2.0 * edit * refactor mmedit * fix __init__.py * fix __init__ * fix formai * fix comment * fix comment * buff * edit test and code refactor * refactor dir * refactor tests/mmedit * fix docstring * add test coverage * fix lint * fix comment * fix comment * Update typehint (open-mmlab#216) * update type hint * update docstring * update * remove file * fix ppl * Refine get_predefined_partition_cfg * fix tensorrt version > 8 * move parse_cuda_device_id to device.py * Fix cascade * onnx2ncnn docstring Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: RunningLeon <maningsheng@sensetime.com> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com>

* introduction to model deployment * remove comments * trim trailing whitespace * add link

OpenMMLab-Assistant001 · 2023-04-04T10:36:35Z

Hi @wilxy ！First of all, we want to express our gratitude for your significant PR in the MMRazor project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.

We would also like to invite you to join our Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/UjgXkPWNqA

If you have WeChat account，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution！❤

Fix spelling mistakes

6ee68fe

wilxy requested review from pppppM and sunnyxiaohu July 26, 2022 13:32

sunnyxiaohu reviewed Jul 27, 2022

View reviewed changes

wilxy changed the title ~~Fix spelling mistakes~~ [Feature] Add connector components and FitNet Jul 27, 2022

1.Rename general connectors. 2.Replace nn.conv2 to build_conv_layer, …

1b17505

…replace nn.bn to build_norm_layer.

wilxy requested a review from sunnyxiaohu July 27, 2022 07:30

sunnyxiaohu reviewed Jul 27, 2022

View reviewed changes

wilxy added 2 commits July 27, 2022 18:48

1. Rename function init_parameters to init_weights in SingleConvConne…

f226c9e

…ctor to realize automatically invocation.

1. Add norm_cfg in config and general_connector

a8fb03e

1.Move calculate_student_loss to distillation algorithm. 2.Move mmraz…

cbcd8a3

…or.models.connector to mmrazor.models.architectures. 3.Merge stu_connectors and tea_connectors into connectors, and call connectors by their connector_name.

pppppM reviewed Jul 28, 2022

View reviewed changes

1.Replace connector_name to connector in record_info. 2.Add assert th…

d75d120

…at each connector must be in connectors.

pppppM approved these changes Jul 28, 2022

View reviewed changes

sunnyxiaohu approved these changes Jul 28, 2022

View reviewed changes

sunnyxiaohu merged commit 63d46b5 into open-mmlab:dev-1.x Jul 28, 2022

pppppM mentioned this pull request Jul 28, 2022

Revert "[Feature] Add connector components and FitNet" #211

Merged

humu789 pushed a commit to humu789/mmrazor that referenced this pull request Feb 13, 2023

Tutorial-01: introduction to model deployment (open-mmlab#207)

8adb41f

* introduction to model deployment * remove comments * trim trailing whitespace * add link

		import torch.nn as nn


		class BaseConnector(nn.Module, metaclass=ABCMeta):



		@MODELS.register_module()
		class ReLUConnector(BaseConnector):

Conversation

wilxy commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Features

Improvement

Uh oh!

codecov bot commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pppppM commented Jul 28, 2022

Uh oh!

pppppM left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OpenMMLab-Assistant001 commented Apr 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wilxy commented Jul 26, 2022 •

edited

Loading

codecov bot commented Jul 26, 2022 •

edited

Loading