[Feature] Support TIMMBackbone by shinya7y · Pull Request #7020 · open-mmlab/mmdetection

shinya7y · 2022-01-17T17:14:23Z

Motivation

timm (https://github.com/rwightman/pytorch-image-models) contains a lot of backbones, which are not available in this repository. It would be convenient to add examples to use timm backbones.

Related to the features of mmcls and mmseg.
open-mmlab/mmpretrain#427
open-mmlab/mmsegmentation#998

Modification

This PR adds example configs and a document to use timm backbones via the TIMMBackbone wrapper in mmcls.

Use cases (Optional)

Please see config files.

We can use ConvNeXt with timm>=0.5.4. However, other efforts are needed to support the settings of the official ConvNeXt (e.g., LearningRateDecayOptimizerConstructor). Related to open-mmlab/mmsegmentation#1216

based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998

codecov · 2022-01-17T17:32:05Z

Codecov Report

Merging #7020 (414c73a) into dev (4bdb312) will decrease coverage by 0.27%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev    #7020      +/-   ##
==========================================
- Coverage   62.41%   62.14%   -0.28%     
==========================================
  Files         330      330              
  Lines       26199    26018     -181     
  Branches     4436     4436              
==========================================
- Hits        16353    16168     -185     
- Misses       8976     8977       +1     
- Partials      870      873       +3

Flag	Coverage Δ
unittests	`62.12% <ø> (-0.28%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmdet/utils/misc.py	`95.23% <0.00%> (-4.77%)`	⬇️
mmdet/models/dense_heads/base_mask_head.py	`40.00% <0.00%> (-4.45%)`	⬇️
mmdet/models/seg_heads/base_semantic_head.py	`34.37% <0.00%> (-3.87%)`	⬇️
mmdet/models/roi_heads/base_roi_head.py	`76.47% <0.00%> (-3.53%)`	⬇️
mmdet/core/bbox/coder/yolo_bbox_coder.py	`50.00% <0.00%> (-2.95%)`	⬇️
mmdet/models/detectors/base.py	`50.78% <0.00%> (-2.56%)`	⬇️
...panoptic_fusion_heads/base_panoptic_fusion_head.py	`52.63% <0.00%> (-2.37%)`	⬇️
mmdet/datasets/pipelines/formatting.py	`65.78% <0.00%> (-1.98%)`	⬇️
mmdet/models/detectors/lad.py	`32.35% <0.00%> (-1.94%)`	⬇️
mmdet/core/anchor/point_generator.py	`52.08% <0.00%> (-1.92%)`	⬇️
... and 61 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bdb312...414c73a. Read the comment docs.

.circleci/config.yml

ZwwWayne · 2022-01-19T11:29:18Z

Can we directly use the TIMMBackbone in MMCls? I think we can directly use the config like

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

After we installed mmcls/timm

shinya7y · 2022-01-19T21:10:49Z

We need to update mmcls's TIMMBackbone for mmdet and mmseg because mmcls's TIMMBackbone does not support features_only=True.
After updating, the config will be as follows.

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        features_only=True,  # mmcls assumes False only, causing error
        pretrained=True,
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

# import to trigger register_module in mmcls
custom_imports = dict(imports=['mmcls.models'], allow_failed_imports=False)

# disable NumClassCheckHook
custom_hooks = []

ZwwWayne · 2022-01-20T05:13:58Z

We need to update mmcls's TIMMBackbone for mmdet and mmseg because mmcls's TIMMBackbone does not support features_only=True. After updating, the config will be as follows.

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        features_only=True,  # mmcls assumes False only, causing error
        pretrained=True,
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

# import to trigger register_module in mmcls
custom_imports = dict(imports=['mmcls.models'], allow_failed_imports=False)

# disable NumClassCheckHook
custom_hooks = []

Great! Thank you @shinya7y , would you like to directly create a PR in mmcls to support that? We can also create an issue there to let them fix this in this version.

ZwwWayne · 2022-01-25T08:50:53Z

Hi @shinya7y ,
It seems that mmcls already support that. We can add comments in the config to indicate that it requires the newest mmcls (should be v0.20.0) and clean the code. Hopefully this PR can be merged by this week as a feature in the next release of mmdet.

shinya7y · 2022-01-25T13:01:25Z

I will do them and put example configs in configs/timm_example like configs/albu_example.
If other directories (e.g., configs/timm_backbone, docs/*/tutorials) are preferred, please let me know.

ZwwWayne · 2022-01-25T13:47:43Z

I will do them and put example configs in configs/timm_example like configs/albu_example. If other directories (e.g., configs/timm_backbone, docs/*/tutorials) are preferred, please let me know.

Hi @shinya7y,
This idea is great! I think we can do that~

hhaAndroid · 2022-02-08T07:15:55Z

@shinya7y Great. LGTM.

hhaAndroid · 2022-02-08T07:29:37Z

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

shinya7y · 2022-02-08T08:48:57Z

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

It will be difficult. Related discussion: open-mmlab/mmpretrain#605

hhaAndroid · 2022-02-11T06:58:17Z

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

It will be difficult. Related discussion: open-mmlab/mmclassification#605

OK.

RangiLyu

LGTM

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

* [Enhancement] Upgrade isort in pre-commit hook (#7130) * upgrade isort to v5.10.1 * replace known_standard_library with extra_standard_library * upgrade isort to v5.10.1 replace known_standard_library with extra_standard_library * imports order changes * [Fix] cannot to save the best checkpoint when the key_score is None (#7101) * [Fix] Fix MixUp transform filter boxes failing case. Added test case (#7080) * [Fix] Update the version limitation of mmcv-full and pytorch in CI. (#7133) * Update * Update build.yml * Update build.yml * [Feature] Support TIMMBackbone (#7020) * add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs * Create 2_new_data_model.md (#6476) fix some typo Co-authored-by: PJLAB\huanghaian <1286304229@qq.com> * [FIX] add Ci of pytorch 1.10 and comments for bbox clamp (#7081) (#7083) * add comments for bbox clamp * add CI of pytorch1.10 * add ci of pytorch1.10.1 * mmcv1.9.0->mmcv1.9 * add ci of pytorch1.10 * Add daily issue owners (#7163) * Add code owners Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com> * Update code owners Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com> * [Feature] Support visualization for Panoptic Segmentation (#7041) * First commit of v2 * split the functions * Support to show panoptic result * temp * Support to show gt * support show gt * fix lint * Support to browse datasets * Fix unit tests * Fix findContours * fix comments * Fix pre-commit * fix lint * Add the type of an argument * [Fix] confusion_matrix.py analysis tool handling NaNs (#7147) * [Fix] Added missing property in SABLHead (#7091) * Added missing property in SABLHead * set pre-commit-hooks to v0.1.0 * set maskdownlint to v0.11.0 * pre-commit-hooks Co-authored-by: Cedric Luo <luochunhua1996@outlook.com> * Update config.md (#7215) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Update config.md fix some typos Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> * [Feature] Support DyHead (#6823) * add DyHead * move and update DYReLU * update * replace stack with sum to reduce memory * clean and update * update to align inference accuracy (incomplete) * fix pad * update to align training accuracy and pick #6867 * add README and metafile * update docs * resolve comments * revert picking 6867 * update README.md * update metafile.yml * resolve comments and update urls * Fix broken colab link (#7218) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Fix broken colab link Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> * Remove the inplace addition in `FPN` (#7175) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Remove the inplace addition in `FPN` * update Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> Co-authored-by: PJLAB\huanghaian <1286304229@qq.com> * [Feature] Support OpenImages Dataset (#6331) * [Feature] support openimage group of eval * [Feature] support openimage group of eval * support openimage dataset * support openimage challenge dataset * fully support OpenImages-V6 and OpenImages Challenge 2019 * Fix some logic error * update config file * fix get data_infos error * fully support OpenImages evaluation * update OpenImages config files * [Feature] support OpenImages datasets * fix bug * support load image metas from pipeline * fix bug * fix get classes logic error * update code * support get image metas * support openimags * support collect image metas * support Open Images * fix openimages logic * minor fix * add a new function to compute openimages tpfp * minor fix * fix ci error * minor fix * fix indication * minor fix * fix returns * fix returns * fix returns * fix returns * fix returns * minor fix * update readme * support loading image level labels and fix some logic * minor fix * minor fix * add class names * minor fix * minor fix * minor fix * add openimages test unit * minor fix * minor fix * fix test unit * minor fix * fix logic error * minor fix * fully support openimages * minor fix * fix docstring * fix docstrings in readthedocs * update get image metas script * label_description_file -> label_file * update openimages readme * fix test unit * fix test unit * minor fix * update readme file * Update get_image_metas.py * [Enhance] Speed up SimOTA matching. (#7098) * [Feature] Add Maskformer to mmdet (#7212) * first commit * add README * move model description from config to readme add description for binary_input add description for dice loss add a independent panoptic gt processing function add a independent panoptic gt processing function remove compatibility of pretrain in maskformer * update comments in maskformer_head * update docs format * Add deprecation message for deploy tools (#7242) * Add CI for windows (#7228) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Add mmrotate (#7216) * fix description for args in CSPDarknet (#7187) * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * fix test_find_latest_checkpoint * fix data_infos__default_db_directories * fix test_custom_classes_override_default * Update test_custom_dataset.py * Update test_common.py * Update assign_result.py * use os.path.join * fix bug * Update test_common.py * Update assign_result.py * Update sampling_result.py * os.path -> osp * os.path -> osp Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> Co-authored-by: Hyeokjoon Kwon <dsazz801@gmail.com> * add Chinese version of init_cfg (#7188) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Create init_cfg.md * Update docs/zh_cn/tutorials/init_cfg.md Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com> * update init_cfg.md * update init_cfg.md * update init_cfg.md * update init_cfg.md Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com> * update MaskFormer readme and docs (#7241) * update docs for maskformer * update readme * update readme format * update link * update json link * update format of ConfigDict * update format of function returns * uncomment main in deployment/test.py * [Feature] ResNet Strikes Back. (#7001) * [Feature] ResNet Strikes Back. * add more cfg * add readme * update * update * update * update * update * update * Maskformer Visualization (#7247) * maskformer visualization * compatible with models that do not contain arg of pretrained * compatible with models that do not contain arg of pretrained * Bump versions to v2.22.0 (#7240) * Bump versions to v2.22.0 * Fix comments and add the latest PRs * fix the id of contributor * relax the version of mmcv * Add ResNet Strikes Back * Update README_zh-CN.md * Update README.md * fix typo * Update README_zh-CN.md Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com> * Maskformer metafile and rsb readme format (#7250) * [Fix] Fix Open Images testunit to avoid error in Windows CI (#7252) * [Feature] support openimage group of eval * [Feature] support openimage group of eval * support openimage dataset * support openimage challenge dataset * fully support OpenImages-V6 and OpenImages Challenge 2019 * Fix some logic error * update config file * fix get data_infos error * fully support OpenImages evaluation * update OpenImages config files * [Feature] support OpenImages datasets * fix bug * support load image metas from pipeline * fix bug * fix get classes logic error * update code * support get image metas * support openimags * support collect image metas * support Open Images * fix openimages logic * minor fix * add a new function to compute openimages tpfp * minor fix * fix ci error * minor fix * fix indication * minor fix * fix returns * fix returns * fix returns * fix returns * fix returns * minor fix * update readme * support loading image level labels and fix some logic * minor fix * minor fix * add class names * minor fix * minor fix * minor fix * add openimages test unit * minor fix * minor fix * fix test unit * minor fix * fix logic error * minor fix * fully support openimages * minor fix * fix docstring * fix docstrings in readthedocs * update get image metas script * label_description_file -> label_file * update openimages readme * fix test unit * fix test unit * minor fix * update readme file * Update get_image_metas.py * fix oid testunit to avoid some error in windows * minor fix to avoid some error in windows * minor fix * add comments in oid test unit * minor fix Co-authored-by: Cedric Luo <luochunhua1996@outlook.com> Co-authored-by: LuooChen <33743370+LuooChen@users.noreply.github.com> Co-authored-by: Daniel van Sabben Alsina <13849741+dvansa@users.noreply.github.com> Co-authored-by: jbwang1997 <jbwang1997@gmail.com> Co-authored-by: Yosuke Shinya <42844407+shinya7y@users.noreply.github.com> Co-authored-by: siatwangmin <csustwangmin@163.com> Co-authored-by: PJLAB\huanghaian <1286304229@qq.com> Co-authored-by: del-zhenwu <dele.zhenwu@gmail.com> Co-authored-by: Guangchen Lin <347630870@qq.com> Co-authored-by: VIKASH RANJAN <ranjanvikash26@gmail.com> Co-authored-by: Range King <RangeKingHZ@gmail.com> Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com> Co-authored-by: BigDong <yudongwang@tju.edu.cn> Co-authored-by: Haofan Wang <frankmiracle@outlook.com> Co-authored-by: Zhijian Liu <zhijianliu.cs@gmail.com> Co-authored-by: BigDong <yudongwang1226@gmail.com> Co-authored-by: RangiLyu <lyuchqi@gmail.com> Co-authored-by: Yue Zhou <592267829@qq.com> Co-authored-by: Hyeokjoon Kwon <dsazz801@gmail.com> Co-authored-by: Kevin Ye <1752391457@qq.com>

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

shinya7y added 3 commits January 16, 2022 23:07

add TIMMBackbone

4450e11

based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998

update and clean

f982211

fix unit test

0503dd4

RangiLyu requested review from RangiLyu and hhaAndroid January 18, 2022 05:52

RangiLyu added the enhancement New feature or request label Jan 18, 2022

RangiLyu reviewed Jan 18, 2022

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

ZwwWayne mentioned this pull request Jan 20, 2022

[Feature] Support features_only in TIMMBackbone open-mmlab/mmpretrain#665

Closed

shinya7y mentioned this pull request Jan 23, 2022

[Feature] Support features_only in TIMMBackbone open-mmlab/mmpretrain#668

Merged

10 tasks

shinya7y added 3 commits January 26, 2022 03:01

Revert

045f781

add example configs

cf0988f

Merge remote-tracking branch 'upstream/dev'

414c73a

hhaAndroid approved these changes Feb 11, 2022

View reviewed changes

RangiLyu approved these changes Feb 11, 2022

View reviewed changes

ZwwWayne approved these changes Feb 11, 2022

View reviewed changes

ZwwWayne merged commit ffff556 into open-mmlab:dev Feb 11, 2022

AronLin mentioned this pull request Feb 24, 2022

Bump versions to v2.22.0 #7240

Merged

ZwwWayne pushed a commit that referenced this pull request Jul 18, 2022

[Feature] Support TIMMBackbone (#7020)

19a9904

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support TIMMBackbone#7020

[Feature] Support TIMMBackbone#7020
ZwwWayne merged 6 commits intoopen-mmlab:devfrom
shinya7y:feat_timm_backbone

shinya7y commented Jan 17, 2022 •

edited

Loading

Uh oh!

codecov bot commented Jan 17, 2022 •

edited

Loading

Uh oh!

Uh oh!

ZwwWayne commented Jan 19, 2022

Uh oh!

shinya7y commented Jan 19, 2022

Uh oh!

ZwwWayne commented Jan 20, 2022

Uh oh!

ZwwWayne commented Jan 25, 2022

Uh oh!

shinya7y commented Jan 25, 2022

Uh oh!

ZwwWayne commented Jan 25, 2022

Uh oh!

hhaAndroid commented Feb 8, 2022

Uh oh!

hhaAndroid commented Feb 8, 2022

Uh oh!

shinya7y commented Feb 8, 2022

Uh oh!

hhaAndroid commented Feb 11, 2022

Uh oh!

RangiLyu left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shinya7y commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Use cases (Optional)

Uh oh!

codecov bot commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ZwwWayne commented Jan 19, 2022

Uh oh!

shinya7y commented Jan 19, 2022

Uh oh!

ZwwWayne commented Jan 20, 2022

Uh oh!

ZwwWayne commented Jan 25, 2022

Uh oh!

shinya7y commented Jan 25, 2022

Uh oh!

ZwwWayne commented Jan 25, 2022

Uh oh!

hhaAndroid commented Feb 8, 2022

Uh oh!

hhaAndroid commented Feb 8, 2022

Uh oh!

shinya7y commented Feb 8, 2022

Uh oh!

hhaAndroid commented Feb 11, 2022

Uh oh!

RangiLyu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shinya7y commented Jan 17, 2022 •

edited

Loading

codecov bot commented Jan 17, 2022 •

edited

Loading