[Feature] Add TIMM and HuggingFace wrappers to build classifiers from them directly. by mzr1996 · Pull Request #1102 · open-mmlab/mmpretrain

mzr1996 · 2022-10-18T10:50:57Z

Motivation

Pytorch-image-models and HuggingFace are famous model hubs, this PR can integrate their models in mmcls and use mmcls to train and inference models come from timm and huggingface directly.

Use cases

To train a model from TIMM

For example, to train a ResNet-50 model from timm with our vit-mae schedule.
Use config file as below

# configs/resnet/resnet50_timm.py
_base_ = ['../vision_transformer/vit-base-p16_pt-32xb128-mae_in1k-224.py']

model = dict(
    type='TimmClassifier',
    model_name='resnet50',
    loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
    train_cfg=dict(augments=[
        dict(type='Mixup', alpha=0.8),
        dict(type='CutMix', alpha=1.0)
    ]),
    _delete_=True)

To fine-tune a model from hugging-face

For example, to fine-tune a ResNet-50 model from hugging face on CIFAR dataset.
Use config file as below

# configs/resnet/resnet50_huggingface_cifar.py
_base_ = [
    '../_base_/datasets/cifar10_bs16.py',
    '../_base_/schedules/cifar10_bs128.py',
    '../_base_/default_runtime.py',
]

model = dict(
    type='HuggingFaceClassifier',
    model_name='microsoft/resnet-50',
    pretrained=True,
    loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
    num_labels=10,
    ignore_mismatched_sizes=True,
)

To inference a TIMM model

>>> from mmengine import Config
>>> from mmcls.apis import init_model, inference_model
>>> from mmcls.utils import register_all_modules
>>> from mmcls.datasets.categories import IMAGENET_CATEGORIES
>>> register_all_modules()
>>> # Use data pipeline from the resnet config
>>> cfg = Config.fromfile("./configs/resnet/resnet50_8xb32_in1k.py")
>>> cfg.model = dict(type='TimmClassifier', model_name='resnet50', pretrained=True)
>>> model = init_model(cfg)
>>> pred_label = inference_model(model, 'demo/demo.JPEG')['pred_label']
>>> print(IMAGENET_CATEGORIES[pred_label])
'sea snake'

To inference a HuggingFace model

>>> from mmengine import Config
>>> from mmcls.apis import init_model, inference_model
>>> from mmcls.utils import register_all_modules
>>> from mmcls.datasets.categories import IMAGENET_CATEGORIES
>>> register_all_modules()
>>> # Use data pipeline from the resnet config
>>> cfg = Config.fromfile("./configs/resnet/resnet50_8xb32_in1k.py")
>>> cfg.model = dict(type='HuggingFaceClassifier', model_name='microsoft/resnet-50', pretrained=True)
>>> model = init_model(cfg)
>>> pred_label = inference_model(model, 'demo/demo.JPEG')['pred_label']
>>> print(IMAGENET_CATEGORIES[pred_label])
'sea snake'

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

… them directly.

codecov · 2022-10-18T10:59:23Z

Codecov Report

Base: 0.02% // Head: 90.19% // Increases project coverage by +90.17% 🎉

Coverage data is based on head (617b741) compared to base (b8b31e9).
Patch has no changes to coverable lines.

Additional details and impacted files

@@             Coverage Diff              @@
##           dev-1.x    #1102       +/-   ##
============================================
+ Coverage     0.02%   90.19%   +90.17%     
============================================
  Files          121      140       +19     
  Lines         8217    10383     +2166     
  Branches      1368     1645      +277     
============================================
+ Hits             2     9365     +9363     
+ Misses        8215      798     -7417     
- Partials         0      220      +220

Flag	Coverage Δ
unittests	`90.19% <ø> (+90.17%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/apis/inference.py	`0.00% <0.00%> (ø)`
mmcls/datasets/transforms/compose.py
mmcls/models/necks/reduction.py	`100.00% <0.00%> (ø)`
mmcls/models/retrievers/__init__.py	`100.00% <0.00%> (ø)`
mmcls/models/backbones/mvit.py	`92.46% <0.00%> (ø)`
mmcls/models/backbones/edgenext.py	`95.20% <0.00%> (ø)`
mmcls/structures/utils.py	`77.77% <0.00%> (ø)`
mmcls/models/backbones/mobileone.py	`94.47% <0.00%> (ø)`
mmcls/models/backbones/swin_transformer_v2.py	`89.47% <0.00%> (ø)`
mmcls/models/heads/arcface_head.py	`85.48% <0.00%> (ø)`
... and 130 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

mmcls/models/classifiers/hugging_face.py

mmcls/models/classifiers/timm.py

Ezra-Yu

LGTM.

… them directly. (open-mmlab#1102) * [Feature] Add TIMM and HuggingFace wrappers to build classifiers from them directly. * Support `with_cp` and add docstring. * Add unit tests. * Update CI. * Update docs.

[Feature] Add TIMM and HuggingFace wrappers to build classifiers from…

2f5d1b7

… them directly.

mzr1996 requested review from Ezra-Yu and tonysy October 18, 2022 10:50

mzr1996 force-pushed the 1x-timm-hg branch from a7ec998 to 2f5d1b7 Compare October 19, 2022 01:12

okotaku reviewed Oct 19, 2022

View reviewed changes

mmcls/models/classifiers/hugging_face.py Show resolved Hide resolved

mmcls/models/classifiers/timm.py Show resolved Hide resolved

mzr1996 added 3 commits November 8, 2022 15:59

Support with_cp and add docstring.

3bb10f5

Add unit tests.

9632638

Update CI.

03e5ab8

mzr1996 requested a review from okotaku November 8, 2022 08:02

mzr1996 added 2 commits November 8, 2022 16:03

Merge remote-tracking branch 'origin/dev-1.x' into 1x-timm-hg

a842d7e

Update docs.

617b741

Ezra-Yu approved these changes Nov 9, 2022

View reviewed changes

mzr1996 merged commit 542143c into open-mmlab:dev-1.x Nov 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add TIMM and HuggingFace wrappers to build classifiers from them directly.#1102

[Feature] Add TIMM and HuggingFace wrappers to build classifiers from them directly.#1102
mzr1996 merged 6 commits intoopen-mmlab:dev-1.xfrom
mzr1996:1x-timm-hg

mzr1996 commented Oct 18, 2022

Uh oh!

codecov bot commented Oct 18, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Ezra-Yu left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mzr1996 commented Oct 18, 2022

Motivation

Use cases

To train a model from TIMM

To fine-tune a model from hugging-face

To inference a TIMM model

To inference a HuggingFace model

Checklist

Uh oh!

codecov bot commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Ezra-Yu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Oct 18, 2022 •

edited

Loading