[Feature] Dedicated MMClsWandbHook for MMClassification (Weights and Biases Integration) by ayulockin · Pull Request #764 · open-mmlab/mmpretrain

ayulockin · 2022-04-05T15:36:49Z

I have raised a PR in MMDetection contributing a dedicated MMDetWandbHook for MMDetection. I was playing around with MMClassification and realized that MMClassification can use a similar dedicated hook (with few minor modifications).

Motivation

The goal of this PR is to contribute a dedicated Weights and Biases hook for MMClassification.

Modification

The PR adds two new files:

wandblogger_hook.py where all the Weights and Biases related logic lives and,
eval_hooks.py was added so that the MMClsWandbHook can reuse the validation results.

The feature can easily be used this:

log_config = dict(
            interval=10,
            hooks=[
                dict(type='MMClsWandbHook',
                     wandb_init_kwargs={
                         'entity': WANDB_ENTITY,
                         'project': WANDB_PROJECT_NAME
                     },
                     log_checkpoint=True,
                     log_checkpoint_metadata=True,
                     num_eval_images=100)
            ])

Use cases (Optional)

Here are some of the use cases that this PR introduces and should be helpful to the community in general.

Metrics

The WandbLogger will automatically log training and validation metrics.
It will log system (CPU/GPU) metrics.

Screen.Recording.2022-04-05.at.7.55.19.PM.mov

Checkpointing with Metadata

If log_checkpoint is True, the checkpoint saved at every checkpoint interval will be saved as W&B Artifacts. On top of this, if log_checkpoint_metadata is True, every checkpoint artifact will have metadata associated with it as shown in the recording below.

Screen.Recording.2022-04-05.at.7.58.24.PM.mov

Log Model Prediction 🎉

If num_eval_images > 0, at every evaluation interval, theMMClsWandbHook logs the model prediction as interactive W&B Tables. To know more about W&B Tables, please refer to the docs here. The 'MMClsWandbHook' logs the predicted class labels along with ground truth labels.

Screen.Recording.2022-04-05.at.8.03.59.PM.mov

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

CLAassistant · 2022-04-05T15:36:54Z

All committers have signed the CLA.

codecov · 2022-04-05T15:53:38Z

Codecov Report

Merging #764 (1f26f10) into dev (7c5ddb1) will decrease coverage by 1.34%.
The diff coverage is 22.09%.

@@            Coverage Diff             @@
##              dev     #764      +/-   ##
==========================================
- Coverage   86.80%   85.46%   -1.35%     
==========================================
  Files         130      132       +2     
  Lines        8571     8750     +179     
  Branches     1478     1513      +35     
==========================================
+ Hits         7440     7478      +38     
- Misses        909     1050     +141     
  Partials      222      222

Flag	Coverage Δ
unittests	`85.38% <22.09%> (-1.34%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/core/hook/wandblogger_hook.py	`18.57% <18.57%> (ø)`
mmcls/core/evaluation/eval_hooks.py	`28.94% <28.94%> (ø)`
mmcls/apis/train.py	`14.94% <100.00%> (-0.97%)`	⬇️
mmcls/core/evaluation/__init__.py	`100.00% <100.00%> (ø)`
mmcls/core/hook/__init__.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c5ddb1...1f26f10. Read the comment docs.

mzr1996 · 2022-04-06T02:42:40Z

Looks good, but I haven't used Wandb, I will try it recently. 👍

ayulockin · 2022-04-06T08:08:09Z

Looks good, but I haven't used Wandb, I will try it recently. 👍

Looking forward to your feedback.

ayulockin · 2022-04-11T14:23:45Z

Hey @mzr1996, did you get a chance to play with W&B. Would love your feedback on this work?

ayulockin · 2022-05-03T10:46:29Z

Hey @mzr1996, sorry to bug you again but would love it if you can give some feedback on the hook that I have implemented. I think it will add value to the community. :)

mzr1996 · 2022-05-05T02:25:40Z

Hey @mzr1996, sorry to bug you again but would love it if you can give some feedback on the hook that I have implemented. I think it will add value to the community. :)

Sorry for the late reply. This function is planned to be merged in the version v0.23.1, and I'll review it today.

ayulockin · 2022-05-05T05:51:19Z

Thanks for the reply. Looking forward to the review.

mzr1996 · 2022-05-05T15:35:24Z

Hello, I have reviewed your PR and made some modifications. You can check the difference here
If you think my PR works well, please merge that PR, and then this PR will be updated.

ayulockin · 2022-05-05T15:45:40Z

Thanks for looking into it in such detail. I will test out the PR and let you know. :)

* [Enhance] Add extra dataloader settings in configs. (open-mmlab#752) * Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings in the `data` field to specify different arguments. * Fix bug * Fix bug * [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762) * [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749) * [Feature]: Add resize rel pos embed * [Refactor]: Create a separated resize_rel_pos_bias_table func * [Refactor]: Refactor rel pos embed bias * [Refactor]: Move interpolate into func * Remove index buffer only when window_size changes Co-authored-by: mzr1996 <mzr1996@163.com> * [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746) * add PoolFormer * fix some typos in PoolFormer * fix lint error * modify out_indices and gap * fix typo * fix lint * fix typo * fix typo in poolforemr README * fix lint * Update some paths * Refactor freeze_stages method * Add unit tests * Fix lint Co-authored-by: mzr1996 <mzr1996@163.com> * Bump version to v0.22.1 (open-mmlab#785) * [Docs] Refine API reference. (open-mmlab#774) * [Docs] Refine API reference * Add PoolFormer * [Docs] Fix docs. * [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759) * [Feature] Support VAN. (open-mmlab#739) * add van * fix config * add metafile * add test * model convert script * fix review * fix lint * fix the configs and improve docs * rm debug lines * add VAN into api Co-authored-by: Yu Zhaohui <1105212286@qq.com> * [Feature] Support DenseNet. (open-mmlab#750) * init add densenet implementation * Add config and converted models * update meta * add test for memory efficient * Add docs * add doc for jit * Update checkpoint path * Update readthedocs Co-authored-by: mzr1996 <mzr1996@163.com> * [Fix] Use symbolic link in the API reference of Chinese docs. * [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai> * [Fix] Fix lint and mmcv version requirement for IPU. * Bump version to v0.23.0 (open-mmlab#809) * Refacoter Wandb hook and refine docstring Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai>

ayulockin · 2022-05-06T17:48:10Z

Hey @mzr1996 I have tested the changes and made a few of my own (they are minor). Can you please take a look and let me know if anything else is needed. Thank you again for a really helpful PR.

ayulockin · 2022-05-15T15:32:52Z

Hey @mzr1996, would love to get a confirmation about the changes made to the PR after your feedback. Also can you let me know what else is required from my end? Thanks. :)

ayulockin · 2022-05-23T19:54:15Z

Hey @mzr1996, apologises for the ping again. But I would like to know if anything else is needed for this PR?

mzr1996

Looks good

…Biases Integration) (open-mmlab#764) * wandb integration * visualize using wandb tables * wandb tables enhanced * Refactor MMClsWandbHook (open-mmlab#1) * [Enhance] Add extra dataloader settings in configs. (open-mmlab#752) * Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings in the `data` field to specify different arguments. * Fix bug * Fix bug * [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762) * [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749) * [Feature]: Add resize rel pos embed * [Refactor]: Create a separated resize_rel_pos_bias_table func * [Refactor]: Refactor rel pos embed bias * [Refactor]: Move interpolate into func * Remove index buffer only when window_size changes Co-authored-by: mzr1996 <mzr1996@163.com> * [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746) * add PoolFormer * fix some typos in PoolFormer * fix lint error * modify out_indices and gap * fix typo * fix lint * fix typo * fix typo in poolforemr README * fix lint * Update some paths * Refactor freeze_stages method * Add unit tests * Fix lint Co-authored-by: mzr1996 <mzr1996@163.com> * Bump version to v0.22.1 (open-mmlab#785) * [Docs] Refine API reference. (open-mmlab#774) * [Docs] Refine API reference * Add PoolFormer * [Docs] Fix docs. * [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759) * [Feature] Support VAN. (open-mmlab#739) * add van * fix config * add metafile * add test * model convert script * fix review * fix lint * fix the configs and improve docs * rm debug lines * add VAN into api Co-authored-by: Yu Zhaohui <1105212286@qq.com> * [Feature] Support DenseNet. (open-mmlab#750) * init add densenet implementation * Add config and converted models * update meta * add test for memory efficient * Add docs * add doc for jit * Update checkpoint path * Update readthedocs Co-authored-by: mzr1996 <mzr1996@163.com> * [Fix] Use symbolic link in the API reference of Chinese docs. * [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai> * [Fix] Fix lint and mmcv version requirement for IPU. * Bump version to v0.23.0 (open-mmlab#809) * Refacoter Wandb hook and refine docstring Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai> * shuffle val data * minor updates * minor fix Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai>

lsch0lz · 2023-12-12T18:20:49Z

Thanks for implementing this hook! I just tried it out, but can't quite get it to run. How do I log into my WandDB Account or do I have to specify somewhere an API Key to log the informations into my WanDB Project? Training works as usual, but I cant see the logs in the specified WanDB Project.

My config file looks like this:

_base_ = [
    '../_base_/models/swin_transformer/tiny_224.py',
    '../_base_/datasets/imagenet_bs64_swin_224.py',
    '../_base_/schedules/imagenet_bs1024_adamw_swin.py',
    '../_base_/default_runtime.py'
]

# schedule settings
optim_wrapper = dict(clip_grad=dict(max_norm=5.0))

# Dataset
data_root = 'XXX'
train_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='train',
    ))
val_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='val',
    ))
test_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='test',
    ))

# Logging
log_config = dict(
            interval=10,
            hooks=[
                dict(type='MMClsWandbHook',
                     wandb_init_kwargs={
                         'entity': "MY_USER_NAME",
                         'project': "MY_PROJECT_NAME"
                     },
                     log_checkpoint=True,
                     log_checkpoint_metadata=True,
                     num_eval_images=100)
            ])

ayulockin added 3 commits April 4, 2022 17:35

wandb integration

af6b60e

visualize using wandb tables

0f25068

wandb tables enhanced

81fade1

Ezra-Yu changed the base branch from master to dev April 5, 2022 15:41

Ezra-Yu changed the title ~~Dedicated MMClsWandbHook for MMClassification (Weights and Biases Integration)~~ [Feature] Dedicated MMClsWandbHook for MMClassification (Weights and Biases Integration) Apr 5, 2022

mzr1996 and others added 4 commits May 6, 2022 15:23

Merge branch 'dev' into master

4a26dc7

shuffle val data

7f48aa3

minor updates

16a59b5

minor fix

1f26f10

mzr1996 approved these changes Jun 2, 2022

View reviewed changes

mzr1996 merged commit ccdbc82 into open-mmlab:dev Jun 2, 2022

Conversation

ayulockin commented Apr 5, 2022

Motivation

Modification

Use cases (Optional)

Metrics

Checkpointing with Metadata

Log Model Prediction 🎉

Checklist

Uh oh!

CLAassistant commented Apr 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mzr1996 commented Apr 6, 2022

Uh oh!

ayulockin commented Apr 6, 2022

Uh oh!

ayulockin commented Apr 11, 2022

Uh oh!

ayulockin commented May 3, 2022

Uh oh!

mzr1996 commented May 5, 2022

Uh oh!

ayulockin commented May 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mzr1996 commented May 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayulockin commented May 5, 2022

Uh oh!

ayulockin commented May 6, 2022

Uh oh!

ayulockin commented May 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayulockin commented May 23, 2022

Uh oh!

mzr1996 left a comment

Choose a reason for hiding this comment

Uh oh!

lsch0lz commented Dec 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Apr 5, 2022 •

edited

Loading

codecov bot commented Apr 5, 2022 •

edited

Loading

ayulockin commented May 5, 2022 •

edited

Loading

mzr1996 commented May 5, 2022 •

edited

Loading

ayulockin commented May 15, 2022 •

edited

Loading