Dedicated WandbLogger for MMDetection by ayulockin · Pull Request #7413 · open-mmlab/mmdetection

ayulockin · 2022-03-15T21:09:05Z

As per the discussion in this issue #7391, I have opened this PR to introduce a dedicated WandbLogger for MMDetection.

Motivation

The goal and motivation for this PR are mentioned in detail in this #7391 issue.

Modification

The PR adds a new file wandblogger_hook.py where all the Weights and Biases related logic lives. The features that are introduced are explained in detail in my initial issue.

Other than this file, there is a small modification to the eval_hook.py file. It was made so that the WandbLogger can access the evaluation results.

I am really looking forward to all the feedback. I would love to answer questions if any. If you require a colab to test out the feature, I can provide that as well. However, using it is straightforward as shown in the code snippet below.

log_config = dict(
            interval=10,
            hooks=[
                dict(type='WandbLogger',
                     wandb_init_kwargs={
                         'entity': WANDB_ENTITY,
                         'project': WANDB_PROJECT_NAME
                     },
                     logging_interval=10,
                     log_checkpoint=True,
                     log_checkpoint_metadata=True,
                     num_eval_images=100)
            ])

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

CLAassistant · 2022-03-15T21:09:11Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

ayulockin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2022-03-16T02:06:32Z

Codecov Report

Merging #7413 (7bbb876) into master (01b55b2) will decrease coverage by 0.33%.
The diff coverage is 15.81%.

❗ Current head 7bbb876 differs from pull request most recent head 532b860. Consider uploading reports for the commit 532b860 to get more accurate results

@@            Coverage Diff             @@
##           master    #7413      +/-   ##
==========================================
- Coverage   63.45%   63.12%   -0.34%     
==========================================
  Files         340      341       +1     
  Lines       27493    27687     +194     
  Branches     4645     4688      +43     
==========================================
+ Hits        17446    17477      +31     
- Misses       9119     9284     +165     
+ Partials      928      926       -2

Flag	Coverage Δ
unittests	`63.09% <15.81%> (-0.34%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmdet/core/hook/wandblogger_hook.py	`14.50% <14.50%> (ø)`
mmdet/core/evaluation/eval_hooks.py	`81.81% <100.00%> (ø)`
mmdet/core/hook/__init__.py	`100.00% <100.00%> (ø)`
mmdet/core/bbox/assigners/max_iou_assigner.py	`73.68% <0.00%> (+1.31%)`	⬆️
mmdet/utils/misc.py	`100.00% <0.00%> (+4.76%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2733aa9...532b860. Read the comment docs.

mmdet/core/evaluation/eval_hooks.py

mmdet/core/hook/wandblogger_hook.py

jbwang1997 · 2022-03-16T02:20:14Z

Hellow @ayulockin. Thank you for your contributions. I have left some comments above.

ZwwWayne · 2022-03-16T02:24:56Z

Hi @ayulockin ,
Could you sign the CLA so that we can merge this PR?

mmdet/core/hook/wandblogger_hook.py

ayulockin · 2022-03-16T23:54:06Z

Hi @ayulockin ,
Could you sign the CLA so that we can merge this PR?

Hey @ZwwWayne, even though I am signing the CLA it's not showing up here. Do you have any clue as to why this is the case?

mmdet/core/hook/wandblogger_hook.py

ayulockin · 2022-03-17T00:01:07Z

mmdet/core/hook/wandblogger_hook.py

+
+    @master_only
+    def after_run(self, runner):
+        self.wandb.finish()


As you can see I have overwritten the after_run method of mmcv.WandbLoggerHook. Do you want me to use super() instead?

The current mmcv.WandbLoggerHook logs the files with output_prefix as artifacts, however, this dedicated logger makes use of the artifacts better. If you think, it's best to not override, I will revert the change.

ayulockin · 2022-03-17T18:10:07Z

Hey @jbwang1997 @ZwwWayne, I am not able to sign the CLA probably because, the email id that was configured with git for few of the initial commits is not linked to my GitHub account (there might be a typo).

If it's not an issue, can I make a new PR from a new branch so that I can sign the CLA. Ideally, would love to get the review process done and then we can close this PR and I can open a new PR for merger.

Apologies for inconvenience.

mmdet/core/hook/wandblogger_hook.py

jbwang1997 · 2022-03-18T03:16:22Z

Hey @jbwang1997 @ZwwWayne, I am not able to sign the CLA probably because, the email id that was configured with git for few of the initial commits is not linked to my GitHub account (there might be a typo).

If it's not an issue, can I make a new PR from a new branch so that I can sign the CLA. Ideally, would love to get the review process done and then we can close this PR and I can open a new PR for merger.

Apologies for inconvenience.

You are welcome to open a new pr. This is awesome work and we can continue to discuss this function in the new pr.

hhaAndroid · 2022-03-18T07:35:45Z

@ayulockin Very meaningful work！Thank you very much for your contribution.

jbwang1997 · 2022-03-18T09:33:44Z

mmdet/core/hook/wandblogger_hook.py

+        super(WandbLogger, self).before_run(runner)
+        self.cfg = self.wandb.config
+        # Check if configuration is passed to wandb.
+        if len(dict(self.cfg)) == 0:
+            warnings.warn(
+                'To log mmdetection Config, '
+                'pass it to init_kwargs of WandbLogger.', UserWarning)


The config in mmdetection is too huge for users to be added by init_kwargs. Is it possible to add config from the file?

By adding config from file, do you mean log something like this - here's an example command line:

python tools/train.py configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py

Do you want to log the config from configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py? It's doable in my opinion.

On second thought, here's another solution for logging config easily, however it will require a PR in the MMCV repo.

mmdet/core/hook/wandblogger_hook.py

jbwang1997 · 2022-03-18T10:27:19Z

@ayulockin Thank you so much for your awesome contributions again. I am now running some samples with this function. After running, I will give you more detailed feedback on this function. I left some small comments above. You can fix them and open a new pr.

ayulockin · 2022-03-18T16:30:39Z

Closing this PR in favor of #7459.

ayulockin and others added 11 commits March 9, 2022 10:18

evaluation table

6f26805

better eval table + smart checkpointing

b220ce7

memory deduping

60174bb

refactor+eval optimization

dfdfa84

priority checking+eval metric+silent eval logging

6fdf4c9

minor changes

24a7f54

define metric for val metrics

051d378

better eval metric handling

7d6985b

docs string

080077c

pass to continue

5309584

num of images logegd+polishing

3b7b3aa

mm-assistant bot added the size/XS label Mar 15, 2022

jbwang1997 self-requested a review March 16, 2022 01:02

jbwang1997 reviewed Mar 16, 2022

View reviewed changes

mmdet/core/evaluation/eval_hooks.py Show resolved Hide resolved

jbwang1997 reviewed Mar 16, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 16, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Outdated Show resolved Hide resolved

kwargs

7bbb876

jbwang1997 reviewed Mar 16, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Outdated Show resolved Hide resolved

jbwang1997 reviewed Mar 16, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Outdated Show resolved Hide resolved

multi gpus+better checkings

532b860

ayulockin commented Mar 16, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Outdated Show resolved Hide resolved

ayulockin commented Mar 17, 2022

View reviewed changes

better image+bbox handling

9384b49

ayulockin commented Mar 17, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

ayulockin commented Mar 17, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

jbwang1997 reviewed Mar 18, 2022

View reviewed changes

mmdet/core/hook/wandblogger_hook.py Show resolved Hide resolved

ayulockin mentioned this pull request Mar 18, 2022

[NEW PR] Dedicated WandbLogger for MMDetection #7459

Merged

ayulockin closed this Mar 18, 2022

Conversation

ayulockin commented Mar 15, 2022

Motivation

Modification

Checklist

Uh oh!

CLAassistant commented Mar 15, 2022

Uh oh!

codecov bot commented Mar 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbwang1997 commented Mar 16, 2022

Uh oh!

ZwwWayne commented Mar 16, 2022

Uh oh!

Uh oh!

Uh oh!

ayulockin commented Mar 16, 2022

Uh oh!

Uh oh!

ayulockin Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

ayulockin commented Mar 17, 2022

Uh oh!

Uh oh!

Uh oh!

jbwang1997 commented Mar 18, 2022

Uh oh!

hhaAndroid commented Mar 18, 2022

Uh oh!

jbwang1997 Mar 18, 2022

Choose a reason for hiding this comment

Uh oh!

ayulockin Mar 18, 2022

Choose a reason for hiding this comment

Uh oh!

ayulockin Mar 18, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbwang1997 commented Mar 18, 2022

Uh oh!

ayulockin commented Mar 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Mar 16, 2022 •

edited

Loading