[Feature] Support Segmenter by rstrudel · Pull Request #952 · open-mmlab/mmsegmentation

rstrudel · 2021-10-11T09:42:32Z

Motivation

Add Segmenter method to mmsegmentation.

Modification

I added configuration files to train segmenter on ADE20K and Cityscapes. I also added a script to convert the original ViT checkpoints in JAX to checkpoints compatible with the ViT class of mmsegmentation.

To be done

What's not there yet:

use img_norm_cfg=[127.5, 127.5, 127.5] as default for ViT checkpoints
checkpoints, I reported the performances in the readme for the ones I trained

CLAassistant · 2021-10-11T09:42:35Z

All committers have signed the CLA.

mmseg/models/decode_heads/__init__.py

Junjun2016 · 2021-10-11T13:55:42Z

Hi @rstrudel
Nice PR!
Please fix the CI error and we will review it ASAP.

Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

rstrudel · 2021-10-11T15:31:39Z

Hi @Junjun2016 ,
The CI error seems to come from the fact that I am using einops. I thought https://github.com/open-mmlab/mmsegmentation/pull/952/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R11 would fix it and include einops as a dependency but it does not seem to be the case. Could you please tell me how can I add a dependency?

einops is quite useful, especially for weights conversion for example:
https://github.com/open-mmlab/mmsegmentation/pull/952/files#diff-c3a364134054b115f5191ac5679e696ff049bab602857b3d7fd29904e108d5c3R1-R116

Junjun2016 · 2021-10-12T05:21:43Z

Hi @Junjun2016 , The CI error seems to come from the fact that I am using einops. I thought https://github.com/open-mmlab/mmsegmentation/pull/952/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R11 would fix it and include einops as a dependency but it does not seem to be the case. Could you please tell me how can I add a dependency?

einops is quite useful, especially for weights conversion for example: https://github.com/open-mmlab/mmsegmentation/pull/952/files#diff-c3a364134054b115f5191ac5679e696ff049bab602857b3d7fd29904e108d5c3R1-R116

Our principle is to use as few third-party dependencies as possible.
Maybe we can use einops in PyTorch.

Junjun2016 · 2021-10-12T05:30:41Z

configs/_base_/models/segmenter_vit-b_linear.py

+        loss_decode=dict(
+            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
+    ),
+    test_cfg=dict(mode='slide', crop_size=(512, 512), stride=(512, 512)),


It seems that the sliding window has no overlap.
Is it the same with your paper setting?

Junjun2016 · 2021-10-12T05:31:25Z

configs/_base_/models/segmenter_vit-b_mask.py

+        loss_decode=dict(
+            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
+    ),
+    test_cfg=dict(mode='slide', crop_size=(512, 512), stride=(512, 512)),


It seems that the sliding window has no overlap.
Is it the same with your paper setting?

Junjun2016 · 2021-10-12T05:40:49Z

configs/_base_/models/segmenter_vit-s_linear.py

@@ -0,0 +1,34 @@
+# model settings


We should not put too many configs in _base_, just put one base config.
And inherit the base config to generate different configs with different settings (model, dataset, or setting ) in configs/segmenter by overriding the keys with different values.

I moved the files to configs/segmenter in https://github.com/rstrudel/mmsegmentation/commit/58f7bece4e7a63837fa453147a3c91724ed2f2a5

Junjun2016 · 2021-10-12T05:45:10Z

configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py

@@ -0,0 +1,20 @@
+_base_ = [


Rename configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py to configs/segmenter/segmenter_vit-b_linear_8x1_512x512_160k_ade20k.py
Refer to https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv1.

Update other configs according to this config's comments.

Addressed in 9deb2dc

Junjun2016 · 2021-10-12T05:47:00Z

configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py

+    '../_base_/default_runtime.py',
+    '../_base_/schedules/schedule_160k.py',
+]
+find_unused_parameters = True


Suggested change

find_unused_parameters = True

Remove find_unused_parameters.

Junjun2016 · 2021-10-12T05:48:40Z

configs/_base_/models/segmenter_vit-b_linear.py

+        type='SegmenterLinearHead',
+        in_channels=768,
+        channels=768,
+        num_classes=20,


Suggested change

num_classes=20,

num_classes=19,

The number of classes is 19 on cityscapes.

Junjun2016 · 2021-10-12T05:50:04Z

configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py

+    auxiliary_head=[],
+    test_cfg=dict(mode='slide', crop_size=(512, 512), stride=(512, 512)),


Can be removed since these are the same with base config.

Addressed in 6f5ecd5

Junjun2016 · 2021-10-12T05:50:25Z

configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py

+optimizer = dict(lr=0.001, weight_decay=0.0)
+
+# num_gpus: 8 -> batch_size: 8
+data = dict(samples_per_gpu=1, )


Suggested change

data = dict(samples_per_gpu=1, )

data = dict(samples_per_gpu=1)

Addressed in 4f5762f

Junjun2016 · 2021-10-12T05:54:16Z

configs/segmenter/segmenter_vit-s_linear_512x512_160k_bs8_ade20k.py

+# num_gpus: 8 -> batch_size: 8
+data = dict(samples_per_gpu=1, )
+# TODO: handle img_norm_cfg
+# img_norm_cfg = dict(mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=True)


Did you use this img_norm_cfg in your paper?

Yes, the normalization of ViT is [0.5, 0.5, 0.5] for mean and std (assuming the input tensor is in [0, 1]).
In my repository, I checked thtat the loading and normalization used for ViT checkpoints was valid by checking the resulting performances on ImageNet validation set which were correct.

So, it means that we should use this img_norm_cfg in the segmenter' base config?

Yes, the normalization of ViT is [0.5, 0.5, 0.5] for mean and std (assuming the input tensor is in [0, 1]). In my repository, I checked thtat the loading and normalization used for ViT checkpoints was valid by checking the resulting performances on ImageNet validation set which were correct.

Great, but we also need to align the inference performance first for semantic segmentation and the next step is to align the training performance.

Junjun2016 · 2021-10-12T05:55:06Z

configs/segmenter/segmenter_vit-b_linear_512x512_160k_bs8_ade20k.py

+# num_gpus: 8 -> batch_size: 8
+data = dict(samples_per_gpu=1, )
+# TODO: handle img_norm_cfg
+# img_norm_cfg = dict(mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=True)


Did you use this img_norm_cfg in your paper?

Junjun2016 · 2021-10-12T11:01:50Z

mmseg/models/decode_heads/segmenter_linear_head.py

+from .decode_head import BaseDecodeHead
+
+
+def init_weights(m, std=0.02):


It is suggested that using init_cfg to control init weights (inherit from BaseModule), refer to https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/backbones/resnet.py#L370 and https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/base_module.py#L56.
If the init weight strategy can not be fully covered by init_cfg, we should override the init_weights in BaseModule (https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/base_module.py#L56), refer to https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/backbones/swin.py#L661, https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/backbones/vit.py#L262, and https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/backbones/mit.py#L365.

Junjun2016 · 2021-10-12T11:29:17Z

mmseg/models/decode_heads/segmenter_linear_head.py

+
+
+@HEADS.register_module()
+class SegmenterLinearHead(BaseDecodeHead):


Can inherit from FCNHead and override forward.

Junjun2016 · 2021-10-12T12:17:10Z

mmseg/models/decode_heads/segmenter_linear_head.py

+    def __init__(self, in_channels, init_std=0.02, **kwargs):
+        super(SegmenterLinearHead, self).__init__(
+            in_channels=in_channels, **kwargs)
+        self.head = nn.Linear(in_channels, self.num_classes)


Can use 1x1 conv instead.

Junjun2016 · 2021-10-12T12:26:13Z

Hi @rstrudel
Could you please push a new PR from a new branch instead of the master branch?
Since we can not push code to your master branch.

Junjun2016 · 2021-10-12T12:34:01Z

mmseg/models/decode_heads/segmenter_mask_head.py

+
+
+@HEADS.register_module()
+class SegmenterMaskTransformerHead(BaseDecodeHead):


Should also refactor this segmentation head according to the above comments.

Junjun2016 · 2021-10-12T12:34:45Z

mmseg/models/decode_heads/segmenter_linear_head.py

+class SegmenterLinearHead(BaseDecodeHead):
+
+    def __init__(self, in_channels, init_std=0.02, **kwargs):
+        super(SegmenterLinearHead, self).__init__(


Missing docstring and unittests.

Junjun2016 · 2021-10-12T12:34:56Z

mmseg/models/decode_heads/segmenter_mask_head.py

+class SegmenterMaskTransformerHead(BaseDecodeHead):
+
+    def __init__(
+            self,


Missing docstring and unittests.

rstrudel · 2021-10-12T13:42:11Z

Hi @rstrudel Could you please push a new PR from a new branch instead of the master branch? Since we can not push code to your master branch.

Thanks for all the comments @Junjun2016, I will work on them as soon as I can. Let me create a PR from a new branch.

rstrudel · 2021-10-14T15:17:29Z

I close this PR following #952 (comment) , the new PR is #955 and I will make progress there.

* fix demo * update * fix * fix bug * fix bug * update doc

rstrudel added 11 commits October 6, 2021 10:53

segmenter: add model

eaef5e8

update

cc8e3f4

readme: update

d609009

config: update

785a9cb

segmenter: update readme

e0c0766

segmenter: update

6bcec05

segmenter: update

5342698

segmenter: update

5570f10

configs: set checkpoint path to pretrain folder

48cb36b

segmenter: modify vit-s/lin, remove data config

b93ad3a

rreadme: update

e8366e3

rstrudel changed the title ~~[Feature] Add Segmenter~~ [Feature] Support Segmenter Oct 11, 2021

Junjun2016 reviewed Oct 11, 2021

View reviewed changes

mmseg/models/decode_heads/__init__.py Outdated Show resolved Hide resolved

Update mmseg/models/decode_heads/__init__.py

6441e97

Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

Junjun2016 reviewed Oct 12, 2021

View reviewed changes

rstrudel mentioned this pull request Oct 12, 2021

[Feature] Support Segmenter #955

Merged

rstrudel closed this Oct 14, 2021

michaelzhang-ai pushed a commit to michaelzhang-ai/mmsegmentation that referenced this pull request Mar 22, 2024

[Improvement] Make demo more robust in cross-platforms (open-mmlab#952)

a29e4aa

* fix demo * update * fix * fix bug * fix bug * update doc

		auxiliary_head=[],
		test_cfg=dict(mode='slide', crop_size=(512, 512), stride=(512, 512)),

	data = dict(samples_per_gpu=1, )
	data = dict(samples_per_gpu=1)

		from .decode_head import BaseDecodeHead


		def init_weights(m, std=0.02):



		@HEADS.register_module()
		class SegmenterLinearHead(BaseDecodeHead):



		@HEADS.register_module()
		class SegmenterMaskTransformerHead(BaseDecodeHead):

Conversation

rstrudel commented Oct 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

To be done

Uh oh!

CLAassistant commented Oct 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Junjun2016 commented Oct 11, 2021

Uh oh!

rstrudel commented Oct 11, 2021

Uh oh!

Junjun2016 commented Oct 12, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rstrudel Oct 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rstrudel Oct 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Junjun2016 commented Oct 12, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rstrudel commented Oct 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rstrudel commented Oct 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

rstrudel commented Oct 11, 2021 •

edited

Loading

CLAassistant commented Oct 11, 2021 •

edited

Loading

rstrudel Oct 12, 2021 •

edited

Loading

rstrudel Oct 12, 2021 •

edited

Loading

rstrudel commented Oct 12, 2021 •

edited

Loading