use logsigmoid at multilabel_soft_margin_loss, and change output from shape=(N, C)to (N,) by weiyangfb · Pull Request #9965 · pytorch/pytorch

weiyangfb · 2018-07-27T23:35:02Z

fixes Make torch.nn.functional.multilabel_soft_margin_loss more stable #9141, bug of multilable loss in v0.4 #9301
use logsigmoid at multilabel_soft_margin_loss to make it more stable (NOT fixing legacy MultiLabelSoftMarginCriterion)
return (N) instead of (N, C) to match the same behavior as MultiMarginLoss
Note that with this PR, the following behavior is expected:

loss = F.multilabel_soft_margin_loss(outputs, labels, reduction='none')
loss_mean = F.multilabel_soft_margin_loss(outputs, labels, reduction='elementwise_mean')
loss_sum = F.multilabel_soft_margin_loss(outputs, labels, reduction='sum')

loss.sum() == loss_sum  # True
loss.mean() == loss_mean  # True

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/nn/functional.py

soumith

approved. change comment.

torch/nn/functional.py

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/nn/functional.py

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

li-roy · 2018-08-02T16:25:21Z

Why are we changing the behavior of all these losses?

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/nn/functional.py

-    return binary_cross_entropy(input, target, weight, None, None, reduction)
+
+    loss = -(target * logsigmoid(input) + (1 - target) * logsigmoid(-input))
+    loss.sum(dim=0)  # only return N loss values


weiyangfb · 2018-08-03T03:12:34Z

@li-roy I made changes accordingly, maybe take a look when you get a chance?

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

li-roy

Looks good for the most part, just need to be consistent in the behavior.

torch/nn/functional.py

+    loss = loss.sum(dim=1)  # only return N loss values
+
+    if reduction == 'none':
+        return loss


li-roy · 2018-08-03T21:27:22Z

If you're planning to go with averaging across classes, don't forget to change the formula in the doc in torch/nn/modules/loss.py

facebook-github-bot

weiyangfb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

… shape=(N, C)to (N,) (pytorch#9965) Summary: - fixes pytorch#9141, pytorch#9301 - use logsigmoid at multilabel_soft_margin_loss to make it more stable (NOT fixing legacy MultiLabelSoftMarginCriterion) - return (N) instead of (N, C) to match the same behavior as MultiMarginLoss - Note that with this PR, the following behavior is expected: ``` loss = F.multilabel_soft_margin_loss(outputs, labels, reduction='none') loss_mean = F.multilabel_soft_margin_loss(outputs, labels, reduction='elementwise_mean') loss_sum = F.multilabel_soft_margin_loss(outputs, labels, reduction='sum') loss.sum() == loss_sum # True loss.mean() == loss_mean # True ``` Pull Request resolved: pytorch#9965 Differential Revision: D9038402 Pulled By: weiyangfb fbshipit-source-id: 0fa94c7b3cd370ea62bd6333f1a0e9bd0b8ccbb9

vadimkantorov · 2023-12-11T22:36:40Z

torch/nn/functional.py

-    input = torch.sigmoid(input)
-    return binary_cross_entropy(input, target, weight, None, None, reduction)
+
+    loss = -(target * logsigmoid(input) + (1 - target) * logsigmoid(-input))


this could probably use torch.lerp (yet AFAIK it fails export to ONNX and has NaN problems: #71701)

weiyangfb requested review from apaszke, colesbury, ezyang, gchanan, soumith and zdevito as code owners July 27, 2018 23:35

facebook-github-bot reviewed Jul 27, 2018

View reviewed changes

soumith reviewed Jul 30, 2018

View reviewed changes

torch/nn/functional.py Outdated

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

soumith approved these changes Jul 30, 2018

View reviewed changes

soumith reviewed Jul 30, 2018

View reviewed changes

torch/nn/functional.py Outdated

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

facebook-github-bot reviewed Jul 30, 2018

View reviewed changes

weiyangfb force-pushed the stable_multilabel_soft_margin_loss branch from 1376a8b to 210f88e Compare July 31, 2018 02:39

facebook-github-bot reviewed Jul 31, 2018

View reviewed changes

li-roy mentioned this pull request Jul 31, 2018

bug of multilable loss in v0.4 #9301

Closed

li-roy reviewed Jul 31, 2018

View reviewed changes

torch/nn/functional.py Outdated

This comment was marked as off-topic.

Sign in to view

torch/nn/functional.py Outdated

This comment was marked as off-topic.

Sign in to view

facebook-github-bot reviewed Aug 1, 2018

View reviewed changes

weiyangfb changed the title ~~use logsigmoid at multilabel_soft_margin_loss~~ [wip] use logsigmoid at multilabel_soft_margin_loss, fix 'elementwise_mean' Aug 1, 2018

weiyangfb mentioned this pull request Aug 2, 2018

MSELoss wrongly sums instead of averages when reduction='elementwise_mean' #10148

Closed

weiyangfb added 2 commits August 2, 2018 08:21

use logsigmoid at multilabel_soft_margin_loss

4c2b1b8

elementwise_mean should be global mean

f778ce0

weiyangfb force-pushed the stable_multilabel_soft_margin_loss branch from 9ea862e to 5a10812 Compare August 2, 2018 15:23

changed output of MultiLabelMarginLoss from (N,C) to (N)

c2b71e0

weiyangfb force-pushed the stable_multilabel_soft_margin_loss branch from 5a10812 to c2b71e0 Compare August 2, 2018 20:39

facebook-github-bot reviewed Aug 2, 2018

View reviewed changes

weiyangfb changed the title ~~[wip] use logsigmoid at multilabel_soft_margin_loss, fix 'elementwise_mean'~~ use logsigmoid at multilabel_soft_margin_loss, and change output from shape=(N, C)to (N,) Aug 2, 2018

li-roy suggested changes Aug 2, 2018

View reviewed changes

weiyangfb added 2 commits August 2, 2018 16:30

addresses comment

41cf0bb

dim=1

55c48e0

fix tests

f8fb411

facebook-github-bot reviewed Aug 3, 2018

View reviewed changes

li-roy reviewed Aug 3, 2018

View reviewed changes

torch/nn/functional.py

loss = loss.sum(dim=1) # only return N loss values

if reduction == 'none':

return loss

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

weiyangfb added 2 commits August 3, 2018 13:38

addresses comments, follow legacy behavior for reduction

58fcb2e

lint

2106e3f

update loss formula in doc

5ae40c4

li-roy approved these changes Aug 3, 2018

View reviewed changes

facebook-github-bot reviewed Aug 3, 2018

View reviewed changes

facebook-github-bot closed this in 149d4f7 Aug 4, 2018

ezyang added the merged label Jun 26, 2019

vadimkantorov reviewed Dec 11, 2023

View reviewed changes

Conversation

weiyangfb commented Jul 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

soumith left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

li-roy commented Aug 2, 2018

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

weiyangfb commented Aug 3, 2018

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

li-roy left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

li-roy commented Aug 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

vadimkantorov Dec 11, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

weiyangfb commented Jul 27, 2018 •

edited

Loading

li-roy commented Aug 3, 2018 •

edited

Loading