Improve numerical stability of `torch.distributions.wishart.Wishart` by nonconvexopt · Pull Request #72993 · pytorch/pytorch

nonconvexopt · 2022-02-17T02:42:08Z

Maintanance of #70377
Multiple modifications of the merged initial implementation of Wishart distribution.

Key modifications:

torch/distributions/wishart.py: Clamp (Clip) float type values to calculate reciprocal in numerically stable manner, by using the eps value paired to each torch.dtype
test/distributions/test_distributions.py: Test Wishart distribution implementation in numerically unstable zones, i.e df values are at ndim - 1 < df < ndim where ndim is the one dimenstion of the Wishart parameter & sample matrix.

Re-opened reverted PR #72059
cc @neerajprad @vadimkantorov

pytorch-bot · 2022-02-17T02:42:12Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/nonconvexopt/pytorch/blob/1a2703c358cb97de0cb567bbf5a0fc704fee7bc4/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
linux-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-manywheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
linux-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/trunk`	✅ triggered
linux-bionic-rocm4.5-py3.7	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/rocm`, `ciflow/trunk`	✅ triggered
linux-docs	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/docs`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-vulkan-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
macos-arm64-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
macos-arm64-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
macos-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
macos-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
macos-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
macos-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
windows-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
docker-builds	`ciflow/all`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`, `ciflow/trunk`	🚫 skipped
linux-docs-push	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-11-py3-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.7-gcc7-debug	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
periodic-win-vs2019-cuda11.5-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
pytorch-xla-linux-bionic-py3.7-clang8	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`, `ciflow/xla`	🚫 skipped

nonconvexopt · 2022-02-25T04:19:30Z

@neerajprad @vadimkantorov @fritzo I think I am not allowed to reproduce the macos errors. Does the waiting for the further maintenance of CI will solve the issue?

test (default, 1, 2, macos-11)

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:iam::308535385114:user/pytorch-pytorch-ossci-s3-read-write is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:308535385114:function:rds-proxy because no identity-based policy allows the lambda:InvokeFunction action
Error: Process completed with exit code 1.

fritzo

LGTM. This is fine, but imho x / 2 is easier to read than 0.5 * x everywhere. I believe test failures are unrelated (pip failures, onnx registration errors).

…convexopt/pytorch into distributions/wishart_improvement_v1

nonconvexopt · 2022-02-28T04:35:49Z

I think it is ready to be merged.

fritzo

LGTM.

Test errors appear unrelated:

Details

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:iam::308535385114:user/pytorch-pytorch-ossci-s3-read-write is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:308535385114:function:rds-proxy because no identity-based policy allows the lambda:InvokeFunction action

@neerajprad would you be able to merge this?

nonconvexopt · 2022-03-07T11:28:05Z

Added trivial change: fix wrong parameter input (df with ndim - 1 < df < ndim) in Scipy Wishart class.
macos test failure was due to my mistake.

nonconvexopt · 2022-03-07T11:28:36Z

@pytorchbot ciflow rerun

pytorch-bot · 2022-03-07T11:28:37Z

This command didn't do anything.
You don't need to manually issue ciflow rerun commands anymore. Just adding a ciflow/ label will trigger the workflow.

nonconvexopt · 2022-03-09T00:26:20Z

LGTM

fritzo

LGTM @neerajprad I believe this is ready to merge

facebook-github-bot · 2022-03-14T01:17:55Z

@neerajprad has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

nonconvexopt · 2022-03-15T09:53:50Z

Thank you for the richful feedbacks and reviews. I appreciated your kindness. I am considering implementing Multivariate T distribution for PyTorch if there is no one trying after I finish Inverse Wishart #70275. It's a shame that I am delaying it to this far.

I guess some people are trying to use T Process similary to Gaussian Process these days. It seems that probabilistic ML libraries are actively implementing it. It might be great if PyTorch offers official implementation for Multivariate T used for T process.

fritzo · 2022-03-15T13:06:00Z

@nonconvexopt it's a pleasure to work with you. Yes Multivariate Student't t might be useful for T processes, but I'm not sure: IIUC it is just a Gamma-MultivariateNormal chain where the gamma rv can be analytically integrated out, and I believe in T processes there are local gammas, but in mutlivariate T there is a single gamma? Let's move this discussion to a new issue. Also cc @fehiepsi who worked out a lot of that math in Pyro.

fehiepsi · 2022-03-15T13:16:43Z

There is MVT distribution in Pyro, but it only supports scale_tril input. If we port/reimplement in PyTorch, I think we need to support covariance_matrix, precision_matrix inputs too.

…72993) Summary: Maintanance of #70377 Multiple modifications of the merged initial implementation of Wishart distribution. Key modifications: * torch/distributions/wishart.py: Clamp (Clip) float type values to calculate reciprocal in numerically stable manner, by using the eps value paired to each torch.dtype * test/distributions/test_distributions.py: Test Wishart distribution implementation in numerically unstable zones, i.e df values are at ndim - 1 < df < ndim where ndim is the one dimenstion of the Wishart parameter & sample matrix. Re-opened reverted PR #72059 cc neerajprad vadimkantorov Pull Request resolved: #72993 Reviewed By: samdow Differential Revision: D34853807 Pulled By: neerajprad fbshipit-source-id: eb62dca19bf8a934fdf59b4ffc58587447fe8378

…ytorch#72993) Summary: Maintanance of pytorch#70377 Multiple modifications of the merged initial implementation of Wishart distribution. Key modifications: * torch/distributions/wishart.py: Clamp (Clip) float type values to calculate reciprocal in numerically stable manner, by using the eps value paired to each torch.dtype * test/distributions/test_distributions.py: Test Wishart distribution implementation in numerically unstable zones, i.e df values are at ndim - 1 < df < ndim where ndim is the one dimenstion of the Wishart parameter & sample matrix. Re-opened reverted PR pytorch#72059 cc neerajprad vadimkantorov Pull Request resolved: pytorch#72993 Reviewed By: samdow Differential Revision: D34853807 Pulled By: neerajprad fbshipit-source-id: eb62dca19bf8a934fdf59b4ffc58587447fe8378 (cherry picked from commit 99240c0)

nonconvexopt added 28 commits January 31, 2022 23:14

Clean codes

edf8257

Improve numerical stability

bd5ac88

Check numerical stability more rigorously

ec43666

Fix typo

e533bb4

Set df values in test larger than (ndim - 1)

5fde921

Fix typo

4bf3f2c

Extend .entropy to consider multivariate natural parameters

2a7b806

Increase tolerance value for .assertEqual

5a18141

Consider SciPy version and adjust test conditions

6614bb6

Merge branch 'master' into distributions/wishart_improvement_v1

8012a14

Remove trailing spaces

57f618b

Fix typo and debug error in version comparison

aa71dd1

reduce variance in MC estimate

ade2dba

Check correct scipy version allowing df < ndim for Wishart

2147975

Allow higher error in MC estimate for moment

8af89d9

Apply harder clamping

934a626

Remove unnecessary code pieces

ba5b413

Remove experimental changes

3591ea6

Test numerical instability only in relevant tests

fc4faf7

Merge branch 'master' into distributions/wishart_improvement_v1

0dda7ca

Return correct natural paramters

d89bf08

Extend exponential family to multivariate

a578507

Debug _log_noramlizer

4e2022b

Fit flake8

7e2e240

Extend ExponentialFamily to multivariate

e630822

Remove trailing spaces

4651368

Add comment on SciPy version check

596c0d5

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

1a2703c

pytorch-bot Bot added the ciflow/default label Feb 17, 2022

nonconvexopt added 2 commits February 25, 2022 12:09

revert modifications

929d8c7

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

c731fe1

fritzo approved these changes Feb 25, 2022

View reviewed changes

Comment thread torch/distributions/wishart.py

Comment thread torch/distributions/wishart.py Outdated

Comment thread torch/distributions/wishart.py

Comment thread torch/distributions/wishart.py

nonconvexopt added 5 commits February 26, 2022 23:44

improve code structure and reflect feedbacks of fritzo

29da325

Merge branch 'distributions/wishart_improvement_v1' of github.com:non…

b7fc990

…convexopt/pytorch into distributions/wishart_improvement_v1

fix typo

24f7803

fix implementation error and optimize ops

7fa623b

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

0f0c57c

fritzo approved these changes Feb 28, 2022

View reviewed changes

nonconvexopt added 3 commits March 2, 2022 17:22

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

4a84905

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

b4592ca

add missed modification of parameter for lower scipy version

46107e9

Merge branch 'pytorch:master' into distributions/wishart_improvement_v1

bf1f987

fritzo approved these changes Mar 13, 2022

View reviewed changes

pytorchmergebot closed this in c77857c Mar 15, 2022

nonconvexopt deleted the distributions/wishart_improvement_v1 branch March 23, 2022 06:43

WBobby mentioned this pull request Aug 17, 2022

Add ROCm5.2.3/AMDGPU support for PyTorch WBobby/pytorch#2

Closed

Conversation

nonconvexopt commented Feb 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Feb 17, 2022

⚛️ CI Flow

Uh oh!

nonconvexopt commented Feb 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fritzo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nonconvexopt commented Feb 28, 2022

Uh oh!

fritzo left a comment

Choose a reason for hiding this comment

Uh oh!

nonconvexopt commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nonconvexopt commented Mar 7, 2022

Uh oh!

pytorch-bot Bot commented Mar 7, 2022

Uh oh!

nonconvexopt commented Mar 9, 2022

Uh oh!

fritzo left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Mar 14, 2022

Uh oh!

nonconvexopt commented Mar 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fritzo commented Mar 15, 2022

Uh oh!

fehiepsi commented Mar 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nonconvexopt commented Feb 17, 2022 •

edited

Loading

nonconvexopt commented Feb 25, 2022 •

edited

Loading

nonconvexopt commented Mar 7, 2022 •

edited

Loading

nonconvexopt commented Mar 15, 2022 •

edited

Loading