Implement torch.standard_gamma and distributions.Gamma by fritzo · Pull Request #3841 · pytorch/pytorch

fritzo · 2017-11-22T19:48:14Z

Addresses #3813

This implements a torch.standard_gamma() random number generator and a distributions.Gamma distribution that implements the Gamma distribution. Note that this is named torch.standard_gamma to avoid confusion with the Gamma function that is already implemented as torch.lgamma.

We follow scipy in generating standard Gamma variables Gamma(alpha, 1) rather than fully-parameterized Gamma(alpha, beta) random variables for two reasons: (1) this partial parameterization makes it easier to implement reparameterized gradients, and (2) the community is split between the scale parameter theta and the rate parameter beta = 1/theta. This PR uses the beta parameter in distributions.Gamma, but remain agnostic in torch.standard_gamma(alpha).

Tested

Added deterministic tests of shape and .log_prob() method
Added a randomized test of .sample() method (also added a test for Normal.sample())

test_distributions.py runs in under 1 second.

pytorchbot · 2017-11-22T19:48:15Z

@fritzo, thanks for your PR! We identified @zdevito to be a potential reviewer.

fritzo · 2017-11-22T19:49:29Z

@colesbury Is there anything else I should do beyond writing tests?

apaszke

LGTM

Sign in to view

+
+    def sample_n(self, n):
+        # cleanly expand float or Tensor or Variable parameters
+        def expand(v):


Sign in to view

+    - floating_point
+  backends:
+    - CPU
+    - CUDA


fritzo · 2017-11-23T03:42:31Z

Ok I've added lots more boilerplate to expose torch.random_gamma. There are some still some numerical bugs.

fritzo · 2017-11-23T06:17:40Z

@apaszke @colesbury this is ready for another review. I believe it is correct now. Thanks for the help!

Sign in to view

    return tensor.normal_(mean, std)


+def random_gamma(tensor, alpha=1, beta=1):


Sign in to view

+            else:
+                alpha = type(beta)(*beta.size()).fill_(alpha)
+        elif isinstance(beta, Number):
+            beta = type(alpha)(*alpha.size()).fill_(beta)


Sign in to view

+
+        self._check_log_prob(Gamma(alpha, beta), ref_log_prob)
+
+    # FIXME this fails due to bad numerics


Sign in to view

+        np.random.seed(seed)
+        torch.manual_seed(seed)
+        if torch.cuda.is_available():
+            torch.cuda.manual_seed_all(seed)


Sign in to view

 import math
+import numpy as np
+import scipy.stats
+import scipy.special


Sign in to view

 import math
+import numpy as np
+import scipy.stats
+import scipy.special


Sign in to view

+    def test_normal_sample(self):
+        self._set_rng_seed()
+        for mean in [-1.0, 0.0, 1.0]:
+            for std in [0.1, 1.0, 10.0]:


Sign in to view

+            else:
+                alpha = type(beta)(*beta.size()).fill_(alpha)
+        elif isinstance(beta, Number):
+            beta = type(alpha)(*alpha.size()).fill_(beta)


fritzo · 2017-11-23T22:17:29Z

Thanks for the review @apaszke ! I've addressed all your comments.

fritzo · 2017-11-26T01:41:13Z

I've simplified this PR to implement torch.standard_gamma(alpha) rather than torch.random_gamma(alpha, beta) so as to make it easier to implement reparameterized gradients in a subsequent PR. These two relate by the simple equation

random_gamma(alpha, beta) = standard_gamma(alpha) / beta

apaszke · 2017-11-26T18:27:20Z

I'm wondering if standard_gamma is distinct enough to not be confused with the regular gamma function. Couldn't we expose it as an internal method in _C and realize the sampling only through the torch.distributions API?

fritzo · 2017-11-27T21:57:25Z

Ok, I've addressed all review comments.

apaszke · 2017-11-28T08:51:59Z

@pytorcbot add to whitelist

apaszke · 2017-11-28T10:35:43Z

I think this is good to go, but it might not be exposed in ATen. cc @zdevito @colesbury what needs to be done to have it there? Just add to Declaractions.yaml?

Sign in to view

+        elif alpha_num and beta_num:
+            alpha, beta = torch.Tensor([alpha]), torch.Tensor([beta])
+        else:
+            alpha = alpha.expand_as(beta)


fritzo · 2017-11-30T20:01:33Z

@apaszke How does the merge process work? Did I set back the merge process by adding a commit after pytorchbot whitelisting? (I'm eager to get this merged because 2 other PRs are stacked on top: probtorch#26 probtorch#28)

apaszke · 2017-12-01T11:27:30Z

Yeah I think it was ready to merge, but now the broadcasting doesn't look correct to me (correct me if I'm wrong). It's all good to land once this is fixed

fritzo · 2017-12-01T17:02:31Z

Ok, I've replaced the .expand_as() with a check that the sizes are equal. This disallow-broadcasting behavior is already adopted by Normal, and will also be used by Beta.

fritzo · 2017-12-01T17:04:38Z

Any idea what's going on with the failed builds?

fritzo added 2 commits November 21, 2017 20:20

Add first sketch of THRandom_gamma generator

2056dc0

Implement boilerplate around random_gamma and Gamma distribution

35147eb

This was referenced Nov 22, 2017

Implement Gamma distribution probtorch/pytorch#19

Closed

Implement torch.random_gamma() sampler probtorch/pytorch#18

Closed

Add simple smoke test

93463e1

apaszke approved these changes Nov 22, 2017

View reviewed changes

Comment thread torch/distributions.py Outdated

def sample_n(self, n):

# cleanly expand float or Tensor or Variable parameters

def expand(v):

This comment was marked as off-topic.

Sign in to view

fmassa reviewed Nov 22, 2017

View reviewed changes

Comment thread aten/src/ATen/Declarations.cwrap Outdated

- floating_point

backends:

- CPU

- CUDA

This comment was marked as off-topic.

Sign in to view

fritzo added 2 commits November 22, 2017 16:46

Updates per review

1130234

Add wiring to expose random_gamma

e16fc77

Fix numerical bug, add more tests

7d01cf1

apaszke reviewed Nov 23, 2017

View reviewed changes

Updates per review

c7d3a72

fritzo mentioned this pull request Nov 25, 2017

Implement random_gamma() sampler (with gradients) #3813

Closed

2 tasks

fritzo changed the title ~~Implement torch.random_gamma and distributions.Gamma~~ Implement torch.standard_gamma and distributions.Gamma Nov 26, 2017

Replace torch.random_gamma() with torch.standard_gamma()

37f5fcf

Remove accidental TH_API prefix in .c file

57bc747

fritzo mentioned this pull request Nov 26, 2017

Implement Reparameterized version of Gamma probtorch/pytorch#26

Closed

Make _standard_gamma() function internal

660c881

apaszke approved these changes Nov 28, 2017

View reviewed changes

apaszke closed this Nov 28, 2017

apaszke reopened this Nov 28, 2017

Expand alpha to shape of beta in distributions.Gamma

7387792

apaszke reviewed Nov 30, 2017

View reviewed changes

Disallow broadcasting in distributions.Gamma

ed31ede

Merge branch 'upstream' into random-gamma

1872fa2

apaszke merged commit 165d089 into pytorch:master Dec 2, 2017

fritzo deleted the random-gamma branch December 13, 2017 01:56

ezyang added the open source label Jun 24, 2019

laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026

Implement distributions.Gamma (pytorch#3841)

fc1a628

		return tensor.normal_(mean, std)


		def random_gamma(tensor, alpha=1, beta=1):


		self._check_log_prob(Gamma(alpha, beta), ref_log_prob)

		# FIXME this fails due to bad numerics

Conversation

fritzo commented Nov 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tested

Uh oh!

pytorchbot commented Nov 22, 2017

Uh oh!

fritzo commented Nov 22, 2017

Uh oh!

apaszke left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

fritzo commented Nov 23, 2017

Uh oh!

fritzo commented Nov 23, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

fritzo commented Nov 23, 2017

Uh oh!

fritzo commented Nov 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apaszke commented Nov 26, 2017

Uh oh!

fritzo commented Nov 27, 2017

Uh oh!

apaszke commented Nov 28, 2017

Uh oh!

apaszke commented Nov 28, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

fritzo commented Nov 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apaszke commented Dec 1, 2017

Uh oh!

fritzo commented Dec 1, 2017

Uh oh!

fritzo commented Dec 1, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

fritzo commented Nov 22, 2017 •

edited

Loading

fritzo commented Nov 26, 2017 •

edited

Loading

fritzo commented Nov 30, 2017 •

edited

Loading