[Bug] Default noise constraint may be incompatible with some priors

[gpleiss]

🐛 Bug

Using a GammaPrior in conjunction with the GaussianLikelihood can lead to sampling errors with Pyro. This is because the default GaussianLikelihood noise_constraint is [1e-4, \infty), while the GammaLikelihood has support over (0, \infty). When the GammaLikelihood samples a < 1e-4 value, the resulting noise parameter is NaN.

In general, we should set the default noise constraint to match the support of the prior, or we should remove the default noise constraint when using a prior.

@Balandat what would be the least breaking approach with BoTorch?

(See #2643)

To reproduce

** Code snippet to reproduce **

# need pyro installed
import gpytorch
import torch

noise_prior = gpytorch.priors.GammaPrior(1.0, 0.5)
likelihood = gpytorch.likelihoods.GaussianLikelihood(
    noise_prior=noise_prior,
    # noise_constraint=gpytorch.constraints.Positive(),  # <- error goes away if we uncomment this line
    batch_shape=torch.Size([100,])
)
for i in range(1000):
    sampled_likelihood = likelihood.pyro_sample_from_prior()
    noise = sampled_likelihood.noise
    if not torch.equal(noise, noise):
        raise RuntimeError("Sampled noise has NaNs")

** Stack trace/error message **

RuntimeError: Sampled noise has NaNs

System information

• GPyTorch Version 1.4
• PyTorch Version 2
• MacOS X