Latent function inference produces absurd output small fraction of the time.

[pscicluna]

I'm working with the low-level interface to pyro to perform inference, and my model is a latent GP which predicts the base-10 logarithm of a density field, whose line-of-sight integral corresponds to an observable which has been sampled at some points on the sky - something like https://github.com/Thavisha/Dustribution. The log is to ensure that the model is positive-definite at all points.

Since I'm working with logs, the training can sometimes diverge to produce unphysically-large integrated values in the final result, but seemingly without influencing the ELBO, which still mostly seems to be sensible. In these cases, the theme seems to be that a small fraction of the variational mean grows to larger values, and since we're in log-space this translates into a huge shift in the density. However, the optimiser doesn't avoid this, even though it is contra-indicated by the observations. This is presumably because either the fraction of the observed sample that is affected is small, or because this improves enough other samples that it is beneficial.

As a result, I'm looking for ways to prevent this a-priori (e.g. I previously asked about placing constraints on the means in an issue, but it sounds like this might be a bad idea) or to e.g. 'reverse course' if it is encountered (and see if returning to a previous parameter set and continuing training can improve this), or any other tips or tricks that could be applied to mitigate this. I'm not currently z-scoring or minmaxing the data, since in log-space the dynamic range is quite reasonable so it is nice to stick with physical parameters.

Any suggestions you might have would be greatly appreciated! Please let me know if any further information would help.

[gpleiss]

Hi @pscicluna - sorry for the slow reply.

Rather than log-transforming your data to ensure positivity, I usually have better luck when I apply the softplus transform (torch.nn.functional.softplus). It behaves linearly when x > 1, but behaves logarithmically when 0 < x < 1.

[pscicluna]

Thanks for the suggestion. I'm not sure how effective softplus would be in this case, however - the physically-relevant range is ~10^-5 to 10^-2 (so -5 to -2 in the space I'm actually working in), so results could easily become nonsensical even within the logarithmic range. Also please note this isn't the data that is transformed, but the model is operating in the space of the logarithm of the derivative of the data (because that is the physically-interesting quantity, not the data itself) and then integrating the exponentiated model to compare to the observations.