Where are the loss functions/Linear Algebra Routines actually implemented?

[kevinli1324]

Hi all,

I'm trying to use some of the excellent Gpytorch linear algebra back end and loss function implementations as a starting point for scaling some new model ideas. However, I'm having trouble finding the files where the losses and linear algebra steps are actually calculated. Could someone give some pointers on

1. are the linear algebra backend steps (Conjugate gradients, MVM speed ups etc...) all automatically implemented in the linear operator library. I.E, torch.solve(K, y) will automatically use CG if K, y are linear operators.

2. Where are the forms of the loss functions are actually coded? I looked in the mlls folder and could not find the actual loss function implementation/formulas.

I apologize if the question seems silly, as I do not have a strong software engineering background. Thanks!

[Balandat]

Hi, thanks for your interest!

1. Yes, the core linear algebra routines are all implemented in the linear_operator library. torch.solve(K, y) will automatically dispatch to the solve method that is implemented on the particular LinearOperator class of the tensor K (https://github.com/cornellius-gp/linear_operator/blob/main/linear_operator/operators/_linear_operator.py#L2162 registers the function on the base class). So if there is a simplification (e.g. the LinearOperator is diagonal) then this may not use linear CG but some other method.
2. These are there in the MLL module. E.g. for an exact GP the MLL is evaluated in these lines: https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/mlls/exact_marginal_log_likelihood.py#L62-L65. For instance, in the simples case the likelihood called on the function_dist returned by the GP will add a noise level, and then the log_prob call in the following line will evaluate the log probability.

There is some more (ideally we could expand on this with more time) explanation of this in the linear_operator demo notebook: https://github.com/cornellius-gp/linear_operator/blob/main/examples/LinearOperator_demo.ipynb