Inter-task covariance

[ekinugurel]

In the multi-task GP regression setting, the inter-task covariance matrix is supposed to be "a lookup table containing inter-task covariance". How exactly does this work, mathematically? Originally, I thought that this was simply the covariance between output vectors, e.g. for two output vectors this matrix would be 2x2, with the diagonal entries being 1 and the non-diagonal entries being the covariance between two random variables. Is this the case?

I run the following snippet of code, expecting to get a very high covariance between the two output vectors (as they are identical). However, it seems that the two covariances change each time I run this, even though I don't change anything else in the code. What may be the reason for this?

from gpytorch.kernels import RBFKernel
from gpytorch.means import ConstantMean
from gpytorch.distributions import MultitaskMultivariateNormal
from gpytorch.models import ExactGP
from gpytorch.kernels import MultitaskKernel
from gpytorch.means import ConstantMean
from gpytorch.likelihoods import MultitaskGaussianLikelihood
import torch

output1 = torch.tensor([1, 2, 3, 4, 5])
output2 = torch.tensor([1, 2, 3, 4, 5])

input1 = torch.tensor([1, 2, 3, 4, 5])

class MultitaskGPModel(ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(MultitaskGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = MultitaskMean(ConstantMean(), num_tasks=2)
        self.covar_module = MultitaskKernel(
            RBFKernel(), num_tasks=2, rank=1
        )

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return MultitaskMultivariateNormal(mean_x, covar_x)
    
likelihood = MultitaskGaussianLikelihood(num_tasks=2)
model = MultitaskGPModel(
    train_x=input1,
    train_y=torch.cat([output1, output2]),
    likelihood=likelihood,
)

model.covar_module.task_covar_module.covar_factor

[Balandat]

covariance". How exactly does this work, mathematically? Originally, I thought that this was simply the covariance between output vectors, e.g. for two output vectors this matrix would be 2x2, with the diagonal entries being 1 and the non-diagonal entries being the covariance between two random variables. Is this the case?

Yes.

I run the following snippet of code, expecting to get a very high covariance between the two output vectors (as they are identical). However, it seems that the two covariances change each time I run this, even though I don't change anything else in the code. What may be the reason for this?

Assuming the above is all the code you have:

• model.covar_module.task_covar_module.covar_factor only returns the factor. The inter-task covariance matrix is parameterized as RR^T + D, where D is a diagonal matrix and R is the covar_factor you're looking at: https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/index_kernel.py#L95-L97 The factorization is not necessarily unique (in fact there is a normalization step so an

• The other issue here is that it seems that you're not even fitting the model? So in that case the covar factor (and thus the inter-task covariance) are just randomly initialized. You'll need to fit the model first for this to contain any meaningful parameter values.

[ekinugurel]

I see. Then I guess I need to multiply what I see in covar_factor by its transpose to get the inter-task covariances? Would model.covar_module.task_covar_module.covar_factor.matmul( model.covar_module.task_covar_module.covar_factor.t() ) do that?

Also, is D stored anywhere?

[Balandat]

You can just call model.covar_module.task_covar_module.covar_matrix to get the covariance. The elements of D are stored in the IndexKernel's var attribute as you can see here: https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/index_kernel.py#L96

[ekinugurel]

Max, thanks for your replies here. I just wanted to clarify one point. Is the inter-task similarity matrix influenced by the inputs at all? I ask this because I've heard interpretations of multi-task learning that regards the "task" as the learning problem between an input-output pair. And therefore multi-task learning in that context refers to the similarity of the relationship between various input-output vectors.

[Balandat]

Is the inter-task similarity matrix influenced by the inputs at all?

The covar factor is a learnable parameter, so the inter-task similarity matrix is learned from the data based on maximizing the MLL which depends on the inputs. Once the model parameters are fit, this matrix is fixed (i.e. doesn't depend on the inputs during prediction on new inputs).

[ekinugurel]

Got it. Thanks for taking the time to explain.