Extending the DKL tutorial to higher dimensional data

[whornsby82]

Hello all, I hope this is the right place for this. Im trying to expand the tutorial for the DKL (https://docs.gpytorch.ai/en/stable/examples/06_PyTorch_NN_Integration_DKL/KISSGP_Deep_Kernel_Regression_CUDA.html#) to a higher dimensional dataset (in my case 7 parameters).

Ive adjusted all the parameters I can see:

similarly here data_dim=7:

'class LargeFeatureExtractor(torch.nn.Sequential):
def init(self):
super(LargeFeatureExtractor, self).init()
self.add_module('linear1', torch.nn.Linear(data_dim, 1000))
self.add_module('relu1', torch.nn.ReLU())
self.add_module('linear2', torch.nn.Linear(1000, 500))
self.add_module('relu2', torch.nn.ReLU())
self.add_module('linear3', torch.nn.Linear(500, 50))
self.add_module('relu3', torch.nn.ReLU())
self.add_module('linear4', torch.nn.Linear(50, 2))

feature_extractor = LargeFeatureExtractor()'

but I'm getting an error when running:

'Traceback (most recent call last):
File "/Users/kz0451/Documents/GitHub/gs2_gpe/analysis/../GPytorchTest7dDKL.py", line 231, in
loss3 = -mll3(output3, train_y)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/module.py", line 30, in call
outputs = self.forward(*inputs, **kwargs)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 64, in forward
res = output.log_prob(target)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/distributions/multivariate_normal.py", line 168, in log_prob
covar = covar.evaluate_kernel()
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/linear_operator/operators/added_diag_linear_operator.py", line 191, in evaluate_kernel
added_diag_linear_op = self.representation_tree()(*self.representation())
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/linear_operator/operators/linear_operator.py", line 1938, in representation_tree
return LinearOperatorRepresentationTree(self)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/linear_operator/operators/linear_operator_representation_tree.py", line 13, in init
representation_size = len(arg.representation())
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/lazy/lazy_evaluated_kernel_tensor.py", line 370, in representation
return self.evaluate_kernel().representation()
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/utils/memoize.py", line 59, in g
return add_to_cache(self, cache_name, method(self, *args, **kwargs), *args, kwargs_pkl=kwargs_pkl)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/lazy/lazy_evaluated_kernel_tensor.py", line 329, in evaluate_kernel
res = self.kernel(
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/kernels/kernel.py", line 408, in call
super(Kernel, self).call(x1, x2, last_dim_is_batch=last_dim_is_batch, **params)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/module.py", line 30, in call
outputs = self.forward(*inputs, **kwargs)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/kernels/grid_interpolation_kernel.py", line 187, in forward
left_interp_indices, left_interp_values = self._compute_grid(x1, last_dim_is_batch)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/kernels/grid_interpolation_kernel.py", line 140, in _compute_grid
interp_indices, interp_values = Interpolation().interpolate(self.grid, inputs)
File "/Users/kz0451/Library/Python/3.9/lib/python/site-packages/gpytorch/utils/interpolation.py", line 49, in interpolate
assert num_dim == len(x_grid)
AssertionError'

which seems to be an issue on the data dimensionality but I can't really tell from the output. Am I on the right track?

Many thanks

Kind regards

Will

[gpleiss]

The final NN output layer (linear4) outputs a 2 dimensional vector - it needs to be increased to 7 dimensions to match your new latent space.

However, expanding to a 7 dim latent space is a bad idea, especially with 100 grid points per dimension (this will result in 100^7 total grid points). One of the main features of DKL is being able to reduce the dimensionality of high dimensional vectors, so the latent dimensionality of the kernel shouldn't exceed 2 or 3.

[whornsby82]

Hello Geoeff,

many thanks for the response. Yes, this is a consideration....im working on a dataset that had 350points in total over the 7 dimensions, so its not as bad as 100^7. Ive tried some dimension reduction but it seems that the parameters are indeed independent. I can try to reduce the number of data points initially and see.