Multi-step time series forecasting

[MarkusTMaier]

Thank you very much for provide this great tool.
I would like to do time series forecasting. My problem is similar to the following toy model:

y(t+1)=2y(t)-1.03y(t-1)+0.05*x(t)

where t is the time step, x(t) is a variable, and y(t) is also a variable. In reality x(t) might be the force and y(t) might by the humidity. In this toy example, the relationship between x(t), y(t+1), y(t-1), and y(t) is known; In reality, the model parameters are unknown to me.

First, I can fit a model using Gaussian process regression, where y(t+1) is an output and y(t), y(t-1), and x(t) are inputs to the Gaussian process regression. I can measure y(t) and x(t) at each time-step. Therefore, I can use these measurements and fit the hyperparameters of the Gaussian process model. I think this step is works.
Since the y(t) measurements are very expensive, I would like to stop doing these experiments at some point and just measure x(t). At that point, I need to start forecasting y(t). The idea is to forecast y(t+1) and then use that prediction as an uncertain input to the model to predict y(t+2). This process is repeated for several steps up to y(t+n). As I understand it, the problem is that the outputs will become non-Gaussian and I need to use approximation methods or Monto-Carlo sampling. Is there a way to implement this in GPyTorch? I think my problem is somehow related to the following paper:
https://www.dcs.gla.ac.uk/~rod/publications/GirRasMur02-TR-2002-119.pdf

Thanks for your help and feedback.

[Balandat]

Yeah you can "fantasize" values for y(t+1) by sampling from the model posterior, then condition your model on those fantasized observations (which yields a batched model with each element of the batch corresponding to one sample). You can telescope that out by sampling from that posterior, essentially building a tree of samples. This is implemented in gpytorch through the following:

gpytorch/gpytorch/models/exact_gp.py

Line 138 in f12c56b

def get_fantasy_model(self, inputs, targets, **kwargs):

See https://proceedings.neurips.cc/paper/2020/hash/d1d5923fc822531bbfd9d87d4760914b-Abstract.html for a discussion of this technique. This is implemented (using gpytorch under the hood) in https://github.com/pytorch/botorch/blob/main/botorch/acquisition/multi_step_lookahead.py#L49

[rhagenbuch]

Hello everyone. First, I want to thank you for providing this great tool.

I would like to do time-series forecasting and make predictions y(t+1), using the preceding predictions y(t), y(t-1), y(t-2) as a new input. Since there is an uncertainty associated with the new inputs, I would like to propagate this through the model to get a prediction, which isn't to overconfident. During my search for a solution I stumbled upon this Q&A Discussion, which addresses pretty much the same topic.

As far as I understand the answer suggests using the Monte Carlo approach by sampling from the posterior, which appears to yield promising results. However, I was wondering if there is also a way to use approximation methods, as discussed in the thesis of Agathe Girard, when working with GPyTorch? Since we don't have to generate predictions for numerous samples, I believe that the approximation methods might offer computational advantages?

Thesis of Agathe Girard: https://www.dcs.gla.ac.uk/~rod/publications/Gir04.pdf

Thank you very much for your help.

[MarkusTMaier]

Dear @Balandat,

thank you very much for your quick and precise answer. This is a very good starting point to begin with the implementation.