What kind of tuning would be helpful for reaching great performance?

[MJimitater]

Hello GPytorch Team! Thank you for always improving and pushing the limits of this package, it makes it really fun!

I'd like to follow up my question from #1650, where Im training a GP Classifier on large data (corresponding code is given in #1650).
Thanks to the team, I got the idea to input a larger set of induced_points when initializing the model.
For this, I chose a random subset of the dataset of size (3000, 240) - this is about 2.3% of the entire dataset, so quite small.
Together with a batch_size=1024 and epochs=20 I can train my model with decreasing loss (from 0.787 down to 0.502), I can calculate the test accuracy and F1-score and reach about 0.9 in both.
My goal is to take the performance even higher. I ask myself what is the best way to do this:

• Increase epochs? 100? 1000?
• Increase batch_size? 4k? 8k?
• Change induced dataset in any way? Any larger than what I've got now throws a GPU out-of-memory error. Would training on CPU help to take the entire training set as induced dataset? Is this even a good idea?
• Try different kernels? Right now I have standard RBF with constant mean prior.
• Something else I most likely missed?

Which of these ways seems most promising?
When I take sklearn GPClassifier (which is soooo slow), there are far less hyperparameters to set (other than kernel). It might take ages, but the test results are way above 0.9. But I really do not want to go back to sklearn, GPytorch is just too good.

Any recommendations or suggestions? Sorry if these questions might seem dull, Im a beginner in this field, but a curious and eager one. Thanks

[jacobrgardner]

Have you tried your code in #1650 but with just learning the inducing point locations? I notice your code there uses learn_inducing_locations=False -- typically it's much better to learn the inducing point locations.

Additionally, increasing the number of epochs is very likely to help, especially if you decrease the learning rate at some point. 5 epochs is a pretty small amount of training. I'd recommend trying with 300 epochs, lowering the learning rate at least once during training (check out MultiStepLR in PyTorch), and learning the inducing point locations.

[MJimitater]

Okay, Im back @jacobrgardner. I trained 300 epochs with learn_inducing_locations=True and MultiStepLR(optimizer, milestones=[100,200], gamma=0.1), but the confusion matrix shows

[[7595    0]
 [ 470    0]].

Obviously the learning didn't work.
I also noticed that the loss reaches 0.695 at 20 epochs, and then basically stays there until the end (with very slight variance).

So I quickly tested the following:
50 epochs with MLStepLR (milestones=[20,40], gamma=0.1) and learn_inducing_locations=False.
The results are promising:

Test acc: 0.905
Confusion Matrix: 
[[6226 1369]
[ 32 7172]] 
F1: 0.898

Perhaps learn_inducing_locations=False does make the difference?!
The loss however reaches 0.508 after 12 epochs and basically stays there till the end.

Any further recommendations that come to mind by any chance? Thanks

[MJimitater]

Hello Team, any other suggestions perhaps so far? Because I can't wait to optimize my GPytorch Model :)

[gpleiss]

Are you z-scoring your data? Normalizing the x and y to have zero mean and unit variance often helps.

[MJimitater]

Thanks @gpleiss for your reply. I min-max-scaled my X:
X = (X - X.min())/(X.max() - X.min())
Since I have time series data, this brings each sample into the [0,1] range. I will try to z-score X.
My y however is binary, since I have a binary classification task. Z-scoring y doesn't make sense then, right?
I will report back.

[MJimitater]

Wow! Thank you @gpleiss , I thought I had min-max-scaling, but actually I didn't!
I added it and did some experiments:

MinMax scaling + learn_induced_locations = False

acc: 0.94
F1: 0.94
predict on single sample: mean of 0.16, std of 0.37 (true class: 0)

MinMax scaling + learn_induced_locations = True

acc: 0.8
F1: 0.85
predict on single sample: mean of 0.47, std of 0.5 (true class: 0)

I also tried z-scoring the X data:

Standard scaling + learn_induced_locations = False

acc: 0
F1: 0
predict on single sample: mean of 0.48, std of 0.5 (true class: 0)

Standard scaling + learn_induced_locations = True

acc: 0
F1: 0
predict on single sample: mean of 0.48, std of 0.5 (true class: 0)

Obviously z-scoring didn't work, MinMax scaling did however give a boost.
Strangely learn_induced_locations = False gives better results than if True, I don't know why as I expect it to be the other way around.

Thank you for hinting me, I wouldn't have realized that I missed the scaling!
If you have any more ideas, don't hesitate to add them here, I appreciate any thought.
Thanks