SOAP optimizer

[jdart1]

Ceres chess (https://github.com/dje-dev/Ceres) is using this recently published optimization algorithm: https://arxiv.org/abs/2409.11321. There is a Python implementation. It is reportedly faster and more performant than AdamW.

[jw1912]

@jdart1 I am currently in the lead up to final university exams so I probably will not have time to implement this (as it is non-trivial and requires new operations on each backend) for around 2-3 months.
Some thoughts about it though:

• This seems likely to be an insane training slowdown (in pos/sec terms) for king bucketed networks
• I can only see evidence for its improved performance on deep neural networks, in particular transformers (as Ceres net is)

[jdart1]

It is only a suggestion. If it is less efficient then there is no point in implementing it.

[cosmobobak]

On the topic of alternate optimisers, there's also Muon.