Evaluation with ImageNet

[meigaoms]

Hi there,
"A logistic regression classifier using scikit-learn’s L-BFGS implementation" is used in A.3 Evaluation. It's a brilliant choice for its simplicity for a fair comparison. This classifier works well for most datasets, but takes me a very long time for ImageNet dataset, about 8 days with a single CPU. Could you shed some lights on how to speed it up?
Thank you in advance.

[jongwook]

We used a quite beefy machine with many CPU cores to run each logistic regression taking about 2 days, but it took more than a week for us to finish the hyperparameter sweep. This was okay for this project as a one-off task, but we're looking into ways to perform logistic regression on distributed GPUs so that it can produce equivalent results as sklearn's while running faster.

Using Adam or SGD with momentum instead of L-BFGS allows to use faster mini-batch training instead of full-batch training, but it's tricky to get the optimal result from it, and the gap seems to vary between models/datasets. This was precisely the reason why we sticked to scikit-learn.

During earlier iterations on the project, we used a subset of 50,000 random samples from the dataset for validation, which can be run quite faster.

Btw, we haven't got a chance to look into #45 yet. Thanks for your patience!

[meigaoms]

Exactly! The hyperparameter sweep takes most of the time. Thanks a lot.
I found cuML has a GPU version logistic regression with L-bfgs for multinomial classification. Unfortunately, its L-bfgs implementation with L2 regularization is not exactly the same as sklearn's, so it fails to produce an equivalent result still.