adjustable gradient clipping ("clip_gradients / current_lr")

[madshi]

This paper:

https://arxiv.org/pdf/1511.04587.pdf

... suggests "adjustable gradient clipping", which seems to greatly help in training deep networks quickly and efficiently. Basically they suggest to scale the gradient clipping with "1 / learning rate". So they want to clip the gradients to [- clip_gradients / current_lr, + clip_gradients / current_lr].

As far as I can see, Caffe doesn't support this yet, correct? Might be useful to add?

[shelhamer]

Caffe does not do this, and clips the gradient itself: https://github.com/BVLC/caffe/blob/master/src/caffe/solvers/sgd_solver.cpp#L101-L116

The proposed method is closer to update clipping since it incorporates the learning rate. This could be incorporated into Caffe by adding a new flag and passing rate to ClipGradients(). I'd like to see further empirical demonstration that this is useful to do first.