General support for Float16 and other DTypes

[vchuravy]

So from what I can tell the following operators currently don't support anything than real_t e.g. Float32. I am going to work to fix the ones important for my research and I would welcome any help. I feel that having comprehensive support for other datatypes is important for MXNet.

Up for grabs

• crop
• slice_channel
• softmax_activation
• matrix_op
• l2_normalization
• make_loss
• identity_attach_KL_sparse_reg
• broadcast_reduce
• embedding
• smooth_l1_unary

Depending on a resolution to dmlc/mshadow#125

• leaky_relu [RFC][DTypes] pooling and LeakyReLU #2280
• regression_output DType regression #3018
• lrn
• batch_norm

Done

• roi_pooling OP ROIPooling CPU fix and DType support #3011
• deconvolution Enable DTypes in deconvolution #2322
• dropout
• pooling
• reshape DTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380
• swapaxis DTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380
• elementwise_sum DTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380
• upsampling DTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380
• concat DTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380
• block_gradDTypes for Concat, UpSampling, Reshape, BlockGrad, SwapAxis and ElementWiseSum #2380

[Godricly]

May I ask how the half precision is supported on the CPU?
Can I view the pooling #2280 as an example to do this?

[vchuravy]

I am following the way convolution is implemented and I think on CPU Float16 is implemented by promoting to Float32. https://gcc.gnu.org/onlinedocs/gcc/Half-Precision.html

[piiswrong]

I have a 1080 coming in next week. Let's try to merge in the common ones so we can have benchmark numbers released asap

[vchuravy]

If you start working on one, just post it here so that we don't duplicate effort.

[vchuravy]

@Godricly #2322 Is probably a good template of how to do it.

[vchuravy]

I updated the list a bit.

[tqchen]

I would need to mention one thing. Simply support half_t type was not enough for making things faster with fp16. Usually a explicit vectorization of code is needed. So unless things are operated together in a Packet structure with intrinsics. There is less likely to be speedup.

[Godricly]

It sounds like the underlining mshadow needs optimization for data alignment. Another thing I'm thinking about is whether we should add a option to enable backward computation in higher precision(float) for half_t type. The half_t type cannot represent too small gradients. And it will be messy in coding to enable this.
Working on embedding now.

[Godricly]

@vchuravy I have a updated Dtype pooling branch based on your work. Can you double check this one and submit a pr? My fork of MxNet is kind of messy now.

[xlvector]

@Godricly Hi, do you have some examples of using fp16? Is it used in training or inference?

[Godricly]

Not yet. Basically, u can insert some cast layer to transform input(data and label) into fp16, so the network flows in fp16. Currently, mxnet has compatible issue with fp16, so I cancelled my previous PR #2564.

If you are interested in fp16, u can follow my branch to enable fp16 param init and single machine training, the multi machine one depends ps-lite which is a little bit hard to get it work.

U also need to make some modification on optimizer, which use float type to update weights and convert them back to fp16 in network.

For the lstm case, the provided data type is need, which is painful. If you have any better solution, please let me know. 😆

BTW, the DType BN is only functional using cudnn.

[xlvector]

@Godricly Thanks very much.

[Godricly]

Pooling and Dropout have been merged.