Flexpoint — float16 with 16-bit mantissa
Intel — as part of their Nervana processors — introduced Flexpoint (flex16+5), for tensor operations. Gory details in the paper below, but think of it as
- • all the elements of the tensor have the same exponent
- • the extra 5 bits of precision in a 16bit op can now be allocated to the mantissa.
- • which means you’re basically doing integer operations on the entire tensor
- • which can mean faster, and less expensive, hardware
- • the exponent (and hence the dynamic range) is updated for the tensor as a whole during training
The last bit is particularly useful — and fascinating! — since it turns out that during training, tensors “have a rather peaked (and rightward skewed) distribution sufficiently covered by a 16-bit range. But the positions of these ranges vary from tensor to tensor, and for certain tensors they shift significantly during the course of training” (image above).
Regardless of whether Nervana catches on, flex16+5 seems like a pretty nifty idea for Deep Learning
Comments