Leveraging the bfloat16 Artificial Intelligence Datatype For Higher-Precision Computations

Abstract

In recent years fused-multiply-add (FMA) units with lower-precision multiplications and higher-precision accumulation have proven useful in machine learning/artificial intelligence applications, most notably in training deep neural networks due to their extreme computational intensity. Compared to classical IEEE-754 32 bit (FP32) and 64 bit (FP64) arithmetic, these reduced precision arithmetic can naturally be sped up disproportional to their shortened width. The common strategy of all major hardware vendors is to aggressively further enhance their performance disproportionately. One particular FMA operation that multiplies two BF16 numbers while accumulating in FP32 has been found useful in deep learning, where BF16 is the 16bit floating point datatype with IEEE FP32 numerical range but 8 significant bits of precision. In this paper, we examine the use this FMA unit to implement higher-precision matrix routines in terms of potential performance gain and implications on accuracy. We demonstrate how a decomposition into multiple smaller datatypes can be used to assemble a high-precision result, leveraging the higher precision accumulation of the FMA unit. We first demonstrate that computations of vector inner products and by natural extension, matrix-matrix products can be achieved by decomposing FP32 numbers in several BF16 numbers followed by appropriate computations that can accommodate the dynamic range and preserve accuracy compared to standard FP32 computations, while projecting up to 5.2ˆspeed-up. Furthermore, we examine solution of linear equations formulated in the residual form that allows for iterative refinement. We demonstrate that the solution obtained to be comparable to those offered by FP64 under a large range of linear system condition numbers.

Low-power implementation of Mitchell's approximate logarithmic multiplication for convolutional neural networks

“…By truncating the less significant fractional bits, converting an FP32 value to bfloat16 generates a small negative error from 0% to -0.78% relative to the original FP32 value. The factors discussed in Section 3 also minimize the negative effects of this varying error, and they explain why using the full FP32 accumulator after bfloat16 multiplication produces the best results [24], in agreement with the observation that the accumulations need to be exact. The accumulation of mean error discussed in Section 4 should also be present, but the mean error of bfloat16 is too small to cause any problems for the studied CNNs.…”

Large-Scale Discrete Fourier Transform on TPUs

“…The inputs and outputs of MXU are float32 and the MAC operations on MXU are performed with bfloat16 [38]. However, one float32 number can be decomposed into multiple bfloat16 numbers and with appropriate accumulations, highprecision MAC operation can be achieved [39]. The implementation of both parallel algorithms in this work leverages the strategy of decomposition and accumulation and achieves the precision of float32.…”

Analysis of Posit and Bfloat Arithmetic of Real Numbers for Machine Learning

“…Matrix calculations based on floating point numbers and bfloats have been investigated by Intel engineers using GEMM (General matrix multiply) algorithms as the basis for exploring prospects of low precision computations in comparison with standard data types using more than 16 bits [15]. The authors claim that systolic arrays based on bfloat16 and float16 provide many times faster computations than standard 32-bit floats.…”