Fast Approximations of Activation Functions in Deep Neural Networks when using Posit Arithmetic

Abstract

With increasing real-time constraints being put on the use of Deep Neural Networks (DNNs) by real-time scenarios, there is the need to review information representation. A very challenging path is to employ an encoding that allows a fast processing and hardware-friendly representation of information. Among the proposed alternatives to the IEEE 754 standard regarding floating point representation of real numbers, the recently introduced Posit format has been theoretically proven to be really promising in satisfying the mentioned requirements. However, with the absence of proper hardware support for this novel type, this evaluation can be conducted only through a software emulation. While waiting for the widespread availability of the Posit Processing Units (the equivalent of the Floating Point Unit (FPU)), we can already exploit the Posit representation and the currently available Arithmetic-Logic Unit (ALU) to speed up DNNs by manipulating the low-level bit string representations of Posits. As a first step, in this paper, we present new arithmetic properties of the Posit number system with a focus on the configuration with 0 exponent bits. In particular, we propose a new class of Posit operators called L1 operators, which consists of fast and approximated versions of existing arithmetic operations or functions (e.g., hyperbolic tangent (TANH) and extended linear unit (ELU)) only using integer arithmetic. These operators introduce very interesting properties and results: (i) faster evaluation than the exact counterpart with a negligible accuracy degradation; (ii) an efficient ALU emulation of a number of Posits operations; and (iii) the possibility to vectorize operations in Posits, using existing ALU vectorized operations (such as the scalable vector extension of ARM CPUs or advanced vector extensions on Intel CPUs). As a second step, we test the proposed activation function on Posit-based DNNs, showing how 16-bit down to 10-bit Posits represent an exact replacement for 32-bit floats while 8-bit Posits could be an interesting alternative to 32-bit floats since their performances are a bit lower but their high speed and low storage properties are very appealing (leading to a lower bandwidth demand and more cache-friendly code). Finally, we point out how small Posits (i.e., up to 14 bits long) are very interesting while PPUs become widespread, since Posit operations can be tabulated in a very efficient way (see details in the text).

Vectorizing posit operations on RISC-V for faster deep neural networks: experiments and comparison with ARM SVE

“…As shown by the authors in [13], the posit format gains interesting properties when configured with esbits ¼ 0. This particular configuration allows the implementation of fast versions of common operations (a little approximation is introduced in some of them); the new versions can be computed just by using the arithmetic-logic unit (ALU) of the CPU since they only involve bit manipulation and integer operations.…”

GBRAS-Net: A Convolutional Neural Network Architecture for Spatial Image Steganalysis

“…There are six convolutional layers with (3, 3) filters, and at the end of the stage, there are two convolutional layers with kernel_size (1, 1). The activation function used is Exponential Linear Unit (ELU) [31], [32] for all convolutions and separable convolutions.…”

La Aritmética del Futuro: una Reflexión sobre los Planes de Estudio

“…Bajo ciertas configuraciones (especialmente cuando es = 0), los posits permiten realizar aproximaciones rápidas (realizadas a nivel de bit, sin necesidad de decodificar todo el número) de funciones elementales pero complejas, como el recíproco, y de funciones no elementales, como la sigmoide o la tangente hiperbólica (estasúltimas son muy utilizadas en elárea de la DNNs) [32].…”