Low-precision Logarithmic Number Systems

Abstract

Logarithmic number systems (LNS) are used to represent real numbers in many applications using a constant base raised to a fixed-point exponent making its distribution exponential. This greatly simplifies hardware multiply, divide, and square root. LNS with base-2 is most common, but in this article, we show that for low-precision LNS the choice of base has a significant impact. We make four main contributions. First, LNS is not closed under addition and subtraction, so the result is approximate. We show that choosing a suitable base can manipulate the distribution to reduce the average error. Second, we show that low-precision LNS addition and subtraction can be implemented efficiently in logic rather than commonly used ROM lookup tables, the complexity of which can be reduced by an appropriate choice of base. A similar effect is shown where the result of arithmetic has greater precision than the input. Third, where input data from external sources is not expected to be in LNS, we can reduce the conversion error by selecting a LNS base to match the expected distribution of the input. Thus, there is no one base that gives the global optimum, and base selection is a trade-off between different factors. Fourth, we show that circuits realized in LNS require lower area and power consumption for short word lengths.

uLog: a software-based approximate logarithmic number system for computations on SIMD processors

“…However, most commercial computing systems including off-the-shelf processors support only IEEE 754 standard FP formats, i.e., double-precision (DP) with 64-bits length and single-precision (SP) with 32-bits length [40]. Therefore, FP numbers cannot use less than 32-bits in such systems, which restricts the reduction of data movement between memories and registers in applications where low-precision can be used such as machine learning (ML) algorithms [1,28,33,[36][37][38]. The SP is currently the dominant data type for deep learning training systems, and even recent studies have shown that low-precision FP formats such as 8-and 16-bits data types can be used [33,[36][37][38].…”

“…There are many works in the areas of signal processing [23], video processing [4], and neural networks [28] that have studied logarithmic number system (LNS) as an alternative FP number system for embedded applications [1]. LNS converts FP numbers to fixed-point numbers, which can be represented as an integer in off-the-shelf CPUs.…”

“…It converts integers by pairwise coprime number called moduli, and has simple addition, subtraction and multiplication but complex division and reverse conversion compared to LNS [6]. In LNS, multiplication, exponentiation, and square root operations are reduced to addition, subtraction, multiplication, and shift [1]. Therefore, emulation of these LNS operations in software has a low cost.…”

“…Moreover, Parhami compared different techniques and architectures for applying LNS operations efficiently by focusing on ELM as a case study, and showed that LNS has a high potential for deployment in general-purpose systems [32]. The non-uniform distribution of weights and activations in neural networks (NN) is the same as the distribution of numbers in LNS [1]. Therefore, in recent years, LNS has been explored for NN such as the work of Miyashita et al [28] which encoded the weights and activations of a trained network to 3 bits with base 2 logarithm representation and attained higher classification accuracy than fixed-point.…”

“…[28] also concluded that the total quantization error on NN weights of base √ 2 is 2X smaller than base 2. Recently, Alam et al [1] investigated the impact of choosing different logarithmic bases, and they found that a suitable base could change the distribution and reaccomplished duce the average error. Indeed, careful adjustment of the LNS base can provide a suitable tradeoff between precision and dynamic range [29].…”