Evaluating Binary Encoding Techniques for WiSARD

Kappaun, Andressa; Camargo, Karine Silva; Rangel, Fabio; Firmino, Fabrício; Lima, Priscila M. V.; Oliveira, Jonice

doi:10.1109/bracis.2016.029

Cited by 12 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditionally, WNNs represent their inputs as 1-bit values, where an input is 1 if it rises above some pre-determined threshold 3 and 0 otherwise. However, it is frequently advantageous to use more sophisticated encodings, where each parameter is represented using multiple bits [26]. Integer encodings are not a good choice for WiSARD, since individual bits carry dramatically different amounts of information.…”

Section: E Thermometer Encodingmentioning

confidence: 99%

Weightless Neural Networks for Efficient Edge Inference

Susskind¹,

Arora²,

Miranda³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Weightless Neural Networks (WNNs) are a class of machine learning model which use table lookups to perform inference. This is in contrast with Deep Neural Networks (DNNs), which use multiply-accumulate operations. State-of-the-art WNN architectures have a fraction of the implementation cost of DNNs, but still lag behind them on accuracy for common image recognition tasks. Additionally, many existing WNN architectures suffer from high memory requirements. In this paper, we propose a novel WNN architecture, BTHOWeN, with key algorithmic and architectural improvements over prior work, namely counting Bloom filters, hardware-friendly hashing, and Gaussian-based nonlinear thermometer encodings to improve model accuracy and reduce area and energy consumption. BTHOWeN targets the large and growing edge computing sector by providing superior latency and energy efficiency to comparable quantized DNNs. Compared to state-of-the-art WNNs across nine classification datasets, BTHOWeN on average reduces error by more than than 40% and model size by more than 50%. We then demonstrate the viability of the BTHOWeN architecture by presenting an FPGAbased accelerator, and compare its latency and resource usage against similarly accurate quantized DNN accelerators, including Multi-Layer Perceptron (MLP) and convolutional models. The proposed BTHOWeN models consume almost 80% less energy than the MLP models, with nearly 85% reduction in latency. In our quest for efficient ML on the edge, WNNs are clearly deserving of additional attention.

show abstract

Section: E Thermometer Encodingmentioning

confidence: 99%

Weightless Neural Networks for Efficient Edge Inference

Susskind¹,

Arora²,

Miranda³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Proper binary encoding of real valued inputs is a crucial part of a WNN model, and a naive approach is detrimental to learning. As WiSARD learns from similar patterns in data, making a certain number of bit flips in the encoded input must directly correspond to a similar change in their actual values as well [6].…”

Section: Encodingsmentioning

confidence: 99%

“…Binary representation: It is simply using the input binary representation (i.e., integer or floating-point). Although this seems like a good representation to use, it is not a binarization scheme that is appropriate for the WiSARD model, as a single bit flip can yield a large change in value, instead of a close one, because not all bits in the representation convey similar weights [6].…”

Section: Encodingsmentioning

confidence: 99%

“…As using RAMs implicitly requires inputs to be binary, the encoding of real valued inputs is a crucial part of a WNN model and a naive approach can be detrimental to learning [6]. The literature presents many binary encoding techniques [6], with the linear thermometer [7] being the most prominent one. The linear thermometer works by encoding the real value inputs in unary code, under a uniform distribution assumption.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Distributive Thermometer: A New Unary Encoding for Weightless Neural Networks

Bacellar¹,

Susskind²,

Villon³

et al. 2022

ESANN 2022 Proceedings

View full text Add to dashboard Cite

The binary encoding of real valued inputs is a crucial part of Weightless Neural Networks. The Linear Thermometer and its variations are the most prominent methods to determine binary encoding for input data but, as they make assumptions about the input distribution, the resulting encoding is sub-optimal and possibly wasteful when the assumption is incorrect. We propose a new thermometer approach that doesn't require such assumptions. Our results show that it achieves similar or better accuracy when compared to a thermometer that correctly assumes the distribution, and accuracy gains up to 26.3% when other thermometer representations assume an unsound distribution.

show abstract

“…Thermometer Encoding: Traditionally, inputs to WiSARD are binarized by comparing them against their mean value in the training data. A thermometer encoding is a multi-bit unary encoding which instead compares values against a series of increasing thresholds [5]. Thermometer encoding gives increased resolution and accuracy at the cost of model size.…”

Section: Introductionmentioning

confidence: 99%

Pruning Weightless Neural Networks

Susskind¹,

Bacellar²,

Arora³

et al. 2022

ESANN 2022 Proceedings

View full text Add to dashboard Cite

Weightless neural networks (WNNs) are a type of machine learning model which perform prediction using lookup tables (LUTs) instead of arithmetic operations. Recent advancements in WNNs have reduced model sizes and improved accuracies, reducing the gap in accuracy with deep neural networks (DNNs). Modern DNNs leverage "pruning" techniques to reduce model size, but this has not previously been explored for WNNs. We propose a WNN pruning strategy based on identifying and culling the LUTs which contribute least to overall model accuracy. We demonstrate an average 40% reduction in model size with at most 1% reduction in accuracy.

show abstract

Evaluating Binary Encoding Techniques for WiSARD

Cited by 12 publications

References 12 publications

Weightless Neural Networks for Efficient Edge Inference

Weightless Neural Networks for Efficient Edge Inference

Distributive Thermometer: A New Unary Encoding for Weightless Neural Networks

Pruning Weightless Neural Networks

Contact Info

Product

Resources

About