Reduction of required precision bits for back-propagation applied to pattern recognition

Abstract

The number of precision bits for operations and data are limited in the hardware implementations of backpropagation (BP). Reduction of rounding error due to this limited precision is crucial in the implementation. The new learning algorithm is based on overestimation of significant error in order to alleviate underflow and omission of weight updating for correctly recognized patterns. While the conventional BP algorithm minimizes the squared error between output signals and supervising data, the new learning algorithm minimizes the weighted error function. In the learning simulation of multifont capital recognition, this algorithm converged recognition accuracy to 100% with only 8-b precision. In addition, the recognition accuracy for characters that did not appear in the training data reached 94.9%. This performance is equivalent to that of a conventional BP with 12-b precision. Moreover, it is found that the performance of the weighted error function is high even when only a small number of hidden neurons is used. Consequently, the algorithm reduces the required amount of weight memory.

Worst case analysis of weight inaccuracy effects in multilayer perceptrons

“…The "Breast Cancer Wisconsin" dataset has been obtained from the UCI repository 1 and is a two-class classification problem. We trained a network with nine inputs and two hidden neurons on the 683 patterns of this dataset, obtained discarding the ones with missing values.…”

“…For these reasons, several methods have been proposed in the literature that deal with this problem. One possible solution aims at finding modified learning algorithms [1] that reduce the accuracy requirements of the network; a second, but equally important idea is to analyze the effect of weight errors in order to predict the final performance of the network. This has always been done using statistical or heuristic techniques [2]- [5] that can give an answer only in an average sense; furthermore, several limiting assumptions must be done (e.g., weight probability distributions, linearity conditions, etc.)…”

Analysis of quantization effects on high-order function neural networks

“…As a rule, designers must balance implementation accuracy and performance reliability. Several weightquantization techniques have been developed to further reduce the required accuracy without deterioration of the network performance [4,5].…”

Weight Quantization for Multi-layer Perceptrons Using Soft Weight Sharing

“…The aim is to get an MLP with weights which are represented by less number of bits, which consumes less memory at the expense of loss of precision; example studies are given in [2,3,4,5,6]. In our approach which is novel, we do not reduce the precision; the operations are still with full precision.…”