Error backpropagation using least absolute criterion

Abstract

The least squares error or L2 criterion approach has been commonly used in functional approximation and generalization in the error backpropagation algorithm. The purpose of this study is to present an absolute error criterion rather than the usual least squares error criterion for the sigmoidal backpropagation algorithm. We present the structure of the error function to be minimized and its derivatives with respect to the weights to be updated. The focus of the study is on the single hidden layer multilayer perceptron but the implementation may be extended to include two or more hidden layers. Our research makes use of the fact that the sigmoidal backpropagation function is differentiable and uses this property to implement a two-stage algorithm for non-linear L1 optimization by Madsen et al. [1991, Robust c subroutines for non-linear optimization, Institute for Numerical Analysis, Technical University of Denmark, Report N1-91-03.] to obtain the optimum result. This is a combination of a first-order method that approximates the solution by successive linear programming and a quasi-Newton method using approximate second-order information to solve the problem. The main reason for using the least absolute error criterion rather than the least squares error criterion is that the former is more robust and less easily affected by noise compared with the latter. To validate the performance and efficiency of the two-stage L1 algorithm, a comparison is made on the error made in both the two-stage L1 algorithm and the least squares error algorithm.