On the Selection of Weight Decay Parameter for Faulty Networks

Abstract

Abstract-The weight-decay technique is an effective approach to handle overfitting and weight fault. For fault-free networks, without an appropriate value of decay parameter, the trained network is either overfitted or underfitted. However, many existing results on the selection of decay parameter focus on fault-free networks only. It is well known that the weight-decay method can also suppress the effect of weight fault. For the faulty case, using a test set to select the decay parameter is not practice because there are huge number of possible faulty networks for a trained network. This paper develops two mean prediction error (MPE) formulae for predicting the performance of faulty radial basis function (RBF) networks. Two fault models, multiplicative weight noise and open weight fault, are considered. Our MPE formulae involve the training error and trained weights only. Besides, in our method, we do not need to generate a huge number of faulty networks to measure the test error for the fault situation. The MPE formulae allow us to select appropriate values of decay parameter for faulty networks. Our experiments showed that, although there are small differences between the true test errors (from the test set) and the MPE values, the MPE formulae can accurately locate the appropriate value of the decay parameter for minimizing the true test error of faulty networks.

Effect of Additive Noise for Multi-Layered Perceptron with AutoEncoders

“…Many regularizing methods focus on optimizing network target function during the training [1], [4]- [7]. One commonly used approach is introducing training criterion that discourages converging the network into building a complicated solution and suppresses the number of effective weights by directing the learning algorithm towards a solution with the least possible amount of zero weights such as weight decay [8]. Other approaches encourage less co-adapting between units and reduces overfitting such as Dropout [3].…”

Control del sobreajuste en redes neuronales tipo cascada correlación aplicado a la predicción de precios de contratos de electricidad

“…La función general de regularización está dada por la minimización del riesgo total [15] (ecuación 1): La descomposición de pesos es una de las estrategias de regularización más utilizadas en la literatura [16], dado que su implementación es computacionalmente sencilla, no depende de parámetros adicionales y permite mejorar la capacidad de generalización de la red neuronal. Esta técnica fue propuesta por [8] y también trabajada por [17] bajo el nombre de regresión de borde (ridge regression).…”

Objective Functions of Online Weight Noise Injection Training Algorithms for MLPs

“…For small S b , 1 we can assume thatw is close to w. The Taylor series expansions of f (·, ·) and g(·, ·) are given by , w(t)) + G(x t , w(t))(b ⊗ w(t)). (11) Putting (10) and (11) into (9), and using (4), we can show that h(w(t)) in (8) is given by…”

“…It is normally required in these applications that delayed HNNs (DHNNs) be globally asymptotically stable. In the past decade a large number of publications on the issue of stability of DHNN have appeared (see [7]- [11] and the cited references therein).…”