Matrix Pseudoinversion for Image Neural Processing

Cancelliere, R.; Gai, M.; Artières, Thierry; Gallinari, Patrick

doi:10.1007/978-3-642-34500-5_15

Cited by 3 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When approaching critical size, inversion of singular values causes a wrong evaluation of H + and therefore a significant growth in the error; when the critical dimension is reached, singular values under threshold are automatically removed, thus allowing the subsequent decrease of error. The same trend was detected analysing the astronomical dataset in (Cancelliere et al, 2012).…”

Section: Experiments and Resultssupporting

confidence: 80%

“…Some numerical instability issues have already been evidenced in our previous investigations (Cancelliere et al, 2012); we provided suggestions on possible mitigation techniques like selection of a convenient activation function and normalisation of the input weights. Hereafter we show that adding regularisation to the implementation prescriptions already analysed provides a convenient and effective approach to deal with such problem.…”

Section: Experiments and Resultsmentioning

confidence: 88%

“…A possible explanation is that the setting of input weights in the interval (-1, 1) may allow 'specialisation' of some hidden neurons on particular training instances, thus creating a sort of network partition, carried out thanks to saturation. On the contrary, when weights are randomly selected in the interval (−1/ √ M , 1/ √ M ), as for HypT-unreg and Sigm-unreg cases, input weights are automatically kept small when the network size increases, thus exploiting the central part of both activation functions: consequently saturation is avoided (Cancelliere et al, 2012).…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…To further mitigate saturation issues, in our previous work (Cancelliere et al, 2012) we selected input weights according to a uniform random distribution in the range…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…One aim of this paper is the analysis of these instability issues; a preliminary assessment of the context and our initial results are discussed in (Cancelliere et al, 2012). Here we present further advances on the theoretical framework and we propose a novel approach to carry out a more efficient learning, showing how singular values of SVD can be used to detect the occurrence of numerical instability.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

An analysis of numerical issues in neural training by pseudoinversion

Cancelliere

Deluca

Gai³

et al. 2015

Comp. Appl. Math.

View full text Add to dashboard Cite

Some novel strategies have recently been proposed for single hidden layer neural network training that set randomly the weights from input to hidden layer, while weights from hidden to output layer are analytically determined by pseudoinversion. These techniques are gaining popularity in spite of their known numerical issues when singular and/or almost singular matrices are involved. In this paper we discuss a critical use of Singular Value Analysis for identification of these drawbacks and we propose an original use of regularisation to determine the output weights, based on the concept of critical hidden layer size. This approach also allows to limit the training computational effort. Besides, we introduce a novel technique which relies an effective determination of input weights to the hidden layer dimension. This approach is tested for both regression and classification tasks, resulting in a significant performance improvement with respect to alternative methods.

show abstract

Section: Experiments and Resultssupporting

confidence: 80%

Section: Experiments and Resultsmentioning

confidence: 88%

Section: Experiments and Resultsmentioning

confidence: 99%

“…To further mitigate saturation issues, in our previous work (Cancelliere et al, 2012) we selected input weights according to a uniform random distribution in the range…”

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An analysis of numerical issues in neural training by pseudoinversion

Cancelliere

Deluca

Gai³

et al. 2015

Comp. Appl. Math.

View full text Add to dashboard Cite

show abstract

OCReP: An Optimally Conditioned Regularization for pseudoinversion based neural training

Cancelliere

Gai

Gallinari³

et al. 2015

Neural Networks

Self Cite

View full text Add to dashboard Cite

In this paper we consider the training of single hidden layer neural networks by pseudoinversion, which, in spite of its popularity, is sometimes affected by numerical instability issues. Regularization is known to be effective in such cases, so that we introduce, in the framework of Tikhonov regularization, a matricial reformulation of the problem which allows us to use the condition number as a diagnostic tool for identification of instability. By imposing well-conditioning requirements on the relevant matrices, our theoretical analysis allows the identification of an optimal value for the regularization parameter from the standpoint of stability. We compare with the value derived by cross-validation for overfitting control and optimization of the generalization performance. We test our method for both regression and classification tasks. The proposed method is quite effective in terms of predictivity, often with some improvement on performance with respect to the reference cases considered. This approach, due to analytical determination of the regularization parameter, dramatically reduces the computational load required by many other techniques.

show abstract