Computational Experience with Pseudoinversion-Based Training of Neural Networks Using Random Projection Matrices

Abstract

Recently some novel strategies have been proposed for neural network training that set randomly the weights from input to hidden layer, while weights from hidden to output layer are analytically determined by Moore-Penrose generalised inverse; such non-iterative strategies are appealing since they allow fast learning. Aim of this study is to investigate the performance variability when random projections are used for convenient setting of the input weights: we compare them with state of the art setting i.e. weights randomly chosen according to a continuous uniform distribution. We compare the solutions obtained by different methods testing this approach on some UCI datasets for both regression and classification tasks; this results in a significant performance improvement with respect to conventional method.

Local search and pseudoinversion: an hybrid approach to neural network training

“…Founding on the results obtained in (Rubini et al, 2014), we also investigated the effectivness of initializing inputs weigths C not only with the standard uniform-random setting, but also using random projections.…”

“…In (Rubini et al, 2014) we performed three series of experiments where the input weights and biases are selected according to (i) a conventional strategy, where c ij is sampled from a uniform random distribution in the interval [−1, 1] or (ii) using random projection matrices belonging to two different types, respectively with elements c ij gaussian distributed, with mean value zero and variance 1 or sparsely distributed according to eq. ( 9).…”

Conditioning Theory for ML-Weighted Pseudoinverse and ML-Weighted Least Squares Problem

“…The study of generalized inverses of matrices has been a very important research field since the middle of last century and remains one of the most active research branches in the world [1][2][3]. Generalized inverses, including the weighted pseudoinverse, have numerous applications in various fields, such as control, networks, statistics, and econometrics [4][5][6][7]. The ℳℒ -weighted pseudoinverse of m × n matrix 𝒦 with the entries of two weight matrices ℳ and ℒ (with order s × m and l × n, respectively) is defined as…”