Solving the linear interval tolerance problem for weight initialization of neural networks

Abstract

Determining good initial conditions for an algorithm used to train a neural network is considered a parameter estimation problem dealing with uncertainty about the initial weights. Interval Analysis approaches model uncertainty in parameter estimation problems using intervals and formulating tolerance problems. Solving a tolerance problem is defining lower and upper bounds of the intervals so that the system functionality is guaranteed within predefined limits. The aim of this paper is to show how the problem of determining the initial weight intervals of a neural network can be defined in terms of solving a linear interval tolerance problem. The proposed Linear Interval Tolerance Approach copes with uncertainty about the initial weights without any previous knowledge or specific assumptions on the input data as required by approaches such as fuzzy sets or rough sets. The proposed method is tested on a number of well known benchmarks for neural networks trained with the back-propagation family of algorithms. Its efficiency is evaluated with regards to standard performance measures and the results obtained are compared against results of a number of well known and established initialization methods. These results provide credible evidence that the proposed method outperforms classical weight initialization methods.

Mutual information based weight initialization method for sigmoidal feedforward neural networks

“…In the case, the active region is generally the region where the derivative of the active function is greater than one-twentieth of the maximum derivative [18,19,21]. Then the active region is À4:36 r a r 4:36…”

“…To guarantee the performance of the algorithm, the input weights and biases of output neurons also should be designed carefully [14,21].…”

“…It has been demonstrated that the performance (convergence rate and training accuracy) of sigmoidal feedforward neural network (SFNN) trained with GB algorithms depends on several factors, including training algorithm, initial conditions, training data and network structure [5][6][7][8]. Use of appropriate weight initialization can shorten the training time and avoid the local minima caused by random initial weights [9][10][11][12][13][14][15][16][17][18][19][20][21][22].…”

“…Yang [22] proposed a weight initialization method based on the theory that the range of the initial values should be greater than the adjustment quantities. Adam [21] proposed a linear interval tolerance approach to guarantee the input of each hidden neuron be in the active region of sigmoid function. Talaska [17] proposed an initialization mechanism based on a Convex Combination method that is easy to be realized in transistor.…”

“…(1) If the initial weights are close to the global optimum, any descent algorithm can train the weights toward the optimum reliably [3]; (2) The network with optimal weights can fully reflect the relationship between the input variables and output variables [17]; (3) Restrain the range of initial weights reasonably can shorten the training time dramatically [16,21].…”