A simple and effective method for removal of hidden units and weights

Hagiwara, Masafumi

doi:10.1016/0925-2312(94)90055-8

Cited by 65 publications

(46 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in most cases people remove hidden neurons based on some criteria. For example, the sensitivity checking is a frequently used technique to find the importance of each weight or each hidden neuron [3][4][5], in particular, the magnitude-based pruning methods [10]. Sometimes, a penalty term is added to the error function [6,7] for the weight decay.…”

Section: Introductionmentioning

confidence: 99%

“…Many different approaches for the removal of hidden neurons have been presented in the past [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18]. Too many weights or hidden neurons may lead to overfitting of data and poor generalization, while too few weights and hidden neurons may not allow the neural network to learn the data sufficiently accurately.…”

Section: Introductionmentioning

confidence: 99%

“…The method does not simply take away the less important hidden neurons like most techniques [10]. Instead, an approach called the weight crosswise propagation (CP) is applied to mitigate the weight information loss to the least.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation

Liang

2006

Neural Comput & Applic

View full text Add to dashboard Cite

A new method of pruning away hidden neurons in neural networks is presented in this paper. The hidden neuron is removed by analyzing the orthogonal projection correlations among the outputs of other hidden neurons. The method guarantees the least loss of weight information in terms of orthogonal projection. The remaining weights and thresholds are updated based on the weight crosswise propagation. A practical technique for penalizing the superfluous hidden neurons is explored. Retraining is needed after pruning. Extensive experiments are conducted, and the results demonstrate that the method gives better initial points for retraining and retraining costs less epochs.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation

Liang

2006

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…However, in most cases people remove hidden neurons based on some criteria. For example, the sensitivity checking is a frequently used technique to find the importance of each weight or each hidden neuron [8,9], in particular, the magnitude-based pruning methods [10]. In [16], a method based on the correlation among the hidden output row vectors and the crosswise propagation (CP) the values of weights was presented.…”

Section: Introductionmentioning

confidence: 99%

An architecture-adaptive neural network online control system

Liang

Chen

Yang

2007

Neural Comput & Applic

View full text Add to dashboard Cite

An architecture-adaptive intelligent self-tuning control system is presented. The system is composed of the supervisor module, the model refinement module, the process plant and the database. In the supervisor module, the user prescribes the desired curve for the plant dynamic process. The model refinement module is in parallel with the process plant, and consists of the self-tuning process model, which contains an architecture-adaptive neural network. The model refinement module could learn intelligently the real process plant by the prompt adjustments based on the difference of the outputs of the two modules, and its learned model is also refined gradually. This diagram is especially versatile in the complex nonlinear and time-variant systems in practice.

show abstract

“…Thus, structural learning is closely related to improvement of the generalization ability. In structural learning, there are methods of successively adding hidden units [28,29], methods of eliminating unnecessary hidden units or links [30][31][32], methods of adding terms expressing the complexity of the structure as evaluation functions [33][34][35][36], and methods of using information criterions [37,38], among others.…”

Section: Introductionmentioning

confidence: 99%

Recurrent neural network with short‐term memory and fast structural learning method

Kikuchi

Nakanishi

2003

Systems & Computers in Japan

View full text Add to dashboard Cite

SUMMARYThe Elman network is used frequently in learning time series. However, it has several problems: (1) a comparatively long learning time, (2) learning is not necessarily successful, (3) lack of discussions on a generalization ability, and (4) the necessity of deciding unit numbers and the like by trial and error. In this study, a network providing short-term memory for the Elman network in order to facilitate learning of time series is proposed. In addition, an algorithm considering both structural learning and fast learning, which have been regarded as having a trade-off relationship in the past, is proposed as a learning algorithm. It has been verified by tests that the learning time, convergence rate, and generalization ability are improved by the proposed method.

show abstract

A simple and effective method for removal of hidden units and weights

Cited by 65 publications

References 5 publications

Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation

Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation

An architecture-adaptive neural network online control system

Recurrent neural network with short‐term memory and fast structural learning method

Contact Info

Product

Resources

About