Two-stage extreme learning machine for regression

Yuan, Liang; Soh, Yeng Chai; Huang, Guang-Bin

doi:10.1016/j.neucom.2010.07.012

Cited by 92 publications

(37 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In an alternative extreme case, as seen here, a residual reduction is lower bounded by C log n if all basis functions are orthogonal. It is natural that |ξ k, α k | is an increasing function of n with a high probability when the number of linear independent basis functions increases with n. Actually, in a greedy version of the extreme learning machine, [16] first chooses candidates of basis functions by FPE in the forward selection while using PRESS for backward elimination. The requirement of the backward elimination may constitute empirical evidence for the above discussion.…”

Section: Greedy Algorithm For a Special Casementioning

confidence: 99%

“…On the other hand, the sparseness of TCR is notable compared to LOOCV, OSER, FPE and LASSO as found in Table 3. Since [16] employs LOOCV for the backward elimination, we can say that TCR can provide a sparser representation than in [16]. In Table 3, we can see that LASSO is not stable in terms of the degree of sparseness while it shows a relatively stable generalization performance; e.g.…”

Section: Some Benchmark Examplesmentioning

confidence: 99%

“…As a model selection method for OLS, [5], [6] applied the prediction sum of squares (PRESS) statistics [3], [21], which is also called the leave-one-out cross validation (LOOCV) error. On the other hand, for training multi-layer perceptron, some extensions of the extreme learning machine (ELM) [28] also employ greedy procedures [9], [16], [17]. In [17], a new hidden node is chosen from a set of randomly generated candidates and the choice is based on the degree of correlation of the basis function output to the residual.…”

Section: Introductionmentioning

confidence: 99%

“…As the model selection method in [17], C p [15] has been applied. In [16], several new hidden nodes are chosen from a set of randomly generated candidates and the choice is based on the degree of reduction of the residual sum of squares; i.e., this method is based on the least squares method. As the model selection method in [16], the final prediction error (FPE) [1] is applied in forward selection and PRESS is used in the backward elimination for candidates selected by FPE; i.e., this is Copyright c 2014 The Institute of Electronics, Information and Communication Engineers a hybrid method.…”

Section: Introductionmentioning

confidence: 99%

“…The model selection strategy is practically implemented by incorporating with the LOOCV method. The greedy procedure considered in this paper and in [18] is essentially the same as the algorithm in [16] except in terms of the introduction of a regularizer and the number of basis functions added in each step. [27] analyzed a condition for the consistency of feature selection for the greedy algorithm that is slightly different from the procedure considered here.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

Hagiwara

2014

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYRegularized forward selection is viewed as a method for obtaining a sparse representation in a nonparametric regression problem. In regularized forward selection, regression output is represented by a weighted sum of several significant basis functions that are selected from among a large number of candidates by using a greedy training procedure in terms of a regularized cost function and applying an appropriate model selection method. In this paper, we propose a model selection method in regularized forward selection. For the purpose, we focus on the reduction of a cost function, which is brought by appending a new basis function in a greedy training procedure. We first clarify a bias and variance decomposition of the cost reduction and then derive a probabilistic upper bound for the variance of the cost reduction under some conditions. The derived upper bound reflects an essential feature of the greedy training procedure; i.e., it selects a basis function which maximally reduces the cost function. We then propose a thresholding method for determining significant basis functions by applying the derived upper bound as a threshold level and effectively combining it with the leave-one-out cross validation method. Several numerical experiments show that generalization performance of the proposed method is comparable to that of the other methods while the number of basis functions selected by the proposed method is greatly smaller than by the other methods. We can therefore say that the proposed method is able to yield a sparse representation while keeping a relatively good generalization performance. Moreover, our method has an advantage that it is free from a selection of a regularization parameter.

show abstract

Section: Greedy Algorithm For a Special Casementioning

confidence: 99%

Section: Some Benchmark Examplesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

Hagiwara

2014

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

Unsupervised/Supervised Feature Extraction and Feature Selection for Multimedia Data (Feature extraction with feature selection for Image Forgery Detection)

Arun Anoop,

Karthikeyan,

Poonkuntran

2024

Supervised and Unsupervised Data Engineering for Multimedia Data

View full text Add to dashboard Cite

Evolutionary extreme learning machine for the behavioral modeling of RF power amplifiers

Xing

2019

Int J Numerical Modelling

View full text Add to dashboard Cite

In this paper, evolutionary extreme learning machine (E‐ELM) is first introduced for RF power amplifiers (PAs) behavioral modeling. This approach combined differential evolution (DE) and extreme learning machine (ELM) to effectively solve the problem that more neurons of hidden layer are required, and repeated trials are necessary in behavioral modeling PAs by conventional ELM. As revealed in the modeling practices on Class‐AB and Class‐E PAs, fewer hidden layer neurons are used than the condition of conventional ELM. Meanwhile, it is found that ELM's unstable generalization ability in modeling PAs is also significantly improved, thanks to the internal DE method in the E‐ELM.

show abstract

Two-stage extreme learning machine for regression

Cited by 92 publications

References 25 publications

A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

Unsupervised/Supervised Feature Extraction and Feature Selection for Multimedia Data (Feature extraction with feature selection for Image Forgery Detection)

Evolutionary extreme learning machine for the behavioral modeling of RF power amplifiers

Contact Info

Product

Resources

About