A k-Nearest Neighbours Based Ensemble via Optimal Model Selection for Regression

Ali, Amjad; Hamraz, Muhammad; Kumam, Poom; Khan, Dost Muhammad; Khalil, Umair; Sulaiman, Muhammad; Khan, Zardad

doi:10.1109/access.2020.3010099

Cited by 31 publications

(12 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This might be very helpful in high dimensional settings. Model selection as given in [50] could also be incorporated for further improvements. Furthermore, considering the ideas of feature engineering and feature weighting [51]- [57], in conjunction with the proposed method, might open further research avenues for improved classification and prediction.…”

Section: Discussionmentioning

confidence: 99%

A Weighted k-Nearest Neighbours Ensemble With Added Accuracy and Diversity

et al. 2022

Self Cite

View full text Add to dashboard Cite

Ensembles based on kNN models are considered effective in reducing the adverse effect of outliers, primarily, by identifying the closest observations to a test point in a given training data. Class label of the test point is estimated by taking a majority vote of the nearest observations' class labels. While identifying the closest observations, certain training patterns might possess high regulatory power than the others. Therefore, assigning weights to observations and then calculating weighted distances are deemed important. This paper proposes a kNN ensemble that identifies nearest observations based on their weighted distance in relation to the response variable via support vectors. This is done while building a large number of kNN models each on a bootstrap sample from the training data along with a randomly selected subset of features from the given feature space. The estimated class of the test observation is decided via majority voting based on the estimates given by all the base kNN models.The ensemble is assessed on 14 benchmark and simulated datasets against other classical methods, including kNN based models using Brier score, classification accuracy and Kappa as performance measures. Average values of the given measures along with boxplots are given for the proposed and the other competitors. On both benchmark and simulated datasets, the proposed ensemble outperformed the other competitive methods in majority of the cases. The suggested ensemble gave overall better classification performance than the other methods on 8 datasets. The ensemble performed better than the other methods in simulation study that considered the presence of noisy features in a dataset. The analyses reveal that the proposed method is effective in classification problems that involve noisy features in the data. Furthermore, feature weighting and randomization also make the method robust to the choice of k, i.e., the number of nearest observations in a base model.INDEX TERMS Classification, feature weighting, k-nearest neighbor ensemble, support vectors.

show abstract

Section: Discussionmentioning

confidence: 99%

A Weighted k-Nearest Neighbours Ensemble With Added Accuracy and Diversity

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, there are several ensembles based on kNN using different approaches for accurately predicting test data. The optimal kNN ensemble given in [48] fits a step-wise regression model on k nearest observations in each base kNN for a test point. Tang and Haibo [49] have proposed a method which estimates test data class labels according to the maximum gain of intra-class coherence.…”

Section: Related Workmentioning

confidence: 99%

A k nearest neighbours classifiers ensemble based on extended neighbourhood rule and features subsets

Ali¹,

Hamraz²,

Gul³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

kNN based ensemble methods minimise the effect of outliers by identifying a set of data points in the given feature space that are nearest to an unseen observation in order to predict its response by using majority voting. The ordinary ensembles based on kNN find out the k nearest observations in a region (bounded by a sphere) based on a predefined value of k. This scenario, however, might not work in situations when the test observation follows the pattern of the closest data points with the same class that lie on a certain path not contained in the given sphere. This paper proposes a k nearest neighbour ensemble where the neighbours are determined in k steps. Starting from the first nearest observation of the test point, the algorithm identifies a single observation that is closest to the observation at the previous step. At each base learner in the ensemble, this search is extended to k steps on a random bootstrap sample with a random subset of features selected from the feature space. The final predicted class of the test point is determined by using a majority vote in the predicted classes given by all base models. This new ensemble method is applied on 17 benchmark datasets and compared with other classical methods, including kNN based models, in terms of classification accuracy, kappa and Brier score as performance metrics. Boxplots are also utilised to illustrate the difference in the results given by the proposed and other state-of-the-art methods. The proposed method outperformed the rest of the classical methods in the majority of cases. The paper gives a detailed simulation study for further assessment.

show abstract

“…In recent times, ANNs are used as universal function approximation procedures to develop stochastic numerical techniques. Due to their strength and stability, they are widely used for the solutions of variety of real world problems including multi-phase flow through porous media for imbibition phenomena [24], longitudinal heat transformation fins model [25], [26], Beam-Column designs [27], Optimal Model Selection for Regression [28], fractional models of damping material [26], nonlinear dusty plasma system [29], corneal Model for Eye Surgery [30] and temperature profile of porous fin model [31]. A plant propagation algorithm (PPA) and its modified version were developed to solve design engineering problems [32]- [35].…”

Section: Introductionmentioning

confidence: 99%

Analysis of Third-Order Nonlinear Multi-Singular Emden–Fowler Equation by Using the LeNN-WOA-NM Algorithm

Zhang¹,

Lin²,

Hu³

et al. 2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

In this paper, a novel soft computing algorithm is designed for the numerical solution of third-order nonlinear multi-singular Emden-Fowler equation (TONMS-EFE) using the strength of universal approximation capabilities of Legendre polynomials based Legendre neural networks supported with optimization power of the Whale Optimization Algorithm (WOA) and Nelder-Mead (NM) algorithm. Unsupervised error functions are constructed in terms of mean square error for governing TONMS-EF equations of first and second order. Unknown designed parameters in LeNN structure are optimized initially by WOA for global search while NM algorithm further enhances the rapid local search convergence. The proposed algorithm's objective is to show the accuracy and robustness in solving challenging problems like TONMS-EFE. To study our designed scheme's performance and effectiveness, LeNN-WOA-NM is implemented on four cases of TONMS-EFE. The results obtained by the proposed algorithm are compared with the Particle Swarm Optimization (PSO) algorithm, Cuckoo search algorithm (CSA), and WOA. Extensive graphical and statistical analysis for fitness value, absolute errors, and performance indicators in terms of mean, median, and standard deviations show the proposed algorithm's efficiency and accuracy.INDEX TERMS Singular Emden-Fowler equation, Soft computing algorithm, Weighted Legendre neural networks, Nelder-Mead algorithm, Whale optimization algorithm.

show abstract

A k-Nearest Neighbours Based Ensemble via Optimal Model Selection for Regression

Cited by 31 publications

References 69 publications

A Weighted k-Nearest Neighbours Ensemble With Added Accuracy and Diversity

A Weighted k-Nearest Neighbours Ensemble With Added Accuracy and Diversity

A k nearest neighbours classifiers ensemble based on extended neighbourhood rule and features subsets

Analysis of Third-Order Nonlinear Multi-Singular Emden–Fowler Equation by Using the LeNN-WOA-NM Algorithm

Contact Info

Product

Resources

About