Robustness of chemometrics-based feature selection methods in early cancer detection and biomarker discovery

Lee, Hae Woo; Lawton, Carl; Na, Young Jeong; Yoon, Seongkyu

doi:10.1515/sagmb-2012-0067

Cited by 11 publications

(6 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In biomedical fields, this is a proxy for reproducible research, in the latter case indicating that the biological features the method has found are likely to be a data artifact, not a real clinical signal worth pursuing with further resources (Lee et al. , 2013 ). Goh and Wong ( 2016 ) recommend augmenting statistical feature selection methods with concurrent analysis on stability and reproducibility to improve the quality of selected features prior to experimental validation (Sze and Schloss, 2016 ; Duvallet et al.…”

Section: Methodsmentioning

confidence: 99%

“…Conversely, if small changes to the data result in significantly different feature subsets, then this method is considered unstable, and we should not trust the output as reflective of the true underlying structure influencing the outcome being predicted. In biomedical fields, this is a proxy for reproducible research, in the latter case indicating that the biological features the method has found are likely to be a data artifact, not a real clinical signal worth pursuing with further resources (Lee et al, 2013). Goh and Wong (2016) recommend augmenting statistical feature selection methods with concurrent analysis on stability and reproducibility to improve the quality of selected features prior to experimental validation (Sze and Schloss, 2016;Duvallet et al, 2017).…”

Section: Estimation Of Stabilitymentioning

confidence: 99%

See 1 more Smart Citation

Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data

Jiang

Haiminen²,

Carrieri³

et al. 2021

Biometrics

View full text Add to dashboard Cite

Feature selection is indispensable in microbiome data analysis, but it can be particularly challenging as microbiome data sets are high dimensional, underdetermined, sparse and compositional. Great efforts have recently been made on developing new methods for feature selection that handle the above data characteristics, but almost all methods were evaluated based on performance of model predictions. However, little attention has been paid to address a fundamental question: how appropriate are those evaluation criteria? Most feature selection methods often control the model fit, but the ability to identify meaningful subsets of features cannot be evaluated simply based on the prediction accuracy. If tiny changes to the data would lead to large changes in the chosen feature subset, then many selected features are likely to be a data artifact rather than real biological signal. This crucial need of identifying relevant and reproducible features motivated the reproducibility evaluation criterion such as Stability, which quantifies how robust a method is to perturbations in the data. In our paper, we compare the performance of popular model prediction metrics (MSE or AUC) with proposed reproducibility criterion Stability in evaluating four widely used feature selection methods in both simulations and experimental microbiome applications with continuous or binary outcomes. We conclude that Stability is a preferred feature selection criterion over model prediction metrics because it better quantifies the reproducibility of the feature selection method.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Estimation Of Stabilitymentioning

confidence: 99%

Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data

Jiang

Haiminen²,

Carrieri³

et al. 2021

Biometrics

View full text Add to dashboard Cite

show abstract

“…According to (Nogueira, Sechidis, and Brown, 2018) the measurement of stability addresses the question — how much we can trust the algorithm? From biomedical standpoint, it is crucial to guarantee the reproducibility of the given feature selection methods when finding proper sets of biomarkers (Lee et al, 2013)…”

Section: Introductionmentioning

confidence: 99%

Stability of feature selection utilizing Graph Convolutional Neural Network and Layer-wise Relevance Propagation

Chereda

Leha

Beißbarth

2021

Preprint

View full text Add to dashboard Cite

Motivation: High-throughput technologies play a more and more significant role in discovering prognostic molecular signatures and identifying novel drug targets. It is common to apply Machine Learning (ML) methods to classify high-dimensional gene expression data and to determine a subset of features (genes) that is important for decisions of a ML model. One feature subset of important genes corresponds to one dataset and it is essential to sustain the stability of feature sets across different datasets with the same clinical endpoint since the selected genes are candidates for prognostic biomarkers. The stability of feature selection can be improved by including information of molecular networks into ML methods. Gene expression data can be assigned to the vertices of a molecular network's graph and then classified by a Graph Convolutional Neural Network (GCNN). GCNN is a contemporary deep learning approach that can be applied to graph-structured data. Layer-wise Relevance Propagation (LRP) is a technique to explain decisions of deep learning methods. In our recent work we developed Graph Layer-wise Relevance Propagation (GLRP) --- a method that adapts LRP to a graph convolution and explains patient-specific decisions of GCNN. GLRP delivers individual molecular signatures as patient-specific subnetworks that are parts of a molecular network representing background knowledge about biological mechanisms. GLRP gives a possibility to deliver the subset of features corresponding to a dataset as well, so that the stability of feature selection performed by GLRP can be measured and compared to that of other methods. Results: Utilizing two large breast cancer datasets, we analysed properties of feature sets selected by GLRP (GCNN+LRP) such as stability and permutation importance. We have implemented a graph convolutional layer of GCNN as a Keras layer so that the SHAP (SHapley Additive exPlanation) explanation method could be also applied to a Keras version of a GCNN model. We compare the stability of feature selection performed by GCNN+LRP to the stability of GCNN+SHAP and to other ML based feature selection methods. We conclude, that GCNN+LRP shows the highest stability among other feature selection methods including GCNN+SHAP. It was established that the permutation importance of features among GLRP subnetworks is lower than among GCNN+SHAP subnetworks, but in the context of the utilized molecular network, a GLRP subnetwork of an individual patient is on average substantially more connected (and interpretable) than a GCNN+SHAP subnetwork, which consists mainly of single vertices.

show abstract

“…in bioinformatics applications -if the alteration/exclusion of just one training example results in a very different choice of biomarkers, we cannot justifiably say the FS is doing a reliable job. In early cancer detection, stability of the identified markers is a strong indicator of reproducible research [6], [12] and therefore selecting a stable set of markers is said to be equally important as their predictive power [7].…”

Section: Introductionmentioning

confidence: 99%

Measuring the Stability of Feature Selection

Nogueira

Brown

2016

Lecture Notes in Computer Science

102

View full text Add to dashboard Cite

Abstract. In feature selection algorithms, "stability" is the sensitivity of the chosen feature set to variations in the supplied training data. As such it can be seen as an analogous concept to the statistical variance of a predictor. However unlike variance, there is no unique definition of stability, with numerous proposed measures over 15 years of literature. In this paper, instead of defining a new measure, we start from an axiomatic point of view and identify what properties would be desirable. Somewhat surprisingly, we find that the simple Pearson's correlation coefficient has all necessary properties, yet has somehow been overlooked in favour of more complex alternatives. Finally, we illustrate how the use of this measure in practice can provide better interpretability and more confidence in the model selection process.

show abstract

Robustness of chemometrics-based feature selection methods in early cancer detection and biomarker discovery

Cited by 11 publications

References 54 publications

Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data

Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data

Stability of feature selection utilizing Graph Convolutional Neural Network and Layer-wise Relevance Propagation

Measuring the Stability of Feature Selection

Contact Info

Product

Resources

About