Classification performance of three approaches for combining data sampling and gene selection on bioinformatics data

Abstract

Bioinformatics datasets pose two major challenges to researchers and data-mining practitioners: class imbalance and high dimensionality. Class imbalance occurs when instances of one class vastly outnumber instances of the other class(es), and high dimensionality occurs when a dataset has many independent features (genes). Data sampling is often used to tackle the problem of class imbalance, and the problem of excessive features in the dataset may be alleviated through feature selection. In this work, we examine various approaches for applying these techniques simultaneously to tackle both of these challenges and build effective classification models. In particular, we ask whether the order of these techniques and the use of unsampled or sampled datasets for building classification models makes a difference. We conducted an empirical study on a series of seven highdimensional and severely imbalanced biological datasets using six commonly used learners and four feature selection rankers from three different families of feature selection techniques. We compared three different data-sampling approaches: data sampling followed by feature selection using the unsampled data (DS-FS-UnSam) and selected features; data sampling followed by feature selection using the sampled data (DS-FS-Sam) and selected features; and feature selection followed by data sampling (FS-DS) using sampled data and selected features. We used Random Undersampling (RUS) to achieve the minority:majority class ratios of 35:65 and 50:50. The experimental results show that there are statistically significant differences among the three data-sampling approaches only when using the class ratio of 50:50, with a multiple comparison test showing that DS-FS-UnSam outperforms the other approaches. Thus, although specific combinations of learner and ranker may favor other approaches, across all choices of learner and ranker we would recommend the use of the DS-FS-UnSam approach for this class ratio. On the other hand, with the 35:65 class ratio, DS-FSSam was most frequently the top-performing approach, and although it was not statistically significantly better than the other approaches, we would generally recommend this approach be used for the 35:65 class ratio (although specific choices of learner and ranker may vary). Overall, we can see that the optimal approach will depend on the choice of class ratio.

Learning From High-Dimensional Biomedical Datasets: The Issue of Class Imbalance

“…The findings of this work are partially consistent with the results recently discussed in [21], where the effectiveness of combining RUS and feature selection is evaluated in conjunction with different classifiers and feature selection methods, but within less severe imbalance settings (min_pct > 10%). The beneficial impact of sampling-based approaches on high-dimensional bioinformatics datasets is also explored in [30]- [32]. In particular, [30] relies on both RUS and feature selection, and investigates the extent to which the order of these pre-processing operations impacts on the classification results.…”

“…The beneficial impact of sampling-based approaches on high-dimensional bioinformatics datasets is also explored in [30]- [32]. In particular, [30] relies on both RUS and feature selection, and investigates the extent to which the order of these pre-processing operations impacts on the classification results. As well, [31] exploits both RUS and feature selection and shows that using fully balanced data significantly improves the SVM performance in protein function prediction tasks.…”

“…While several research efforts have explored the issues of high dimensionality and class imbalance independently, only a few studies have addressed both the problems simultaneously [30]- [35]. Since several biomedical datasets are both high-dimensional and class-imbalanced, the aim of this work is to investigate the effectiveness of learning strategies that are designed to handle simultaneously both the issues, in order to effectively deal with real-world problems that involve the classification of rare pathological conditions (e.g., rare cancer types).…”