Optimally splitting cases for training and testing high dimensional classifiers

Abstract

BackgroundWe consider the problem of designing a study to develop a predictive classifier from high dimensional data. A common study design is to split the sample into a training set and an independent test set, where the former is used to develop the classifier and the latter to evaluate its performance. In this paper we address the question of what proportion of the samples should be devoted to the training set. How does this proportion impact the mean squared error (MSE) of the prediction accuracy estimate?ResultsWe develop a non-parametric algorithm for determining an optimal splitting proportion that can be applied with a specific dataset and classifier algorithm. We also perform a broad simulation study for the purpose of better understanding the factors that determine the best split proportions and to evaluate commonly used splitting strategies (1/2 training or 2/3 training) under a wide variety of conditions. These methods are based on a decomposition of the MSE into three intuitive component parts.ConclusionsBy applying these approaches to a number of synthetic and real microarray datasets we show that for linear classifiers the optimal proportion depends on the overall number of samples available and the degree of differential expression between the classes. The optimal proportion was found to depend on the full dataset size (n) and classification accuracy - with higher accuracy and smaller n resulting in more assigned to the training set. The commonly used strategy of allocating 2/3rd of cases for training was close to optimal for reasonable sized datasets (n ≥ 100) with strong signals (i.e. 85% or greater full dataset accuracy). In general, we recommend use of our nonparametric resampling approach for determing the optimal split. This approach can be applied to any dataset, using any predictor development method, to determine the best split.

Shedding Light on the Role of Sample Sizes and Splitting Proportions in Out-of-Sample Tests: A Monte Carlo Cross-Validation Approach

“…However, the question whether the frequently applied rule-of-thumb of using 2/3 of the data for training and the remaining 1/3 for testing purposes (Cios et al, 2007;Dobbin & Simon, 2011) makes sense and whether its applicability depends on the sample size often remains an open question. Because of this, we stated two research questions: First, does the popular 2/3 rule-ofthumb splitting criterion used in out-of-sample tests generally make sense?…”

“…Thus, these sub-samples are often referred to as the training-and test samples. However, how could someone decide between -say splitting the data sample into equally sized sub-samples and the oftentimes used 2/3 rule-of-thumb (Cios, Pedrycz, Swiniarski, & Kurgan, 2007;Dobbin & Simon, 2011) of using 2/3 of the data for training and the remaining 1/3 for testing purposes? Thus, we state our two research questions as follows:…”

“…Thus, many researchers use heuristics such as the popular 2/3 rule-of-thumb (see for example Cios, Pedrycz, Swiniarski and Kurgan, 2007;Dobbin and Simon, 2011), suggesting to use 2/3 of the data sample for training and the remaining 1/3 for testing purposes.…”

Feature Selection based on Hybrid Binary Cuckoo Search and Rough Set Theory in Classification for Nominal Datasets

“…A training set that has about 70% of the dataset objects, and a learning and test set has about 30% of the dataset objects [44]. The training set is used by FS approaches to achieve features reduction.…”

Clinical prediction rule for delayed hemothorax after minor thoracic injury: a multicentre derivation and validation study

“…The number of rib fractures was recorded from radiologists' reports, and the fractures themselves could have been misclassified, given that radiographic diagnosis of rib fracture may be imperfect. 43 Lastly, because the random assignment of participants to the derivation or validation set may have improved the performance of our model, 29,30 future external validation could show different classification performance.…”