Effects of Sample Size on Differential Gene Expression, Rank Order and Prediction Accuracy of a Gene Signature

Abstract

Top differentially expressed gene lists are often inconsistent between studies and it has been suggested that small sample sizes contribute to lack of reproducibility and poor prediction accuracy in discriminative models. We considered sex differences (69♂, 65♀) in 134 human skeletal muscle biopsies using DNA microarray. The full dataset and subsamples (n = 10 (5♂, 5♀) to n = 120 (60♂, 60♀)) thereof were used to assess the effect of sample size on the differential expression of single genes, gene rank order and prediction accuracy. Using our full dataset (n = 134), we identified 717 differentially expressed transcripts (p<0.0001) and we were able predict sex with ∼90% accuracy, both within our dataset and on external datasets. Both p-values and rank order of top differentially expressed genes became more variable using smaller subsamples. For example, at n = 10 (5♂, 5♀), no gene was considered differentially expressed at p<0.0001 and prediction accuracy was ∼50% (no better than chance). We found that sample size clearly affects microarray analysis results; small sample sizes result in unstable gene lists and poor prediction accuracy. We anticipate this will apply to other phenotypes, in addition to sex.

A standardized randomized 6-month aerobic exercise-training down-regulated pro-inflammatory genes, but up-regulated anti-inflammatory, neuron survival and axon growth-related genes

“…A recent microarray study by Stretch et al examining sex-based differences in gene expression concluded that a larger sample size would be helpful (88). However, compared to the cross-sectional nature of the Stretch et al study, the randomized, controlled, and supervised clinical trial approach of this rigorous exercise study in AA MCIs, limit the impact of the small sample size.…”

Concordance analysis of microarray studies identifies representative gene expression changes in Parkinson’s disease: a comparison of 33 human and animal studies

“…Multiple studies have found that larger sample sizes in microarray experiments allow greater confidence in calling differentially expressed genes and more robust differentially expressed gene lists [20, 21, 25], but the effect of sample size in the context of average concordance across different datasets - i.e., the likelihood of being an unrepresentative ‘outlier’ study - has not been examined directly. When the smallest 25% of human studies were excluded (excluding five studies with sample sizes of less than 10), concordance within the remaining larger studies increased slightly from 0.08 to 0.11 at the differential gene expression level and from 0.15 to 0.17 at the pathway level.…”

“…More recent studies in psoriasis [19] and in healthy tissues [7] still found detectable platform biases, indicating that this issue may not be resolved by the use of newer or more closely related microarray technologies. The effect of sample size on study concordance should also be considered: numerous simulation studies have found that larger sample sizes in microarray studies result in more stable differentially expressed gene lists [20, 21]; however, large numbers of high-quality brain tissue samples are not always easy to obtain [22, 23], and so it is advantageous to examine more directly the impact of sample size on concordance in this context.…”

Transcript-based reannotation for microarray probesets

“…In a typical experiment, we obtain a table with thousands of genes for a small number of samples. This leads to situations where it is difficult, or even impossible, to employ classical prediction algorithms leading to poor prediction accuracy in discriminant models [5]. While a great number of studies have been focused on data dimensionality reduction [6], applying statistical methods like Principal Component Analysis, Linear Discriminant Analysis, k-Nearest Neighbors, etc., other works turned their attention to increase the number of samples [7].…”