Interpretation, Design, and Analysis of Gene Array Expression Experiments

Abstract

Experiments using arrays of cDNA targets to compare patterns of gene expression are beginning to play a prominent role in biogerontology, but drawing reliable conclusions from the resulting data sets requires careful application of statistical methods that discriminate chance events from those likely to reflect real differences among the samples under study. This essay discusses flaws in the logic of studies that base their conclusions on ratio calculations alone, reviews the multiple comparison traps inherent in high throughput systems that test a very large number of mRNAs simultaneously, and advocates a two-stage design in which significance testing applied to exploratory data is used to guide a second round of hypothesis-testing experiments conducted in a separate set of experimental samples.

Gene Microarrays in Hippocampal Aging: Statistical Profiling Identifies Novel Processes Correlated with Cognitive Impairment

“…Because the number of expected false positives equals the p value percentage of the total number of comparisons tested (tests of individual genes), reducing total comparisons is useful for managing false positive error. That is, if 10,000 genes are tested at the p Ͻ 0.05 level of significance, 500 (5%) can be expected to be positive by chance alone (Miller et al, 2001). Such high false positives can rival or obscure true positives and thereby detract from statistical confidence.…”

“…With only a few exceptions (Pletcher et al, 2002), microarray studies have not used the formal statistical analyses and sample sizes (power) necessary to estimate expected false positives or detect moderate changes in expression. Thus, given the large multiple-comparison error anticipated in microarray analyses (Miller et al, 2001), both false positive (type I) and false negative (type II) errors are often high. As corollaries, the statistical reliability of microarray results is often clouded, and many potentially important processes have likely been overlooked (Watson et al, 2000;Miller et al, 2001;Becker, 2002).…”

“…Thus, given the large multiple-comparison error anticipated in microarray analyses (Miller et al, 2001), both false positive (type I) and false negative (type II) errors are often high. As corollaries, the statistical reliability of microarray results is often clouded, and many potentially important processes have likely been overlooked (Watson et al, 2000;Miller et al, 2001;Becker, 2002). Additionally, it has frequently been difficult to assess which of many hundreds of observed gene alterations are potentially relevant to functional endpoints.…”

A two-stage design for multiple testing in large-scale association studies

“…Such strategies have also been proposed to deal with low power due to the Bonferroni correction. For example, for array experiments, Miller et al (2001) suggested first selecting a smaller set of microarray sample and proceeding to the second stage of tests with another data set using only genes found significant in the first stage. Later, Allison and Coffey (2002) pointed out that this two-stage method can be even more conservative if the two significance levels are not chosen carefully.…”

“…After obtaining a smaller and more promising set of markers, we conduct statistical tests with a stringently controlled overall type I error using a combination of data from the first-stage sample and the newly genotyped data. van den Oord and Sullivan (2003a) and others (Miller et al 2001;Saito and Kamatani 2002;Satagopan et al 2002) considered only new data in their secondstage test to avoid complexity in test statistics due to interdependence between the first and second samples. As argued by van den Oord and Sullivan (2003a), inclusion of first-stage data may elevate false positive discoveries, but can reduce the genotyping burden.…”