Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data

Abstract

The analysis of climatological data often involves statistical significance testing at many locations. While the field significance approach determines if a field as a whole is significant, a multiple testing procedure determines which particular tests are significant. Many such procedures are available, most of which control, for every test, the probability of detecting significance that does not really exist. The aim of this paper is to introduce the novel “false discovery rate” approach, which controls the false rejections in a more meaningful way. Specifically, it controls a priori the expected proportion of falsely rejected tests out of all rejected tests; additionally, the test results are more easily interpretable. The paper also investigates the best way to apply a false discovery rate (FDR) approach to spatially correlated data, which are common in climatology. The most straightforward method for controlling the FDR makes an assumption of independence between tests, while other FDR-controlling methods make less stringent assumptions. In a simulation study involving data with correlation structure similar to that of a real climatological dataset, the simple FDR method does control the proportion of falsely rejected hypotheses despite the violation of assumptions, while a more complicated method involves more computation with little gain in detecting alternative hypotheses. A very general method that makes no assumptions controls the proportion of falsely rejected hypotheses but at the cost of detecting few alternative hypotheses. Despite its unrealistic assumption, based on the simulation results, the authors suggest the use of the straightforward FDR-controlling method and provide a simple modification that increases the power to detect alternative hypotheses.

A comprehensive analysis of changes in precipitation regime in Tuscany

“…Both methods are described in Ventura et al (2004) and also seem to fit well for spatial correlated data like the ones present in this analysis. The adopted nominal FDR rate q is 0.05 and the a n a priori unknown proportion of true alternative hypothesis (FDR-mod test) is evaluated as in Ventura et al (2004).…”

“…This figure is available in colour online at www.interscience.wiley.com/ijoc multiple testing methods are applied: the false discovery rate (FDR) as proposed by Benjamini and Hochberg (1995) and the successive modification introduced by Ventura et al (2004), false discovery rate modified (FDRmod), which gives the FDR tighter control and increases the power to detect significant changes. Both methods are described in Ventura et al (2004) and also seem to fit well for spatial correlated data like the ones present in this analysis.…”

“…Compromises to avoid the two types of error are a well-known subject in the literature, and several approaches have been developed to tackle the multiple test problem (Livezey and Chen, 1983;Benjamini and Hochberg, 1995;Benjamini and Yekutieli, 2001). For a recent review of different approaches and further insights, see Ventura et al (2004).…”

“…We refer to this method henceforth as false positive rate (FPR) (Ventura et al, 2004). Subsequently, two different …”

Goodness of Fit of Probability Distributions for Sightings as Species Approach Extinction

“…Field significance is the collective significance of a group of hypothesis tests. Analogously to the work of Ventura et al (2004), in the following sections, we use the ideas of Livezey and Chen (1983), Simes (1986), Vogel and Kroll (1989), and Benjamini and Hochberg (1995) to evaluate the overall field significance associated with the group of individual tests reported in Table 2.…”