Untitled

Abstract

The calculation of the probability of correct selection (PCS) shows how likely it is that the populations chosen as "best" truly are the top populations, according to a well-defined standard. PCS is useful for the researcher with limited resources or the statistician attempting to test the quality of two different statistics. This paper explores the theory behind two selection goals for PCS, G-best and d-best, and how they improve previous definitions of PCS for massive datasets. This paper also calculates PCS for two applications that have already been analyzed by multiple testing procedures in the literature. The two applications are in neuroimaging and econometrics. It is shown through these applications that PCS not only supports the multiple testing conclusions but also provides further information about the statistics used.ENHANCING MULTIPLE TESTING 185 selection of the top populations. After an experiment, one can use these methods to calculate the probability that the researcher has found the actual top t populations.If our previous toy example was an actual experiment, we might have a need to find the top, say, one statistic, but we have the resources to study two. We would then use G-best selection, where we choose a fixed amount of populations, t + G, that contains the top t statistics. On the other hand, if we simply needed the populations to be within a certain threshold of quality, we would use d-best selection. In d-best selection, we are finding a random number of populations, say r , which contains populations that are within a certain distance d from the top t populations. The number r is determined by an interval of prespecified length d.Definition 2.1. Let s be the set of the indices corresponding to the top t +G statistics for some prespecified G. Let A t be a set of indices of the top t parameters. ThenA set s that satisfies CS G,t is called G-best, and the probability that we have chosen a G-best set is denoted by P(CS G,t ). 190ERIN IRWIN AND JASON WILSON t 1 2 3 4 P(CS 0,t ) = P( 0 CS t ) 0.11 0.02 0.00 0.00 P(CS 2,t ) 0.33 0.11 0.03 0.01 P(CS 4,t ) 0.49 0.22 0.08 0.03 P(CS 6,t ) 0.60 0.33 0.15 0.06 P( 0.5 CS t ) 0.19 0.04 0.05 0.03 P( 1 CS t ) 0.24 0.19 0.35 0.29P(CS 0,10 ) = 0.13 P( 0 CS 10 ) = 0.13 P(CS 0,10 ) = 0.71 P( 0 CS 10 ) = 0.71 P(CS 1,9 ) = 0.29 P( 0.5 CS 10 ) = 0.32 P(CS 1,9 ) = 0.98 P( 0.5 CS 10 ) = 0.71 P(CS 2,8 ) = 0.53 P( 1 CS 10 ) = 0.52 P(CS 2,8 ) = 1.00 P( 1 CS 10 ) = 0.95 P(CS 3,7 ) = 0.74 P( 1.5 CS 10 ) = 0.78 P(CS 3,7 ) = 1.00 P( 1.5 CS 10 ) = 0.95 P(CS 4,6 ) = 0.92 P( 2 CS 10 ) = 0.86 P(CS 4,6 ) = 1.00 P( 2 CS 10 ) = 0.97 P(CS 5,5 ) = 1.00 P( 2.5 CS 10 ) = 0.93 P(CS 5,5 ) = 1.