A predictive approach to measuring the strength of statistical evidence for single and multiple comparisons

Abstract

The normalized maximum likelihood (NML) is a recent penalized likelihood that has properties that justify defining the amount of discrimination information (DI) in the data supporting an alternative hypothesis over a null hypothesis as the logarithm of an NML ratio, namely, the alternative hypothesis NML divided by the null hypothesis NML. The resulting DI, like the Bayes factor but unlike the P‐value, measures the strength of evidence for an alternative hypothesis over a null hypothesis such that the probability of misleading evidence vanishes asymptotically under weak regularity conditions and such that evidence can support a simple null hypothesis. Instead of requiring a prior distribution, the DI satisfies a worst‐case minimax prediction criterion. Replacing a (possibly pseudo‐) likelihood function with its weighted counterpart extends the scope of the DI to models for which the unweighted NML is undefined. The likelihood weights leverage side information, either in data associated with comparisons other than the comparison at hand or in the parameter value of a simple null hypothesis. Two case studies, one involving multiple populations and the other involving multiple biological features, indicate that the DI is robust to the type of side information used when that information is assigned the weight of a single observation. Such robustness suggests that very little adjustment for multiple comparisons is warranted if the sample size is at least moderate. The Canadian Journal of Statistics 39: 610–631; 2011. © 2011 Statistical Society of Canada

Empirical Bayes estimation of posterior probabilities of enrichment: A comparative study of five estimators of the local false discovery rate

“…Thus, in our data set, the NML ratio tends to estimate the Bayes factor almost as accurately as methods that simultaneously use information across GO terms. While we do not expect the same for all data sets, we note that similar results have been found for an application of a modified NML [29] to a proteomics data set [30]. …”

“…where ${\hat{\theta }}_i\left(t_i\right|s_i)$ is a Type I MLE with respect to $\mathrm{script\mathscr{H}}$ under observed values T i given S i [28,29]. …”

“…(More generally, the logarithm of an NML ratio is interpreted as a measure of the evidential support for the alternative hypothesis over the null hypothesis [29,30]). Therefore, by combining equations (8) and (9), if we guess the prior probability Π 0 , the LFDR estimate of GO category i in the hypothesis comparison (7) is…”

Game-theoretic probability combination with applications to resolving conflicts between statistical methods

“…Let P + denote the combination of the distributions inP with truth constrained byṖ. IfṖ ∩P = ∅, then P + = centṖ ∩P = P WṖ ∩P , (9) where centṖ ∩P is the centroid ofṖ ∩P, and WṖ ∩P is the weighting distribution induced byṖ ∩P, as defined by Eq. (4).…”

Coherent Frequentism: A Decision Theory Based on Confidence Sets

“…Given the ubiquity of recognizable subsets (Buehler and Feddersen, 1963;Bondar, 1977), this strategy uses pre-data confidence as an approximation to post-data confidence in the sense in which expected Fisher information approximates observed Fisher information (Efron and Hinkley, 1978), aiming not at exact inference but at a pragmatic use of the limited resources available for any particular data analysis. Certain situations may instead call for careful applications of conditional inference (Goutis and Casella, 1995;Sundberg, 2003;Fraser, 2004) or of minimum description length (Bickel, 2011b) for basing decisions more directly on the data actually observed.…”