The Behavior of the P-Value When the Alternative Hypothesis is True

Abstract

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.. The P-value is a random variable derived from the distribution of the test statistic used to analyze a data set and to test a null hypothesis. Under the null hypothesis, the P-value based on a continuous test statistic has a uniform distribution over the interval [0,1], regardless of the sample size of the experiment. In contrast, the distribution of the P-value under the alternative hypothesis is a function of both sample size and the true value or range of true values of the tested parameter. International Biometric SocietyThe characteristics, such as mean and percentiles, of the P-value distribution can give valuable insight into how the P-value behaves for a variety of parameter values and sample sizes. Potential applications of the P-value distribution under the alternative hypothesis to the design, analysis, and interpretation of results of clinical trials are considered. IntroductionThe P-value is one of the most routinely used statistical measures of uncertainty, yet statisticians may in some situations (Goodman, 1992) disagree on its appropriate use and on its interpretation as a measure of evidence. The P-value is derived from the perspective of a test of hypothesis in which a test statistic is calculated from results of a given set of data and, under the assumption that the null hypothesis is true, the distribution of the test statistic is used to obtain the tail probability of observing that result or a more extreme result. Thus, the P-value is a measure of evidence against the null hypothesis. Because the P-value is based upon analysis of random variables, it itself is a random variable whose distribution, for continuous test statistics, is well known to be uniform over the interval [0, 1] under the null hypothesis. It is because of this fact that a cutoff for a P-value at, say 0.05, is used to control the chances that, for any given experiment, one of twenty P-values could be 0.05 or less, even when the null hypothesis is true. This concept, in the Neyman-Pearson theory of hypothesis testing, is known as the Type I error rate, which is a preexperiment error rate that determines the rejection region and is intended to control the overall frequency of making erroneous rejections of the null hypothesis. It is of interest that the distribution of the P-value, when the null hypothesis is true, is uniform over [0,1] regardless of the sample size of an experiment, so there is no way to distinguish P-values derived from large studies from those derived from small samples, nor from studies well powered to detect a posited alternative hypothesis from those underpowered to detect that same posited alternative value. Other statistical measures, such as confidence intervals, ...

The response of African land surface phenology to large scale climate oscillations

“…5). Thus, if an α level of 0.1 is chosen as a cutoff, for any given experiment, one of ten p-values could be 0.1 or less, even when the null hypothesis is true (Hung et al, 1997). If we perform a correlation test for 100 independent pixels for which the null hypothesis is true, then the chance that we will find at least one pixel with a significant correlation (p b 0.1), can be calculated as one minus the chance that there are no significant correlations: where α is the significance level and n is the number of tests.…”

When null hypothesis significance testing is unsuitable for research: a reassessment

“…The larger is the effect size and power the stronger is this right skew (Figure 2). This fact led to the suggestion that comparing this skew allows us to determine the robustness of findings in some fields by studying "p curves" (Hung et al, 1997;Simonsohn et al, 2014a,b). Hence, from this perspective, replication, and unbiased publication of all results FIGURE 2 | The distribution of p-values if the alternative hypothesis (H 1 ) is true.…”

Fast retrievals of tropospheric carbonyl sulfide with IASI

“…Note that p values represent statistical significance on a scale of 0-1, where values close to zero show significance in the sampled correlation, while higher values fail to reject the null hypothesis. As a rule of thumb, p values less than 0.05 are generally regarded as statistically significant (Hung et al, 1997).…”