Statistical Hypothesis Testing: Overview and Application

Abstract

When developing new headache treatments or when discovering relationships among important variables, it is often necessary to infer characteristics about a large population from a sample of observed data. For example, when testing a new headache treatment vs placebo in a sample of 100 individuals, do the observed differences in the sample provide high confidence that the treatment will also work in the population? Because samples contain random sampling error, researchers require a set of methods to help decide if any observed treatment difference, or observed relationship, is simply due to chance. Statistical inference refers to those methods that allow the estimation of population properties (eg, a treatment effect) from observed samples. Very often, this process takes the form of formal hypothesis testing. Although there are many ways a researcher could investigate a hypothesis, in medical research, by far the most common is through the use of some form of statistical hypothesis testing.Statistical hypothesis testing is a set of methods for statistical inference that has a fascinating and contentious history (see: Lenhard 1 ). A famous debate raged for decades between the early creators of these methods about the proper application of the emerging technique that would eventually become the most popular tool for statistical inference. 2 The methods most commonly used today are a blend between the "significance test" developed by Fisher 3 and the "hypothesis test" developed by Neyman and Pearson. 4 Although modern application of statistical hypothesis testing has evolved over time, perhaps tending toward the approaches advocated by Neyman and Pearson, 2 a thorough understanding of the principles of significance-based statistical hypothesis testing is crucial for investigators, consumers of research, and even for the growing number of individuals who wish to abandon the use of any hypothesis testing based on these principles. 5 This editorial is the next in the Journal's methods and statistics primer series. [6][7][8][9][10][11] In this installment, we introduce the concept of significance-based statistical hypothesis testing and the use of this form of statistical inference in headache research. We also describe common problems encountered when applying and reporting findings related to this form of hypothesis testing. DEFINING THE ISSUEMost people who have read a headache research article have seen the signs that significance-based hypothesis testing has been conducted. The use of P values (eg, P < .05), the term "statistically significant," and the array of statistical tests (eg, ANOVA, t-tests) all convey that investigators are testing a hypothesis to make an inference about some population. To understand what these terms indicate, it is important to grasp that all such hypotheses are attempts to refute, rather than prove, something. 12 This reasoning is at first counterintuitive but becomes clearer when the idea of the null hypothesis is fully understood.The Null Hypothesis (H 0 ).-Significance-based hypothe...

Overview of Common Statistical Tests and their Assumptions

“…When examining associations among variables of interest, the characteristics of a population must often be inferred from a sample of that population 6 . This process of statistical inference involves methods that permit estimation of population properties (eg, effect of a new headache treatment) from observed samples 7 . The population properties are referred to as population parameters (eg, population mean) and are inferred from the sample properties, typically referred to as point estimates (eg, sample mean).…”

“…The null hypothesis is rejected if it is found not to be representative of reality. Several approaches have been developed for examining collected evidence and using this evidence to make decisions about a null hypothesis 7 . In Fisher’s approach, a statistical test is used to compare the distribution(s) of data from a research study relative to a null hypothesis.…”

“…Based on the evidence, a choice is made of rejection of the null hypothesis in favor of the alternative hypothesis or failure to reject the null hypothesis. The Neyman‐Pearson method’s use of decision theory also has implications for Type‐I and Type‐II error 2,7 . The approach commonly used today is a combination of the Fisher and Neyman‐Pearson approaches and is referred to as Null Hypothesis Significance Testing (NHST).…”

“…In traditional hypothesis testing, a P value is the probability of observing a test statistic as extreme as or more extreme than that observed from the sample if the null hypothesis were in fact true 1,7 . Stated informally, a P value is the area in the distribution that represents the probability that a test statistic occurred due only to sampling error when the null hypothesis is true (ie, often stated as “by chance”).…”

“…This editorial continues the series of primers on methodological and statistical issues in the Journal 1‐7 . A previous editorial described the process of statistical hypothesis testing, 7 and we recommend that readers become familiar with the content of that editorial before beginning this one. In this editorial, we discuss approaches to statistical tests, assumptions for the use of statistical tests, and common problems encountered in the application of these tests.…”