The null hypothesis significance test in health sciences research (1995-2006): statistical analysis and interpretation

Abstract

BackgroundThe null hypothesis significance test (NHST) is the most frequently used statistical method, although its inferential validity has been widely criticized since its introduction. In 1988, the International Committee of Medical Journal Editors (ICMJE) warned against sole reliance on NHST to substantiate study conclusions and suggested supplementary use of confidence intervals (CI). Our objective was to evaluate the extent and quality in the use of NHST and CI, both in English and Spanish language biomedical publications between 1995 and 2006, taking into account the International Committee of Medical Journal Editors recommendations, with particular focus on the accuracy of the interpretation of statistical significance and the validity of conclusions.MethodsOriginal articles published in three English and three Spanish biomedical journals in three fields (General Medicine, Clinical Specialties and Epidemiology - Public Health) were considered for this study. Papers published in 1995-1996, 2000-2001, and 2005-2006 were selected through a systematic sampling method. After excluding the purely descriptive and theoretical articles, analytic studies were evaluated for their use of NHST with P-values and/or CI for interpretation of statistical "significance" and "relevance" in study conclusions.ResultsAmong 1,043 original papers, 874 were selected for detailed review. The exclusive use of P-values was less frequent in English language publications as well as in Public Health journals; overall such use decreased from 41% in 1995-1996 to 21% in 2005-2006. While the use of CI increased over time, the "significance fallacy" (to equate statistical and substantive significance) appeared very often, mainly in journals devoted to clinical specialties (81%). In papers originally written in English and Spanish, 15% and 10%, respectively, mentioned statistical significance in their conclusions.ConclusionsOverall, results of our review show some improvements in statistical management of statistical results, but further efforts by scholars and journal editors are clearly required to move the communication toward ICMJE advices, especially in the clinical setting, which seems to be imperative among publications in Spanish.

Breaking Free of Sample Size Dogma to Perform Innovative Translational Research

“…This general approach has been criticized as unreliable, wasteful of information, harmful to scientific progress, and contrary to the original statistical theories that supposedly support it (23-28), but it remains engrained in research culture (29, 30). As in the case of sample size, existing conventions are especially inappropriate for early, highly innovative studies, where the potential for unexpected findings is high and where interpretation should focus on estimated effects and the uncertainty around them (as shown by confidence intervals) and whether the ensemble of results have a coherent and plausible biological explanation.…”

Optimizing pedigrees: Using a biasing system to determine likely inheritance systems

“…If we were to have more individuals, then our system will deduce with more certainty if a model fits. [5] Example 2 (Y-Linked, Chance of Autosomal):…”

“…Therefore the proposed model is accurate with respect to our data. Assume p = 0.05 [5] for us to be able to reject the null hypothesis.…”

Optimizing Pedigrees: Using a Biasing System to Determine Likely Inheritance Systems

“…If we were to have more individuals, then our system will deduce with more certainty if a model fits. [5] Example 2 (Y-Linked, Chance of Autosomal Dominant):…”

“…If we were to have more individuals, then our system will deduce with more certainty if a model fits. [5] Now, I will use a simpler example to prove that this method is accurate. Assume we know nothing but the phenotypes.…”