A Critical Assessment of Null Hypothesis Significance Testing in Quantitative Communication Research

Levine, Timothy R.; Weber, René; Hullett, Craig R.; Park, Hee Sun; Lindsey, Lisa L. Massi

doi:10.1111/j.1468-2958.2008.00317.x

Cited by 117 publications

(47 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HCI researchers may think they can ignore these issues for the moment, because they are currently being debated. In reality, the debate mostly opposes strong reformists who think NHST should be banned (e.g., Loftus, 1993;Schmidt and Hunter, 1997;Lambdin, 2012;Cumming, 2013) with weak reformists who think it should be i) de-emphasized and ii) properly taught and used (e.g., Abelson, 1995;Abelson, 1997;Levine et al, 2008a;Levine et al, 2008b). I have already given arguments against i) by explaining that p-values are redundant with confidence intervals (Section 2).…”

Section: Resultsmentioning

confidence: 99%

“…Levine et al (2008a) offer a few quotes: "[NHST] is based upon a fundamental misunderstanding of the nature of rational inference, and is seldom if ever appropriate to the aims of scientific research (Rozeboom, 1960)"; "Statistical significance is perhaps the least important attribute of a good experiment; it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established, or that an experimental report ought to be published (Likken, 1968)". Some go as far as saying that "statistical significance testing retards the growth of scientific knowledge; it never makes a positive contribution" (Schmidt and Hunter, 1997).…”

Section: End User Dissatisfactionmentioning

confidence: 99%

See 1 more Smart Citation

Fair Statistical Communication in HCI

Dragicevic

2016

Human–Computer Interaction Series

258

247

View full text Add to dashboard Cite

Statistics are tools to help end users accomplish their task. In research, to be qualified as usable, statistical tools should help researchers advance scientific knowledge by supporting and promoting the effective communication of research findings. Yet areas such as human-computer interaction (HCI) have adopted toolsi.e., p-values and dichotomous testing procedures -that have proven to be poor at supporting these tasks. The abusive use of these procedures has been severely criticized in a range of disciplines for several decades, suggesting that tools should be blamed, not end users. This chapter explains in a non-technical manner why it would be beneficial for HCI to switch to an estimation approach, i.e., reporting informative charts with effect sizes and interval estimates, and offering nuanced interpretations of our results. Advice is offered on how to communicate our empirical results in a clear, accurate, and transparent way without using any tests or p-values.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: End User Dissatisfactionmentioning

confidence: 99%

Fair Statistical Communication in HCI

Dragicevic

2016

Human–Computer Interaction Series

258

247

View full text Add to dashboard Cite

show abstract

“…Statistical reporting errors and collaboration on statistical analyses in psychological science. PLoS One, 9(12), e114876 Most conclusions in psychological research (and related fields) are based on the results of null hypothesis significance testing (NHST) (Cohen, 1994;Hubbard & Ryan, 2000;Krueger, 2001;Levine, Weber, Hullet, Park, & Lindsey, 2008;Nickerson, 2000;Sterling, Rosenbaum, & Weinkam, 1995). Although the use and interpretation of this method have been criticized (e.g.…”

Section: Statistical Reporting Errors and Collaboration On Statisticamentioning

confidence: 99%

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

“…The inferential shortcomings inherent in the NHST framework have been the subject of methodological criticism for years (see Gelman & Loken, 2014;Levine, Weber, Hullett, Park, & Lindsey, 2008;Nickerson, 2000). Applied researchers are increasingly aware of NHST's oft-cited methodological limitations (e.g., p values' sensitivity to sample size, conceptual divide between statistical and practical significance, overinterpretation or misinterpretation of p values, inflation of Type I error rates through repeated testing).…”

mentioning

confidence: 99%