Promoting Learning from Null or Negative Results in Prevention Science Trials

Abstract

There can be a tendency for investigators to disregard or explain away null or negative results in prevention science trials. Examples include not publicizing findings, conducting spurious subgroup analyses, or attributing the outcome post hoc to real or perceived weaknesses in trial design or intervention implementation. This is unhelpful for several reasons, not least that it skews the evidence base, contributes to research “waste”, undermines respect for science, and stifles creativity in intervention development. In this paper, we identify possible policy and practice responses when interventions have null (ineffective) or negative (harmful) results, and argue that these are influenced by: the intervention itself (e.g., stage of gestation, perceived importance); trial design , conduct , and results (e.g., pattern of null/negative effects, internal and external validity); context (e.g., wider evidence base, state of policy); and individual perspectives and interests (e.g., stake in the intervention). We advance several strategies to promote more informative null or negative effect trials and enable learning from such results, focusing on changes to culture, process, intervention design, trial design, and environment.

Small studies, big decisions: the role of pilot/feasibility studies in incremental science and premature scale-up of behavioral interventions

“…Engineering factors with known influence on both markers of feasibility and preliminary efficacy out of pilot/feasibility studies should reduce unnecessary noise and allow behavioral scientists to focus on more salient underlying causes for either a lack of implementability and/or impact of an intervention at-scale [4]. Ultimately, this would result in a greater number of findings with of substantial scientific merit [26].…”

Transparent, Open, and Reproducible Prevention Science

“…Chief among these detrimental research practices is selective non-reporting of studies and results, which occurs when the nature of study findings (rather than methodological rigor) influences the decision to submit, disseminate, or publish them (Chalmers, 1990;Chan, 2008). There is ample and long-standing evidence across disciplines that "statistically significant" or "positive" results are more likely to be published than results that are "not statistically significant," "negative," "null," "inconclusive," or otherwise countervailing (Axford et al, 2020;Dwan et al, 2013;Fanelli, 2010aFanelli, , 2010bFranco et al, 2014;Hartgerink et al, 2016;Sterling, 1959). Researchers may selectively refrain from writing-up and submitting entire studies for publication based on the nature or direction of results (Rosenthal, 1979), leading to a biased subsample of studies being published in the literature on a research topic.…”

“…There is also selective non-reporting of study results ("reporting bias" or "selective outcome reporting"), which occurs when researchers choose a subset of outcomes to report in manuscripts (Axford et al, 2020;Chan et al, 2004). Selective non-reporting of results may be difficult to detect because it tends to be apparent only when study protocols and statistical analysis plans are registered prospectively, and when reviewers or readers check published results against registered outcomes and analyses.…”

Transparent, Open, and Reproducible Prevention Science

“…Chief among these detrimental research practices is selective non-reporting of studies and results, which occurs when the nature of study findings (rather than methodological rigor) influences the decision to submit, disseminate, or publish them (Chalmers, 1990;A.-W. Chan, 2008). There is ample and long-standing evidence across disciplines that "statistically significant" or "positive" results are more likely to be published than results that are "not statistically significant," "negative," "null," "inconclusive", or otherwise countervailing (Axford, Berry, Lloyd, Hobbs, & Wyatt, 2020;Dwan et al, 2013;Daniele Fanelli, 2010a, 2010bFranco, Malhotra, & Simonovits, 2014;Hartgerink, Aert, Nuijten, Wicherts, & Assen, 2016;Sterling, 1959). Researchers may selectively refrain from writing-up and submitting entire studies for publication based on the nature or direction of results (Rosenthal, 1979), leading to a biased subsample of studies being published in the literature on a research topic.…”

“…There is also selective non-reporting of study results ("reporting bias" or "selective outcome reporting"), which occurs when researchers choose a subset of outcomes to report in manuscripts (Axford et al, 2020;A.-W. Chan, Hróbjartsson, Haahr, Gøtzsche, & Altman, 2004).…”