Quality Control for Scientific Research: Addressing Reproducibility, Responsiveness, and Relevance

Abstract

Efforts to address a reproducibility crisis have generated several valid proposals for improving the quality of scientific research. We argue there is also need to address the separate but related issues of relevance and responsiveness. To address relevance, researchers must produce what decision makers actually need to inform investments and public policy-that is, the probability that a claim is true or the probability distribution of an effect size given the data. The term responsiveness refers to the irregularity and delay in which issues about the quality of research are brought to light. Instead of relying on the good fortune that some motivated researchers will periodically conduct efforts to reveal potential shortcomings of published research, we could establish a continuous quality-control process for scientific research itself. Quality metrics could be designed through the application of this statistical process control for the research enterprise. We argue that one quality control metric-the probability that a research hypothesis is true-is required to address at least relevance and may also be part of the solution for improving responsiveness and reproducibility. This article proposes a "straw man"solution which could be the basis of implementing these improvements. As part of this solution, we propose one way to "bootstrap" priors. The processes required for improving reproducibility and relevance can also be part of a comprehensive statistical quality control for science itself by making continuously monitored metrics about the scientific performance of a field of research.

Moving to a World Beyond “ p < 0.05”

“…In a similar vein, Hubbard and Carriquiry (2019) urge that researchers prominently display the probability the hypothesis is true or a probability distribution of an effect size, or provide sufficient information for future researchers and policy makers to compute it. The authors further describe why such a probability is necessary for decision making, how it could be estimated by using historical rates of reproduction of findings, and how this same process can be part of continuous "quality control" for science.…”

Science is about corroborating empirical evidence, even in academic business research journals

“…Scientifically-based factual knowledge builds on the accumulation of consistently verified empirical findings. Academicians, however, show confusion as to the difference between scientific facts and empirical findings (Hubbard and Carriquiry, 2019). Fundamentally, empirical findings (also referred to as insights) represent researchers' interpretations of statistically significant tests drawn from a single sample or multiple studies reported in an article.…”

“…Scientific facts depend not only on establishing internal reliability, but more importantly, they require external validity or the generalizability of reported empirical research findings. Recently, Hubbard and Carriquiry (2019) reemphasized that the failure of academic journal authors to corroborate and reproduce statistical test results from original research studies feeds a reproducibility crisis that impedes theory development, theory testing, and the growth of a practically reliable body of knowledge in marketing, business, and the social sciences in general. In short, the journals are not producing the type of knowledge that would allow for confident prescriptions.…”

“…Here, we elaborate on the state of business research (with an emphasis on marketing) with respect to what has been called the reproducibility crisis in academic management research (Hubbard and Carriquiry, 2019), offer insight into factors that contribute to the crisis (e.g. publication bias, backing into hypotheses, misuse of p-values and statistical inferences, lack of emphasis on managerial irrelevance), and provide an overview of the corroborative efforts of the researchers successfully responding to the call for research.…”

Crowding in the time of COVID: Effects on rapport and shopping satisfaction

“…Based on this rationale, our work has two objectives, namely, to replicate and extend selected previous findings under the new environment dictated by the pandemic. The replication goal is inspired by the recent calls to corroborate and re-examine findings in the marketing literature in pursuit of higher rigor ( Babin et al, 2020 ; Hubbard, 2015 ; Hubbard and Carriquiry, 2019 ). Specifically, these calls encourage reproduction and replication efforts with “different contextual settings, populations, scale measurements and sampling units” ( Babin et al, 2020 , p.3).…”