Arrested Theory Development: The Misguided Distinction Between Exploratory and Confirmatory Research

Abstract

Science progresses by finding and correcting problems in theories. Good theories are those that help facilitate this process by being hard to vary: They explain what they are supposed to explain, they are consistent with other good theories, and they are not easily adaptable to explain anything. Here we argue that, rather than a lack of distinction between exploratory and confirmatory research, an abundance of flexible theories is a better explanation for the current replicability problems of psychology. We also explain why popular methods-oriented solutions fail to address the real problem of flexibility. Instead, we propose that a greater emphasis on theory criticism by argument might improve replicability.

The case for formal methodology in scientific reform

“…8: 200805 statistical inference, but there is no mathematical mapping from these to exploratory or confirmatory research, especially when clear technical definitions for the latter are not provided, in clear violation of step 1 of our formal approach. Moreover, the general usefulness and relevance of this dichotomy has also been challenged for theoretical reasons [79,80]. In this section, we sidestep issues with the dichotomy but argue against the core claim presented by [13] regarding the nature of exploratory research specifically, advancing the following points:…”

“…Testing hypotheses with no theory to motivate them is a fishing expedition regardless of methodological rigor. See [44,[75][76][77][78][79][80][81][82][83][84] for discussions on scientific theory.…”

Reducing bias, increasing transparency, and calibrating confidence with preregistration

“…Researcher degrees of freedom may be naturally constrained to some extent by strong theory 74,101,128 , community norms 17 , or in replication studies, though this constraint may be more implicit than explicit, leaving room for flexible decision-making.…”

“…Preregistration enables differentiation between the two so confidence in the outcomes of each analysis can be appropriately calibrated. Clarifying the exploratory-confirmatory distinction may also reduce the pressure on researchers to disguise exploratory analyses as confirmatory analyses 26,27,144 , potentially exposing that many extant theories and measurement tools are not sufficiently developed to enable informative hypothesis tests 101,128,228 .…”

“…Preregistration is one tool in a scientist's toolbox. It does not guarantee study quality or sound inference, nor does it directly improve bad theories 68,101,128 . Preregistration has specific pragmatic functions (Section 4.1): to reduce bias and calibrate confidence.…”