Why most psychological research findings are not even wrong

Abstract

Psychology’s replication crisis is typically conceptualised as the insight that the published literature contains a worrying amount of unreplicable, false-positive findings. At the same time, meta-scientific attempts to assess the crisis in more detail have reported substantial difficulties to identify unambiguous definitions of the scientific claims in published articles and to determine how they are connected to the presented evidence. I argue that most claims in the literature are so critically underspecified that attempts to empirically evaluate them are doomed to failure — they are not even wrong. Meta-scientists should beware of the flawed assumption that the psychological literature is a collection of well-defined claims. To move beyond the crisis, psychologists must reconsider and rebuild the conceptual basis of their hypotheses before trying to test them.

Reply to Engzell: Maybe in plain sight but out of focus

“…Overall, Engzell reminds us about careful hypothesis specification, measurement, and causal inference. These are things coming to the fore in recent methodological discussions ( 3 – 5 ), and with which we wholeheartedly agree and see as key takeaways from our main findings.…”

Metascience Is Not Enough – A Plea for Psychological Humanities in the Wake of the Replication Crisis

“…As some scientists have self-critically noted, many researchers tend to skip central steps in the definition and conceptualization of the constructs they investigate, which often leads to a lack of conceptual clarity and threatens the validity of psychological inferences (Vazire et al, 2020). Other psychologists have also pointed to the conceptual vagueness of hypotheses and insufficient definitions of key terms (Scheel, 2021;Scheel et al, 2021) Issues of validity are relevant for assessing the replication crisis. On the one hand, the replication crisis led to a general decline in credibility and distrust in the quality of psychological research, which had a wider impact.…”

“…As some scientists have self-critically noted, many researchers tend to skip central steps in the definition and conceptualization of the constructs they investigate, which often leads to a lack of conceptual clarity and threatens the validity of psychological inferences (Vazire et al, 2020). Other psychologists have also pointed to the conceptual vagueness of hypotheses and insufficient definitions of key terms (Scheel, 2021; Scheel et al, 2021)…”

Cognition and Society: Prolegomenon to a Dialog

“…Demands for deeper ideas and integration across (sub)disciplines are historically common in the human sciences, but have grown stronger and more urgent in light of the ongoing "replication crisis" in psychology and related fields. Many contributions to present debate about the future of the human sciences have emphasized the need for better foundations and transdisciplinary synthesis, alongside essential methodological reform (e.g., Eronen & Bringmann, 2021;Flis, 2022;Scheel, 2022). Good faith dialogues across the cognitive-social divide will help to meet this demand directly.…”