Assessing Preregistration Deviations: A Comparative Analysis of Psychologists and Open Science Experts

Abstract

Undisclosed deviations between preregistration plans and published articles challenge research transparency. This study examines whether open science experts can aid in assessing such deviations by comparing their evaluations with those of psychology experts. Using content analysis, we compared assessments of 25 preregistration plans and articles, following a psychologist studying preregistration discrepancies framework. The 72.6% agreement rate suggests differences stemmed from distinct perspectives. Psychologists leveraged domain knowledge to scrutinize subtle inconsistencies, while open science experts, deeply committed to transparency principles, focused more on procedural adherence. Open science experts, unfamiliar with psychology's implicit norms, faced challenges in discerning field‐specific knowledge and interpreting ambiguities as non‐deviations due to unclear preregistration formats and ambiguous content. Targeted support and collaborative approaches engaging both expert groups could enhance preregistration adherence evaluations, balancing the open science experts' emphasis on transparent processes with the psychologists' domain expertise, ultimately strengthening research transparency and reproducibility.