Replication and Reproducibility in Psychological, Medical and other Sciences

Abstract

As there are no universal constants in psychological, medical and economic sciences, only constructive-phenomenon replications are meaningful. Yet, psychologists continue to perform direct replications, as evidenced by recent preregistered multilab attempts at exact replications of the ego depletion effect. Statistics are driving the replication movement into a ditch because of an overemphasis on the determination of statistical magnitude of effects while ignoring commonsense magnitude and other criteria for evaluating phenomena’s validity, reliability, and viability. The nature of the human mind and the variability of psychological phenomena pose difficult challenges for the scientific method and insurmountable obstacles for precise replications in psychological sciences. The situation is no better in medical and economic sciences. The interaction effect of person (genetics) and environment (lifestyle) calls for constructive replications to determine, for example, drugs’ efficacy as a function of group and individual differences. The vaccine-vaccination paradox is an interesting case because psychological and medical sciences meet at this intersection. In all fields, science advances by theory building and model expansion, not by replication tests of statistical hypotheses. Rigorous logical and theoretical analysis always precedes and guides good empirical tests. The nonexistence of an effect is not viable if it can withstand rigorous logical and theoretical analyses. Empirical studies are mainly evaluated for their theoretical relevance and importance, not their success or failure to exactly reproduce the original findings.

Science of psychological phenomena and their testing.

“…They include but are not limited to the following: No theoretical rationale was reported for the selected experimental tasks, and why different task protocols were employed at different sites, with one protocol being more effortful than others; no evidence was shown that the dependent variables were valid indicators of ego depletion, yet it is well established that the construct validity of the measurements is critical in all studies (Flake et al, 2022; Cronbach & Meehl, 1955; Hussey & Hughes, 2020); over 30% of participants (1,068 individuals) were excluded from analyses for several reasons, making the significance/nonsignificance of the results depend on the number of discarded participants, which is a form of p -hacking. For an additional analysis of the methodological problems of this study, see Iso-Ahola, 2022b. After employing 36 labs from around the world, the 128 researchers concluded that one of the following could be true: (a) there is no depletion effect, (b) the reliability of the effect is still unknown, and (c) there may be a small effect.…”

“…This suggests that constructive or phenomenon replications are important and useful (Crandall & Sherman, 2016; Derksen & Morawski, 2022; Iso-Ahola, 2020, 2022b; Rubin, 2021). Physicists, too, perform constructive replications, as they have done, for example, in testing the presence and effects of quantum entanglement in pairs of subatomic particles.…”

“…Furthermore, all statistical tests have serious limitations on what inferences can be drawn from them, not to mention frequent inappropriate and unknowledgeable uses of them and different interpretations of the same result by different researchers (Goodman et al, 2016). There is not one statistical test that can definitely declare that a phenomenon does not exist, meaning that psychological phenomena cannot be turned into statistical phenomena by some arbitrary quantitative criterion (Amrhein, Greenland, & McShane, 2019; Iso-Ahola, 2022b; Iso-Ahola & Dotson, 2015). If the intent is to generalize to a population, it is not possible for the simple fact that people are not robots that repeat the same feelings, cognitions, and behaviors on demand.…”