Replication Target Selection in Clinical Psychology: A Bayesian and Qualitative Re-evaluation

Abstract

Low rates of replication in clinical psychological research can have serious implications for clinical practice. We aim to (1): assess the strength of statistical evidence for primary findings in a journal representative of clinical psychology using Bayesian statistics, and (2) take a pragmatic approach to dealing with the question of which studies to replicate by proposing a two-step process. This process entails a quantitative assessment of the strength of evidence using Bayes factors, and a qualitative assessment of underlying theory and methodology. We provide proof of concept utilizing papers published in the Journal of Consulting and Clinical Psychology between 2012 and 2016. A total of 75 studies yielding 94 effects were included. Step 1 yielded 52 effects (55.32%) for which Bayes Factors provided strong support for the claimed effect. For the other 42 effects (44.68%), Bayes Factors suggested only modest evidence or even evidence of an absence of an effect. In step 2, these 42 effects were assessed on qualitative grounds independently by two raters. We discuss their recommendations, illustrate their decision process, and discuss the advantages and disadvantages of employing the proposed two-step process. Such an approach could benefit the field and enhance evidence-based treatment.

Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

“…Formula values can be used as a basis for formalized study selection procedures (e.g. Pittelkow et al, 2020). A formalized procedure means the steps that together describe how study selection will be conducted are clearly defined and standardized (e.g.…”

“…during study selection. In fact, selection strategies based on a mix of quantitative and qualitative information have already been proposed (Field et al, 2019;Pittelkow et al, 2020).…”

Replication value as a function of citation impact and sample size

“…We can then invest a more substantial amount of time collecting both qualitative and quantitative information about which studies in this subset would be most worth replicating. How this evaluation should proceed is a topic of discussion (Field et al, 2019;KNAW, 2018;Pittelkow, Hoekstra, Karsten, & van Ravenzwaaij, 2020), and should probably be adapted depending on the goals of the stakeholder. However, the model put forth in Isager et al (2020) suggests that evaluation should always include some consideration of (1) the perceived value of the research claim, (2) uncertainty about the truth status of the claim prior to replication, (3) the ability of the planned replication study design to reduce uncertainty about the claim, and (4) the estimated costs of conducting the replication effort.…”

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

“…In Field et al [37], a worked example is given on how to combine strength of evidence (quantified with a Bayesian reanalysis of published studies) with theoretical and methodological considerations. Pittelkow et al [38] follow a similar procedure, applied to studies from clinical psychology. Isager et al [39] outline a model for deciding on the utility of replicating a study, given known costs, by calculating the value of a claim and the uncertainty about that claim prior to replication.…”