Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)

Skorb, Lauren; Aczél, Balázs; Bakos, Bence E.; Feinberg, Lily; Hałasa, Ewa; Kauff, Mathias; Kovács, Márton; Krasuska, Karolina; Kuchno, Katarzyna; Manfredi, Dylan; Montealegre, Andres; Pękala, Emilian; Pieńkosz, Damian; Ravid, Jonathan; Rentzsch, Katrin; Schulz‐Hardt, Stefan; Sioma, Barbara; Szécsi, Péter; Szuts, Attila; Szöke, Orsolya; Christ, Oliver; Fedor, Anna; Jiménez‐Leal, William; Muda, Rafał; Nave, Gideon; Salamon, Janos; Schultze, Thomas; Hartshorne, Joshua K.

doi:10.1177/2515245920927643

Cited by 3 publications

(9 citation statements)

References 14 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The two project organizers had primary responsibility for drafting the manuscript, and the other authors contributed to revisions, knowing the outcomes of at most one set of replications during the writing process (depending on which individual studies they helped conduct). The full reports of the individual replication studies are reported separately in this issue (Baranski et al, 2020; Buttrick et al, 2020; Chartier et al, 2020; Corker et al, 2020; Ebersole et al, 2020; IJzerman et al, 2020; Lazarević et al, 2020; Mathur et al, 2020; Rabagliati et al, 2020; Skorb et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Ebersole

Mathur

Baranski

et al. 2020

Advances in Methods and Practices in Psychological Science

Self Cite

View full text Add to dashboard Cite

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

show abstract

Section: Methodsmentioning

confidence: 99%

“…However, because it matches neither the RP:P protocol nor what was recommended during review, it is not included in the analysis here. For more detail, see Skorb et al (2020, this issue).…”

mentioning

confidence: 99%

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Ebersole

Mathur

Baranski

et al. 2020

Advances in Methods and Practices in Psychological Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…One of the other full successes had participants observe an exchange (a supposedly unscripted social interaction) and tracked their eye movements during it. Two studies used computers but simulated live, ongoing interactions, and these had some findings, albeit weak ones, consistent with the original (Bouwmeester et al, 2017;Skorb et al, 2020). Thus, the few studies featuring genuine interpersonal interaction replicated reasonably well.…”

Section: Explaining Failuresmentioning

confidence: 56%

“…Findings by Moran et al (2021) and Vohs et al (2021) were significant or not depending on how many participants were excluded from analyses. Skorb et al (2020) tried three different protocols, of which only one provided a significant result, and that was after excluding over half the sample. Baranski et al (2020) replicated a finding about imaginary perpetrators but not the complementary finding for imaginary victims-thus contradicting the overarching theoretical point despite replicating one of the findings.…”

Section: Mixed Successesmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Multisite Replication Projects in Social Psychology: Is It Viable to Sustain Any Confidence in Social Psychology’s Knowledge Base?

Baumeister

Tice

Bushman

2022

Perspect Psychol Sci

View full text Add to dashboard Cite

Multisite (multilab/many-lab) replications have emerged as a popular way of verifying prior research findings, but their record in social psychology has prompted distrust of the field and a sense of crisis. We review all 36 multisite social-psychology replications (plus three articles reporting multiple ministudies). We start by assuming that both the original and the multisite replications were conducted in honest and diligent fashion, despite often yielding different conclusions. Four of the 36 (11%) were clearly successful in terms of providing significant support for the original hypothesis, and five others (14%) had mixed results. The remaining 27 (75%) were failures. Multiple explanations for the generally poor record of replications are considered, including the possibility that the original hypothesis was wrong; operational failure; low engagement of participants; and bias toward failure. The relevant evidence is assessed as well. There was evidence for each of the possibilities listed above, with low engagement emerging as a widespread problem (reflected in high rates of discarded data and weak manipulation checks). The few procedures with actual interpersonal interaction fared much better than others. We discuss implications in relation to manipulation checks, effect sizes, and impact on the field and offer recommendations for improving future multisite projects.

show abstract

Who does big team science?

Buchanan,

Lewis

2023

Preprint

View full text Add to dashboard Cite

This paper examined the nature of publications in Big Team Science (BTS): large-scale collaborations between multiple researchers at multiple institutions. These projects can improve research by initiating collaborations that span across the globe, age groups, education levels, and subfields of research. As the number of BTS publications increase, it is useful to explore who is currently involved in BTS projects to determine diversity in both research subject and researcher representation. We examined the diversity of BTS publications and authors across more than half a million articles to investigate where and what is currently published, and author characteristics including differences in career length, publication metrics, affiliation, and affiliation geopolitical regions. Interestingly, BTS publications are increasingly dominated by early career researchers from WEIRD geopolitical regions with Health and Physical Science accounting for the majority of BTS articles. However, the increase in preprints, BTS articles, and non-WEIRD authors across time demonstrate the efforts of the science community to diversify its researchers.

show abstract

Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)

Cited by 3 publications

References 14 publications

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

A Review of Multisite Replication Projects in Social Psychology: Is It Viable to Sustain Any Confidence in Social Psychology’s Knowledge Base?

Who does big team science?

Contact Info

Product

Resources

About