A Validity-Based Framework for Understanding Replication in Psychology

Fabrigar, Leandre R.; Wegener, Duane T.; Petty, Richard E.

doi:10.1177/1088868320931366

Cited by 59 publications

(57 citation statements)

References 152 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Arguably, however, an even greater threat elicited by "demand characteristics" has to do with ecological validity, one of the four main concerns in the replication crisis (Fabrigar, Wegener, & Petty, 2020). Specifically, even though a valid interpretation may be raised in a given experimental context, cues inherent in the lab setting may elicit motivations and interpretations that do not survive outside the lab, or when moving temporally or spatially from one lab to another, or even when merely changing the identity of the experimenter.…”

Section: Risks Associated With "Demand Characteristics"mentioning

confidence: 99%

Sixty years after Orne’s American Psychologist article: A Conceptual Framework for Subjective Experiences Elicited by Demand Characteristics.

Corneille¹,

Lush²

2021

Preprint

View full text Add to dashboard Cite

The present article proposes a critical discussion of “demand characteristics”. This notion, despite its longstanding and widespread use in behavioral research, suffers from significant conceptual ambiguities. After briefly discussing the historical roots of “demand characteristics” and stressing that this notion remains highly influential in psychological research (Section 1), we proceed to its conceptual analysis (Section 2). Here, we discuss the complex nature of “demand characteristics” regarding effects, mechanisms, and measures. Then, we propose a Multifaceted Demand Characteristics Model (Section 3), and we apply it to two research domains of high contemporary interest: Evaluative Conditioning (Section 4.1) and Rubber Hand Illusion (Section 4.2). In the General Discussion, we summarize the central insights of the current analysis and discuss when and how theorization around “demand characteristics” may prevent and help advance behavioral research and interventions.

show abstract

Section: Risks Associated With "Demand Characteristics"mentioning

confidence: 99%

Sixty years after Orne’s American Psychologist article: A Conceptual Framework for Subjective Experiences Elicited by Demand Characteristics.

Corneille¹,

Lush²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Thus, it seems reasonable to suggest that the compelling replication results should have reduced belief in the credibility of the original results. However, one could also contend that if there were some flaw in the replication study that undermined its validity (Brandt et al, 2014;Vazire et al, 2021;Fabrigar et al, 2020), then this may provide justification to continue favourably citing the original study. Indeed, in these particular case studies, the validity of the replication results has been challenged by proponents of the original findings (Table 5; Baumeister & Vohs, 2016;.…”

Section: Discussionmentioning

confidence: 99%

Citation patterns following a strongly contradictory replication result: Four case studies from psychology

Hardwicke¹,

Szűcs²,

Thibault³

et al. 2021

Preprint

View full text Add to dashboard Cite

Replication studies that contradict prior findings may facilitate scientific self-correction by triggering a reappraisal of the original studies; however, the research community's response to replication results has not been studied systematically. One approach for gauging responses to replication results is to examine how they impact citations to original studies. In this study, we explored post-replication citation patterns in the context of four prominent multi-laboratory replication attempts published in the field of psychology that strongly contradicted and outweighed prior findings. Generally, we observed a small post-replication decline in the number of favourable citations and a small increase in unfavourable citations. This indicates only modest corrective effects and implies considerable perpetuation of belief in the original findings. Replication results that strongly contradict an original finding do not necessarily nullify its credibility; however, one might at least expect the replication results to be acknowledged and explicitly debated in subsequent literature. By contrast, we found substantial citation bias: the majority of articles citing the original studies neglected to cite relevant replication results. Of those articles that did cite the replication, but continued to cite the original study favourably, approximately half offered an explicit defence of the original study. Our findings suggest that even replication results that strongly contradict original findings do not necessarily prompt a corrective response from the research community.

show abstract

“…For example, as the construct of interest changes from hostility to another trait, as the priming method changes from the sentence descrambling task to another priming method, as the outcome variable changes from rating a described individual to another outcome, or some combination of these changes, it requires more assumptions to generalize from the current results to others. These mounting assumptions make loose links between the results of the current study and these other studies and, thus, these mounting assumptions could make the ultimate impact of the current results more of a glance than a wallop (e.g., see Fabrigar et al, 2020).…”

Section: What Does the Lack Of An Effect Mean? What Does The Lack Of mentioning

confidence: 92%

A Multi-Site Collaborative Study of the Hostile Priming Effect

McCarthy¹,

Gervais²,

Aczél³

et al. 2021

Collabra: Psychology

View full text Add to dashboard Cite

In a now-classic study by Srull and Wyer (1979), people who were exposed to phrases with hostile content subsequently judged a man as being more hostile. And this “hostile priming effect” has had a significant influence on the field of social cognition over the subsequent decades. However, a recent multi-lab collaborative study (McCarthy et al., 2018) that closely followed the methods described by Srull and Wyer (1979) found a hostile priming effect that was nearly zero, which casts doubt on whether these methods reliably produce an effect. To address some limitations with McCarthy et al. (2018), the current multi-site collaborative study included data collected from 29 labs. Each lab conducted a close replication (total N = 2,123) and a conceptual replication (total N = 2,579) of Srull and Wyer’s methods. The hostile priming effect for both the close replication (d = 0.09, 95% CI [-0.04, 0.22], z = 1.34, p = .16) and the conceptual replication (d = 0.05, 95% CI [-0.04, 0.15], z = 1.15, p = .58) were not significantly different from zero and, if the true effects are non-zero, were smaller than what most labs could feasibly and routinely detect. Despite our best efforts to produce favorable conditions for the effect to emerge, we did not detect a hostile priming effect. We suggest that researchers should not invest more resources into trying to detect a hostile priming effect using methods like those described in Srull and Wyer (1979).

show abstract

A Validity-Based Framework for Understanding Replication in Psychology

Cited by 59 publications

References 152 publications

Sixty years after Orne’s American Psychologist article: A Conceptual Framework for Subjective Experiences Elicited by Demand Characteristics.

Sixty years after Orne’s American Psychologist article: A Conceptual Framework for Subjective Experiences Elicited by Demand Characteristics.

Citation patterns following a strongly contradictory replication result: Four case studies from psychology

A Multi-Site Collaborative Study of the Hostile Priming Effect

Contact Info

Product

Resources

About