A Bayesian bird's eye view of ‘Replications of important results in social psychology’

Marsman, Maarten; Schönbrodt, Felix D.; Morey, Richard D.; Yao, Yuling; Gelman, Andrew; Wagenmakers, Eric‐Jan

doi:10.1098/rsos.160426

Cited by 52 publications

(48 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Validity of research methods is a timely topic. Large‐scale replication projects in psychology and biomedicine have revealed that a disturbingly low percentage of top journal studies can be successfully replicated (Marsman et al, ; Open Science Collaboration, ). Our study may offer some interesting opportunities for so‐called quasi‐replications where the robustness of previous findings are tested against different methods (Bettis et al, ).…”

Section: Discussionmentioning

confidence: 99%

Are Survey Experiments as Valid as Field Experiments in Management Research? An Empirical Comparison Using the Case of Ethnic Employment Discrimination

Wulff¹,

Villadsen²

2019

European Management Review

View full text Add to dashboard Cite

Field experiments have long been the gold standard in studies of organizational topics such as ethnic discrimination in recruitment. The recent use of survey experiments, also known as experimental vignettes, suggests that some researchers believe that survey experiments could be used as an alternative to field experiments. In this study we put this notion to the test. We perform a field experiment followed by two survey experiments on ethnic discrimination in recruitment. While the results of our field experiment are consistent with previous evidence on discrimination, one survey experiment concludes no difference between native and immigrant employees while another concludes positive discrimination. These results should invoke caution in researchers wanting to investigate organizational topics using survey experiments.We thank an anonymous reviewer for suggesting that the mentioned design difference might confound our results. Field and survey experiments

show abstract

Section: Discussionmentioning

confidence: 99%

Are Survey Experiments as Valid as Field Experiments in Management Research? An Empirical Comparison Using the Case of Ethnic Employment Discrimination

Wulff¹,

Villadsen²

2019

European Management Review

View full text Add to dashboard Cite

show abstract

“…In recent years, a key question in psychology and neuroscience has concerned the credibility of reported findings (Open Science, 2015;Button et al, 2013;Pashler et al, 2012;Munafo et al, 2017;Vazire, 2018) with estimates of replicability ranging between 25 and 75% (Nosek & Lakens, 2014;Matzke et al, 2015;Marsman et al, 2017;Camerer et al, 2018). Studies that integrate experimental and differential approaches are rare in general, and in the context of imitation control, prior studies have typically used small sample sizes (Ainley et al, 2014;Chartrand & Bargh, 1999;Obhi et al, 2013;Santiesteban et al, 2015).…”

Section: Are Individual Differences In Interference Control Robust Anmentioning

confidence: 99%

Darda_individual_differences_cognitive_control

Darda¹,

Butler²,

Ramsey³

2019

Preprint

View full text Add to dashboard Cite

Cognitive control refers to the ability of human beings to adapt flexibly and quickly to continuously changing environments. Several decades of research have identified a diverse range of mental processes that are associated with cognitive control but the extent to which shared systems underlie cognitive control in social and non-social contexts, as well as how these systems may vary across individuals, remains largely unexplored. By integrating methodological approaches from experimental and differential psychology, the current study is able to shine new light on the relationships between stable features of individuals, such as personality and sex, and the architecture of cognitive control systems using paradigms that index social (automatic imitation) and spatial processes. Across three large-sample experiments (>600 participants in total), we demonstrate that cognitive control systems are largely invariant to stable aspects of personality, but exhibit a sex difference, such that females show greater task-interference than males. Moreover, we further qualified this sex difference in two ways. First, we showed that the sex difference was unrelated to the sex of the interaction partner and therefore did not reflect an in-group bias based on sex. Second, we showed that the sex difference was tied to a form of spatial interference control rather than social (imitative) control and therefore it does not reflect a specialised mechanism for guiding social interactions exclusively. Instead, our findings suggest that a robust sex difference exists in the system (or set of subsystems) that operate in resolving a form of spatial interference control, and that such systems are unaffected by social factors such as the sex of the interaction partner. The results highlight the value of integrating approaches from experimental and differential psychology by providing a deeper understanding of the structure of cognitive control systems, whilst also providing new dimensions to incorporate into theories and models of social and non-social control.

show abstract

“…For instance, the Open Science Collaboration (2015) showed that out of 100 replication studies, only 39 supported the conclusions that were drawn in the original article (but see Etz and Vandekerckhove 2016; see also Camerer et al 2018). Similarly disappointing results were obtained for specific subfields (e.g., Marsman et al 2017;Klein et al 2018;Nosek and Lakens 2014), and for particular effects (e.g., Eerland et al 2016;Harris et al 2013;Matzke et al 2015;Meyer et al 2015;de Molière and Harris 2016;Shanks et al 2013;Unsworth et al 2015;Wagenmakers et al 2016, among many others).…”

mentioning

confidence: 95%

Flexible yet fair: blinding analyses in experimental psychology

2019

Self Cite

View full text Add to dashboard Cite

The replicability of findings in experimental psychology can be improved by distinguishing sharply between hypothesis-generating research and hypothesis-testing research. This distinction can be achieved by preregistration, a method that has recently attracted widespread attention. Although preregistration is fair in the sense that it inoculates researchers against hindsight bias and confirmation bias, preregistration does not allow researchers to analyze the data flexibly without the analysis being demoted to exploratory. To alleviate this concern we discuss how researchers may conduct blinded analyses (MacCoun and Perlmutter in Nature 526:187-189, 2015). As with preregistration, blinded analyses break the feedback loop between the analysis plan and analysis outcome, thereby preventing cherry-picking and significance seeking. However, blinded analyses retain the flexibility to account for unexpected peculiarities in the data. We discuss different methods of blinding, offer recommendations for blinding of popular experimental designs, and introduce the design for an online blinding protocol.

show abstract

A Bayesian bird's eye view of ‘Replications of important results in social psychology’

Cited by 52 publications

References 74 publications

Are Survey Experiments as Valid as Field Experiments in Management Research? An Empirical Comparison Using the Case of Ethnic Employment Discrimination

Are Survey Experiments as Valid as Field Experiments in Management Research? An Empirical Comparison Using the Case of Ethnic Employment Discrimination

Darda_individual_differences_cognitive_control

Flexible yet fair: blinding analyses in experimental psychology

Contact Info

Product

Resources

About