A Problem in Theory and More: Measuring the Moderating Role of Culture in Many Labs 2

Abstract

The multi-site replication study, Many Labs 2 (ML2), attempted to test whether population, site and setting variability moderates the likelihood of replication and effect size. The analysis concluded that sample location and setting did not substantially affect the replicability of findings. In this paper, we raise several issues with the ML2 approach to adjudicating the effect of culture that cast doubt on this conclusion. These theoretical and methodological problems (pre-registered at https://osf.io/6exr4) involve the: (1) selection of studies and sample sites for replication that are not theory-driven, (2) sampling of mostly WEIRD people around the world, (3) conflation of participants’ cultural backgrounds with the country where the samples came from, (4) use of the WEIRD backronym by decomposing it into a scale, and (5) application of a mean split of that WEIRD variable. Moreover, simulations reveal strikingly low statistical power for detecting cultural influences in a multi-side study designed like ML2. We propose methodologies to address problems (3) to( 5) by re-analyzing the ML2 dataset using an alternative approach. These results suggest that tackling only some of the design problems is insufficient to overcome the underlying theoretical and methodological deficiencies. We conclude with specific recommendations for assessing the role of population variability in future multi-site studies that address evidentiary value and effect size.

Why music psychology needs larger and more diverse datasets and how citizen science can help

“…By pooling resources from multiple researchers for a single project, this approach can improve the ability to collect large samples, and has been effective for some empirical projects studying music (Hilton, Moser, et al, 2022;Yurdum et al, 2023;Ozaki et al, 2023). Although the diversity of samples obtained by such an approach is limited by the diversity of researchers in the team (Schimmelpfennig et al, 2023), so it is also advisable to think about diversity and inclusion in terms of broader questions of who participates in the research (Savage et al, 2023). Indeed, increasing the diversity of who participates in research is also important because biases in data are just as important as biases in the theories we approach such data with (Ewell, 2020).…”

Heterogeneity in effect size estimates: Empirical evidence and practical implications

“…For our estimates of population heterogeneity, an important caveat is that the reviewed multi-lab replication studies are typically based on university student samples from different western countries, which may involve lower population heterogeneity than in other settings [77][78][79] . Moreover, the multi-lab replication studies included in our review feature a relatively high share of null results, which may arti cially limit the scope of heterogeneity further.…”

More educated people are more WEIRD: Higher education predicts global cultural similarity to the United States and other WEIRD countries

“…However, the Many Labs 2 studies relied heavily on samples from the United States and university student samples when recruiting from outside the US. Their analyses did not adequately account for the socioeconomic background of the international participant samples, who are often more highly educated and richer than the majority of people in their country 12,67 . Our findings demonstrate that when trying to test the generalizability of psychological effects, especially effects that were initially documented in US samples, relying on university-educated samples is likely to provide only a liberal test of psychological universality, as highly-educated individuals respond more similar to US samples across many attitudes, beliefs, and behavior.…”

“…Notably, Klein et al's Many Labs 2 11 project attempted to test the variability in psychological effects across samples and settings by testing 125 samples from 36 countries, and concluded that there was little heterogeneity in effects between more vs. less WEIRD cultures. A recent critique by Schimmelpfennig et al 12 argues that Many Labs 2 was unable to test for cross-cultural variation due to several issues, including issues of power and sampling methodology. These issues notwithstanding, surveying a more diverse set of the worlds' countries may continue to underrepresent the world's true cultural diversity by oversampling, or exclusively sampling from, university-educated people.…”