Sample representation in the social sciences

Abstract

The social sciences face a problem of sample nonrepresentation, where the majority of samples consist of undergraduate students from Euro-American institutions. The problem has been identified for decades with little trend of improvement. In this paper, I trace the history of sampling theory. The dominant framework, called the design-based approach, takes random sampling as the gold standard. The idea is that a sampling procedure that is maximally uninformative prevents samplers from introducing arbitrary bias, thus preserving sample representation. I show how this framework, while good in theory, faces many challenges in application. Instead, I advocate for an alternative framework, called the model-based approach to sampling, where representative samples are those balanced in composition, however they were drawn. I argue that the modelbased framework is more appropriate in the social sciences because it allows for systematic assessment of imperfect samples and methodical improvement in resource-limited scientific contexts. I end with practical proposals of improving sample quality in the social sciences.

Sacrifice is a step beyond convenience: A review of convenience sampling in psychological research in Africa

“…This study reviewed and categorised the current use and reporting practices around convenience samples in African psychology journals. Overall results show that authors frequently used convenience sampling in the included articles, concurring with Hultsch et al (2002) and Zhao (2020) as the popular sampling method in psychology. It also seemed that journals listing sampling method as a requirement in manuscripts increased the used sampling method's reporting, highlighting the influence journals have on sample reporting practices.…”

“…The most popularly used sampling method in psychology is convenience sampling, a non-probability sampling method (Hultsch, MacDonald, Hunter, Maitland, & Dixon, 2002;Zhao, 2020). Convenience sampling collects data from whoever is willing to partake in a study, is the most approachable or is, in other ways, conveniently accessible to the researcher (Wienclaw, 2019).…”

“…Convenience sampling collects data from whoever is willing to partake in a study, is the most approachable or is, in other ways, conveniently accessible to the researcher (Wienclaw, 2019). These samples often consist of much-debated student samples (Ashraf & Merunka, 2017;Zhao, 2020) or organisation's researchers have a relationship with or any persons who are willing to complete their surveys (Landers & Behrend, 2015). The popular use of easily accessible samples may be a result of the publish or perish culture in academia.…”

“…To further complicate research for non-western populations, such as Africa, research samples in psychology are mostly western, educated, industrialised, rich and democratic (WEIRD) (Henrich, Heine, & Norenzayan, 2010), with minimal attempts in practice to increase diversity (Medin, Ojalehto, Marin, & Bang, 2017). Thus, the largest part of the world's population is underrepresented in psychology fields such as developmental psychology (Nielsen, Haun, Kärtner, & Legare, 2017), with most studies consisting of Euro-American samples (Zhao, 2020). Rad et al (2018) also found that in a high-impact international psychology journal only six articles were published with African samples in 2014 and none in 2017.…”

“…Sample response rates are also too often missing in articles which inhibit readers from assessing the quality of the sampling used in studies (Fisher & Sandell, 2015). However, Zhao (2020) encourages balance in what is stated in articles and requested from participants to make stating sampling characteristics in articles feasible and uphold the confidentiality and validity of the included sample. The lack of sample details and precision in sampling debilitates the science of psychology by hampering replication (Trafimow, 2018), which is impossible if study samples do not concur (Simons, Yuichi, & Lindsay, 2017;Stroebe, Gadenne, & Nijstad, 2018), thereby furthering the current replication crisis faced by the field (Open Science Collaboration, 2015).…”