A primer on running human behavioural experiments online

Abstract

Moving from the lab to an online environment opens up enormous potential to collect behavioural data from thousands of participants with the click of a button. However, getting the first online experiment running requires familiarisation with a number of new tools and terminologies. There exist a number of tutorials and hands-on guides that can facilitate this process, but these are often tailored to one specific online platform. The aim of this paper is to give a broad introduction to the world of online testing. This will provide a high-level understanding of the infrastructure before diving into specific details with more in-depth tutorials. Becoming familiar with these tools allows moving from hypothesis to experimental data within hours.

Investigating orthographic versus auditory cross-situational word learning with online and lab-based research

“…The analyses were conducted in the statistical program R (R Core Team, 2014) with the brms package (using Stan) (Bürkner, 2017(Bürkner, , 2018 R Core Team, 2014). Similar approaches have been used previously in Escudero et al, (2020) and Smit et al, (2019;2020;2020a). A comprehensive overview of the methods is described in Smit et al, (2019) and Escudero et al, (2020).…”

“…In contrast with null-hypothesis significant testing (NHST), we here assess the evidence for a hypothesis by using evidence ratios (probability ratios or odds) which quantify the likelihood of the tested hypothesis against its alternative (Escudero et al, 2020;Smit et al, 2019;2020;2020a). Following Smit et al, (2019;2020;2020a,), and Escudero et al, (2020), we also use the guidelines that were proposed by Jeffreys (1998) as cited by Kruschke (2018):…”

“…In recent years, the interest in running behavioural experiments online has been growing steadily in different fields (Gentili & Cristea, 2020;Grootswagers, 2020;Mason & Suri, 2012) as well as in psychology (Birnbaum, 2000;Nosek, 2007). Web-based research offers a lot of opportunities, but also comes with some consequences.…”

“…Web-based research offers a lot of opportunities, but also comes with some consequences. Concerns have been expressed about various aspects of online experiments, such as the lack of experimental control (Grootswagers, 2020), the control over stimulus presentation (Woods et al, 2015), participant's distance from the screen and their attention to the experiment (Grootswagers, 2020). The lack of control over whether a participant is paying attention to the experiment can lead to high variability in the results (Grootswagers, 2020).…”

“…Concerns have been expressed about various aspects of online experiments, such as the lack of experimental control (Grootswagers, 2020), the control over stimulus presentation (Woods et al, 2015), participant's distance from the screen and their attention to the experiment (Grootswagers, 2020). The lack of control over whether a participant is paying attention to the experiment can lead to high variability in the results (Grootswagers, 2020). There has also been doubt as to whether the quality of data collected online compares to that in the lab, with some studies reporting positive experiences (Grootswagers et al, 2018;Grootswagers et al, 2020;Grootswagers et al, 2017), and others reporting mixed results (Crump et al, 2013).…”