Gorilla in our midst: An online behavioral experiment builder

Abstract

Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing-an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test-retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform's aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant's own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the Bconflict network^effect in all these populations, demonstrating the platform's capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform.

Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online

“…Additionally, PsychoPy auto-translates base-Python to JS (but not functions from specific libraries). There are plenty other experiment builders available: Gorilla [24], Inquisit Web [25], LabVanced [26] and Testable [27], from the integrated-service providers (see Table 1: A) and lab.js [28,29], jsPsych [30, 31], PsyToolkit [32][33][34], tatool web [35] from the function-specific solutions (see Table 1: B). Their advantages and shortcomings should be evaluated on a lab's basis depending on individual needs.…”

A primer on running human behavioural experiments online

“…Our lab has had an overwhelmingly positive experience with running online studies (Grootswagers et al, in press, 2017(Grootswagers et al, in press, , 2018. While early days involved extensive JavaScript programming for relatively simple online studies, recent advancements have made it much easier to get complex studies up and running (Anwyl-Irvine, Massonnié, et al, 2020;Barnhoorn et al, 2014;De Leeuw, 2015;Henninger et al, 2019;Peirce et al, 2019). These generally come with associated tutorials and hands-on guides, but these are often specific to a single platform or method.…”

Reaction Time Data in Music Cognition: Comparison of Pilot Data From Lab, Crowdsourced, and Convenience Web Samples

“…Traditionally, reaction time data has been collected in a lab. However, recent years have seen the development of software capable of collecting accurate response time data online, for instance PsyToolkit (Stoet, 2010(Stoet, , 2017, PsychoPy (Peirce et al, 2019), Gorilla (Anwyl-Irvine et al, 2019), and Qualtrics' QRTEngine (Barnhoorn et al, 2015) amongst others. In the early days of web-based reaction time studies, there was considerable skepticism about the viability of RT data collected online.…”