Comparability, stability, and reliability of internet-based mental chronometry in domestic and laboratory settings

Abstract

The internet-based assessment of response time (RT) and error rate (ERR) has recently become a well-validated alternative to traditional laboratory-based assessment, because methodological research has provided evidence for negligible setting- and setup-related differences in RT and ERR measures of central tendency. However, corresponding data on potential differences in the variability of such performance measures are still lacking, to date. Hence, the aim of this study was to conduct internet-based mental chronometry in both poorly standardized domestic and highly standardized laboratory environments and to compare the variabilities of the corresponding performance measures. Using the Millisecond Inquisit4Web software, 127 men and women completed three different RT-based cognitive paradigms (i.e., go/no-go, two-back, and number-letter). Each participant completed all paradigms in two environments (i.e., at home and in the laboratory), with a time lag of seven days and in a counterbalanced order. Mixed-effects modeling was employed to estimate the between-setting variability across a comprehensive set of performance measures, including conventional measures of central tendency (i.e., mean RT and ERR) and further measures characterizing the joint distribution of RT/ERR. The latter measures were estimated using the diffusion model. The results suggested negligible differences between the domestic and laboratory settings. Thus, this study provides novel evidence suggesting that the statistical power of internet-based mental chronometry is commonly not compromised by increased environmental variance. The within- and between-session reliabilities were in a satisfactory range-that is, comparable to performance measures collected offline in laboratory settings. In consequence, our results support the broad applicability, robustness, and cost efficiency of mental chronometry assessment using the internet.

Empathy and schadenfreude in human–robot teams

“…One important consideration is the online nature of the present experiments. While online experiments can be robust and lead to reliable results that are comparable to lab-based experiments (Bridges et al, 2020;de Leeuw & Motz, 2016;Miller et al, 2018), and similar biases have been found in screen-based experiments using narratives or ostensible interaction (Cikara et al, 2014), the players in the present experiments were disembodied agents presented on a screen. Embodiment is a crucial factor that drives engagements with other humans (Schilbach et al, 2013) and robots (Wykowska et al, 2016).…”

Thinking Against Burnout? An Individual’s Tendency to Engage in and Enjoy Thinking as a Potential Resilience Factor of Burnout Symptoms and Burnout-Related Impairment in Executive Functioning

“…Participants gave written informed consent and received either monetary rewards (18€) or course credit for their participation. Participants worked on several questionnaires and three executive function tasks measuring inhibition, shifting, and working memory updating (for task description, see below) once in the laboratory and once at home with 1 week in between and in a randomized order because a further goal was to measure whether task performance differed between home compared to the lab which is reported elsewhere (Miller et al, 2018). There were no substantial differences between the lab and home context.…”

“…If not, they had to press the key “M” (non-target). Each position was presented for 2.5 s. Each of the cognitive tasks started with a short training period and lasted about 10 min (for a more detailed task description, see Miller et al, 2018).…”

Gorilla in our Midst: An online behavioral experiment builder

“…In fact, several researchers have provided evidence that response times are comparable between browser-based applications and local applications (Barnhoorn, Haasnoot, Bocanegra, & Steenbergen, 2015) even in poorly standardized domestic environments -i.e. at home (Miller, Schmidt, Kirschbaum, & Enge, 2018).…”