Moving Developmental Research Online: Comparing In-Lab and Web-Based Studies of Model-Based Reinforcement Learning

Abstract

For years, adult psychological research has benefitted from web-based data collection. There is growing interest in harnessing this approach to facilitate data collection from children and adolescents to address foundational questions about cognitive development. To date, however, few studies have directly tested whether findings from in-lab developmental psychology tasks can be replicated online, particularly in the domain of value-based learning and decision-making. To address this question, we set up a pipeline for online data collection with children, adolescents, and adults, and conducted a replication of Decker et al. (2016). The original in-lab study employed a sequential decision-making paradigm to examine shifts in value-learning strategies from childhood to adulthood. Here, we used the same paradigm in a sample of 151 children (N = 50; ages 8 - 12 years), adolescents (N = 50; ages 13 - 17 years), and adults (N = 51; ages 18 - 25 years) and replicated the main finding that the use of a “model-based” learning strategy increases with age. In addition, we adapted a new index of abstract reasoning (MaRs-IB; Chierchia et al. 2019) for use online, and replicated a key result from Potter et al. (2017), which found that abstract reasoning ability mediated the relation between age and model-based learning. Our re-analyses of two previous in-lab datasets alongside our analysis of our online dataset revealed few qualitative differences across task administrations. These findings suggest that with appropriate precautions, researchers can effectively examine developmental differences in learning computations through unmoderated, online experiments.

Validating Habitual and Goal-Directed Decision-Making Performance Online in Healthy Older Adults

“…However, further work is needed to explore this. Nonetheless, to our knowledge, our study is the first to replicate this effect in an online study, although we note that Nussenbaum and colleagues also showed an age-effect of decision-making balance in children, adolescents, and young adults in an online version of the two-stage task (Nussenbaum et al, 2020). This suggests that the effect of age is quite robust and can be replicated both in-lab and online.…”

“…While the original two-stage Markov tasks were conducted in-person ( Gläscher et al, 2010 ; Daw et al, 2011 ), a handful of studies using the task have been conducted online ( Gillan et al, 2015 ; Nussenbaum et al, 2020 ). In general, web-based studies benefit from convenience (i.e., eliminating travel) and larger samples ( Berinsky et al, 2012 ).…”

“…Data collected through online experiments has also been shown to be comparable to the quality of data collected in person ( Crump et al, 2013 ). However, to our knowledge, only Nussenbaum et al (2020) has compared online to in-person study participation on the two-stage task in particular on children, adolescents, and young adults, but this has not yet been validated in the older adult population.…”

Can Musical Ability be Tested Online?

“…How similar are the findings from in-person and online experiments? Positive results come from an online study about reinforcement learning (Nussenbaum et al, 2020), which replicated a main effect of age from the original in-person study (Decker et al, 2016). In other developmental research, online data replicated a mediating role for abstract-reasoning ability in the link between age and model-based learning (Chierchia et al, 2019).…”

Research at a Distance: Replicating Semantic Differentiation Effects Using Remote Data Collection With Children Participants

“…Thus, it is important to ensure that -despite these potential sources of variability -data collected remotely with young children participants is comparable to data obtained from in-person assessments. While recent work has shown that remote data collection procedures can replicate the effects of lab-based studies in older children and adolescents (Nussenbaum et al, 2020), it remains an open question whether data collected remotely with young children is comparable to data obtained in-person.…”