Tracing the Trajectory of Skill Learning With a Very Large Sample of Online Game Players

Abstract

We analyze data from a very large (n=854064) sample of players of an online game involving rapid perception, decision-making and motor responding. Use of game data allows us to connect, for the first time, rich details of training history with measures of performance, for participants who are engaged for a sustained amount of time in effortful practice. We show that lawful relations exist between practice amount and subsequent performance, and between practice spacing and subsequent performance. This allows an in-situ confirmation of results long established in the experimental literature on skill acquisition. Additionally, we show that higher initial variation in performance is linked to subsequent higher performance, a result we link to the exploration-exploitation trade-off from the computational framework of reinforcement learning. We discuss the benefits and opportunities of behavioral datasets with very large sample sizes and suggest that this approach could be particularly fecund for studies of skill acquisition.

Beyond Time-on-Task: The Relationship Between Spaced Study and Certification in MOOCs

“…This is because the bootstrapping procedure preserves the conditional dependence of certification on the bin of total time and also preserves the relationship between session count and total time, but removes the relationships of certification with all else. Therefore, observing data that deviates from this null distribution would indicate that session count predicts certification at a level beyond what would be expected through its relationship with total time (see Figure 3 of Stafford & Dewar, 2014, for an analogous use of bootstrap). Figure 5 shows the results of applying this bootstrap analysis.…”

“…In an effort to accelerate integration in practice, studies have appeared in recent years of real-world classroom demonstrations of the spacing effect in many contexts, such as vocabulary learning (Bird, 2010;Carpenter, Pashler, & Cepeda, 2009;Gallo & Odu, 2009;Kang, Lindsey, Mozer, & Pashler, 2014;Khajah, Lindsey, & Mozer, 2014;Lindsey, Shroyer, Pashler, & Mozer, 2014;Rohrer, 2009;Rohrer & Taylor, 2006;Sobel, Cepeda, & Kapler, 2011), fraction learning (Rau, Aleven, Rummel, & Pardos, 2014), and skill acquisition (Moulton et al, 2006;Stafford & Dewar, 2014). Technology has enabled what is perhaps the most visible application of the spacing effect today: spaced repetition software (SPS).…”

“…This maintains the relationship between total time and certification while removing the relationships of certification with all else. We then test whether the observed data deviates from the null distribution (see Figure 3 of Stafford & Dewar, 2014, for an analogous use of bootstrap).…”

Action video game training for cognitive enhancement

“…For this, intervention studies are conducted wherein individuals who do not naturally play video games are first pre-tested on measures of interest before being randomly assigned to play either an action video game or a control video game (a commercial game matched for general interest, flow, arousal, among others, but lacking all action components -see Figure 1). Participants then play their assigned game for a set period of time; work in the field has utilized training durations from 10 to 50 hours spaced over the course of weeks to months -as video game training, like all other forms of learning is far more effective when practice is distributed rather than massed [18,19]. Finally, at least 24 hours after the final play session (the delay ensures that any transient effects of game play are eliminated as potential concerns), the individuals are post-tested on the measures of interest with the critical question being whether the action trained group improves more from pre-test to posttest than the control video game trained group.…”

Model Matching Theory: A Framework for Examining the Alignment between Game Mechanics and Mental Models

“…Further indirect evidence for iterative refinement of mental models manifests as learning curves of game performance (Ritter & Schooler, 2001). Over time, as players repeatedly engage with games in various ways-including play, solitary and group practice, instruction, and the use of other resources-their skill and game performance gradually improve (Charness et al, 2005;Stafford & Dewar, 2014). In line with the notion of productive failure, exploring the possibilities and limitations of game models supports mental model refinement during both initial encounters with a game and as players face later challenges.…”