Video Game Telemetry as a Critical Tool in the Study of Complex Skill Learning

Abstract

Cognitive science has long shown interest in expertise, in part because prediction and control of expert development would have immense practical value. Most studies in this area investigate expertise by comparing experts with novices. The reliance on contrastive samples in studies of human expertise only yields deep insight into development where differences are important throughout skill acquisition. This reliance may be pernicious where the predictive importance of variables is not constant across levels of expertise. Before the development of sophisticated machine learning tools for data mining larger samples, and indeed, before such samples were available, it was difficult to test the implicit assumption of static variable importance in expertise development. To investigate if this reliance may have imposed critical restrictions on the understanding of complex skill development, we adopted an alternative method, the online acquisition of telemetry data from a common daily activity for many: video gaming. Using measures of cognitive-motor, attentional, and perceptual processing extracted from game data from 3360 Real-Time Strategy players at 7 different levels of expertise, we identified 12 variables relevant to expertise. We show that the static variable importance assumption is false - the predictive importance of these variables shifted as the levels of expertise increased - and, at least in our dataset, that a contrastive approach would have been misleading. The finding that variable importance is not static across levels of expertise suggests that large, diverse datasets of sustained cognitive-motor performance are crucial for an understanding of expertise in real-world contexts. We also identify plausible cognitive markers of expertise.

Masters of Control

“…The trinary classification performance when estimating players as {novice, proficient, expert} is comparable to the binary classification performance in [19]. This classifier enables a rapid (requires only a single match) and coarse estimation of skill that can supplement existing methods of matchmaking or skill estimation for online players.…”

“…We also extend this work by using our data to build a trinary classifier of player skill following the binary classifiers introduced in Thompson et al. [19]. In addition to the variation of replay data with player skill levels studied in Thompson et al, we also consider the variation between players at a similar skill level.…”

“…StarCraft 2 replay data has been studied previously to identify which characteristics explain a player's skill level [19]. Several features that were highly relevant to player skill were identified, including the "Perception-Action Cycle" (PAC), and actions per minute (APM).…”

“…We adopt Thompson et al's approach [19] and avoid differentiating between adjacent leagues. 2 We are estimating 1v1 ratings; Blizzard's rating system assigns a different rating for each mode of gameplay e.g.…”

Distributed learning for Random Vector Functional-Link networks

“…As we stated in the introductory section, in the distributed case we can assume that the songs are present over different computers, and we can use our strategies as a way of leveraging over the entire dataset without a centralized controller. Skills is a regression dataset taken from the UCI repository [39]. The task is to assess the skill level of a video game user, based on a set of recordings of its actions in the video game itself.…”