How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?

Abstract

For citizen science games (CSGs) to be successful in advancing scientific research, they must effectively train players. Designing tutorials for training can be aided through developing a skill chain of required skills and their dependencies, but skill chain development is an intensive process. In this work, we hypothesized that free recall may be a simpler yet effective method of directly eliciting skill chains. We elicited 23 skill chains from players and developers and augmented our reflexive thematic analysis with 11 semi-structured interviews in order to determine how players and developers conceptualize skill trees and whether free recall can be used as an alternative to more resource-intensive cognitive task analyses. We provide three main contributions: (1) a comparison of skill chain conceptualizations between players and developers and across prior literature; (2) insights to the process of free recall in eliciting CSG skill chains; and (3) a preliminary toolkit of CSG skill-based design recommendations based on our findings. We conclude CSG developers should: give the big picture up front; embrace social learning and paratext use; reinforce the intended structure of knowledge; situate learning within applicable, meaningful contexts; design for discovery and self-reflection; and encourage practice and learning beyond the tutorial. Free recall was ineffective for determining a traditional skill chain but was able to elicit the core gameplay loops, tutorial overviews, and some expert insights.

A Survey of Citizen Science Gaming Experiences

“…Throughout all participant feedback, their responses highlighted flaws with the current game instruction, both because participants were confused about how to play and because they didn't understand the science of the game, despite wanting to. This agrees with our initial hypothesis that the player experience is one of frustration, and indicates a need for better teaching of the big picture and the science-game loop, or contribution model (Miller et al 2021). This was identified especially in Foldit's tutorial, whose instructions were not thorough enough, not connected to real science, and violated standard playability heuristics -such as taking away tools the player had earned, inconsistent gameplay, and unintuitive controls (Desurvire and Wiberg 2009;Korhonen and Koivisto 2006) -all of which can create further confusion by not meeting standards.…”

“…It is important to teach the core gameplay loop and scientific contribution model early (cf. Miller et al 2021) and iteratively refine your instructions and communication, especially if the project evolves over several years (Keep 2018). Scientific communication is critical since it feeds into the satisfaction of making scientific contributions and can also teach and inform players.…”

Identifying problems in onboarding design for expertise-centric citizen science games

“…Throughout all participant feedback, their responses highlighted flaws with the current game instruction, both because participants were confused about how to play and because they didn't understand the science of the game, despite wanting to. This agrees with our initial hypothesis that the player experience is one of frustration, and indicates a need for better teaching of the big picture and the science-game loop, or contribution model [334]. This was identified especially in Foldit's tutorial, whose instructions were not thorough enough, not connecting to real science, and violating standard playability heuristics -such as taking away tools the player had earned, inconsistent gameplay, and unintuitive controls [129,275] -all of which can create further confusion by not meeting standards.…”

“…It is important to teach the core gameplay loop and scientific contribution model early (cf. [334]) and iteratively refine your instructions and communication, especially if the project evolves over several years [255]. Scientific communication is critical since it feeds into the satisfaction of making scientific contributions and can also teach and inform players.…”

Effects of Player-Level Matchmaking Methods in a Live Citizen Science Game

“…Given these hypotheses, we decided to perform our experiment in Foldit, a live and more complex CSG with a large player base (Miller et al 2021;Curtis 2015). Its tasks involve protein folding, which is often non-intuitive for people without prior biochemistry knowledge, and its puzzles are not as linear as previously-researched CSGs (Miller et al 2021;Khatib et al 2011).…”