Digital Games for Creativity Assessment: Strengths, Weaknesses and Opportunities

Rafner, Janet; Biskjær, Michael Mose; Zana, Blanka; Langsford, Steven; Bergenholtz, Carsten; Rahimi, Seyedahmad; Carugati, Andrea; Noy, Lior; Sherson, Jacob

doi:10.1080/10400419.2021.1971447

Cited by 20 publications

(11 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may dampen generalizability, as the specific behaviors or patterns observed in a game may be a result of the incentive structure of that specific game, rather than reflecting general inductive biases, and may not generalize to other contexts. If using an existing popular game, it may therefore be important to consider complementary games with similar dynamics but different incentives, or by using batteries of games to build cognitive ontologies 57,58 .…”

Section: Games As Intuitive Engineered Environmentsmentioning

confidence: 99%

“…However, if intrinsic motivation is important, standard platforms can be problematic and we suggest recruiting participants over social media, to put the game on different app stores, or -if necessary-to work with a game company directly. Publicly announcing that results are going to be used for research purposes may also call citizen scientists into action 88 , thus further engaging and diversifying participants 58 . Moreover, researchers can also partner with organizations and charities to reach out to different communities to play games.…”

Section: How To Use Games As a Research Platformmentioning

confidence: 99%

See 1 more Smart Citation

Using Games to Understand the Mind

Allen

Brändle²,

Botvinick³

et al. 2023

Preprint

View full text Add to dashboard Cite

Video games are played by over 2 billion people spread across the world population, with both children and adults participating. Games have gained popularity as an avenue for studying cognition. We believe that studying cognition using games can generate progress in psychology and in neuroscience similar to the one that has occurred in artificial intelligence research over the past decades. Using games to understand the mind enables researchers to scale up theories of cognition to more complex settings, reverse-engineer human inductive biases, create experiments that participants want to take part in, and study learning over long time horizons. We describe both the advantages and drawbacks of using games relative to standard lab-based experiments, and lay out a set of recommendations on how to gain the most from using games to study cognition. We hope that this article will lead to a wider use of games as experimental paradigms, elevating the complexity, robustness, and external validity of research on the mind.

show abstract

Section: Games As Intuitive Engineered Environmentsmentioning

confidence: 99%

Section: How To Use Games As a Research Platformmentioning

confidence: 99%

Using Games to Understand the Mind

Allen

Brändle²,

Botvinick³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To address the challenges of subjective scoring, a growing number of researchers are exploring automated scoring methods (Acar & Runco, 2014; Acar et al, 2023; Beaty & Johnson, 2021; Dumas & Runco, 2018; Dumas, Organisciak, & Doherty, 2021; Dumas, Organisciak, Maio, & Doherty, 2021; Forthmann & Doebler, 2022; Gray et al, 2019; Johnson, Kaufman, et al, 2021; Kenett, 2019; Olson et al, 2021; Paulus et al, 1970; Prabhakaran et al, 2014; Rafner et al, 2022; Shute & Rahimi, 2021; Stevenson et al, 2020; Sung et al, 2022; Yu et al, 2023; Zedelius et al, 2019). One prominent approach is to compute semantic distance using distributional semantic models—a class of natural language processing tools that quantifies conceptual similarity in texts (Günther et al, 2019; Jackson et al, 2022).…”

Section: Human and Machine Assessment Of Verbal Creativitymentioning

confidence: 99%

“…To address these issues, researchers are increasingly exploring computational methods for automating the scoring process (Acar & Runco, 2014; Acar et al, 2023; Beaty & Johnson, 2021; Beaty et al, 2021; Bendetowicz et al, 2018; Bossomaier et al, 2009; Dumas, Organisciak, & Doherty, 2021; Forster & Dunbar, 2009; Forthmann & Doebler, 2022; Gray et al, 2019; Heinen & Johnson, 2018; Johnson, Kaufman, et al, 2021; Olson et al, 2021; Paulus et al, 1970; Prabhakaran et al, 2014; Rafner et al, 2022; Shute & Rahimi, 2021; Stevenson et al, 2020; Sung et al, 2022; Volle, 2018; Yu et al, 2023; Zedelius et al, 2019). One promising approach employs distributional semantic models to compute semantic distance , which quantifies how “far away” an idea is from common ideas (Kenett, 2019).…”

mentioning

confidence: 99%

Multilingual semantic distance: Automatic verbal creativity assessment in many languages.

Patterson,

Merseal,

Johnson

et al. 2023

Psychology of Aesthetics, Creativity, and the Arts

View full text Add to dashboard Cite

Creativity research commonly involves recruiting human raters to judge the originality of responses to divergent thinking tasks, such as the alternate uses task (AUT). These manual scoring practices have benefited the field, but they also have limitations, including labor-intensiveness and subjectivity, which can adversely impact the reliability and validity of assessments. To address these challenges, researchers are increasingly employing automatic scoring approaches, such as distributional models of semantic distance. However, semantic distance has primarily been studied in English-speaking samples, with very little research in the many other languages of the world. In a multilab study (N = 6,522 participants), we aimed to validate semantic distance on the AUT in 12 languages: Arabic, Chinese, Dutch, English, Farsi, French, German, Hebrew, Italian, Polish, Russian, and Spanish. We gathered AUT responses and human creativity ratings (N = 107,672 responses), as well as criterion measures for validation (e.g., creative achievement). We compared two deep learning-based semantic models—multilingual bidirectional encoder representations from transformers and cross-lingual language model RoBERTa—to compute semantic distance and validate this automated metric with human ratings and criterion measures. We found that the top-performing model for each language correlated positively with human creativity ratings, with correlations ranging from medium to large across languages. Regarding criterion validity, semantic distance showed small-to-moderate effect sizes (comparable to human ratings) for openness, creative behavior/achievement, and creative self-concept. We provide open access to our multilingual dataset for future algorithmic development, along with Python code to compute semantic distance in 12 languages.

show abstract

“…By its very nature free play is unconstrained, so it is not trivial to analyze and model how it manifests in the wild, and it is not obvious to cast its free form within the logic of lab experiments. Typical experimental approaches for studying creativity are product based: creativity is measured in terms of the quality or quantity of the resulting objects (Kaufman and Sternberg, 2019;Rafner et al, 2022). In contrast, group free play usually does not produce any tangible outcomes.…”

Section: Introductionmentioning

confidence: 99%

A distributed model of collective creativity in free play

Kalaydjian¹,

Laroche

Noy

et al. 2022

Front. Educ.

Self Cite

View full text Add to dashboard Cite

A group of children engaged in collective free play can spontaneously create new rules, learn to follow them, or find opportunities to break established ones. This rule-playing can be considered as a specific manifestation of the more general phenomenon of collective creativity. In behavioral sciences, collective creativity is often discussed as a collection of individuals each being creative. An alternative perspective views collective creativity as a distributed phenomenon: collective creativity is not a property of individual agents but rather, it emerges from the interactions within a group. Approaching free play as a case of distributed collective creativity, we understand rule-playing in terms of two complementary modes – group exploration and group exploitation-, and the transition between them. Free play is not easily amenable to fine-grained observational analysis. To overcome this, we developed the Grid Game, a new experimental setup which supports detailed empirical investigation while preserving the essence of free play. The Grid Game is a group improvisation game that uses the turn taking logic and spatial organization of typical board games, without any other predefined rules. Small groups of kids (4–5 participants) took turns in freely moving or manipulating a provided set of objects on a large 4 × 4 grid on the floor for 10 min, while being video-recorded. Despite the absence of predefined goals, simple proto-games with ad hoc rules often emerge, for example, placing objects on top of each other (create a tower) or an aiming-to-a-target shooting game. We propose an analysis of the emergence of such proto-games in terms of group exploration and exploitation. Building on our previous work on the Creative Foraging Game, we focus on cases of transition from exploration to exploitation underlying the discovery of a new form or rule. Based on Choreographer João Fiadeiro’s body of work, we describe these phase shifts as a distributed process composed of three stages: (1) Suggestion, (2) Recognition, and (3) Confirmation. We provide detailed descriptions of game moments according to this model which demonstrate the distributed nature of collective creativity in free play.

show abstract

Digital Games for Creativity Assessment: Strengths, Weaknesses and Opportunities

Cited by 20 publications

References 95 publications

Using Games to Understand the Mind

Using Games to Understand the Mind

Multilingual semantic distance: Automatic verbal creativity assessment in many languages.

A distributed model of collective creativity in free play

Contact Info

Product

Resources

About