Automating creativity assessment with SemDis: An open platform for computing semantic distance

Abstract

Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants' responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitationslabor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Studies 1-3) and novelty/creativity ratings from two word association tasks (Studies 4-5). We find that a latent semantic distance factor-comprised of the common variance from five semantic models-reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation (https://osf.io/gz4fc/).

The Relationship between Intelligence and Divergent Thinking—A Meta-Analytic Update

“…Since DT is considered to be a marker of creative potential ( Runco and Acar 2012 ), the assessment of the originality of the responses seems required (e.g., Zeng et al 2011 ) and, as a matter of fact, is provided in most of the recent studies. All scoring methods have their weaknesses, and the development of new methods (e.g., based on corpus semantics; Beaty and Johnson 2020 ; Dumas et al 2020 ) continues, and we can be curious about how the weaknesses of the current methods will be mitigated.…”

The Influence of Situational Cues on Children’s Creativity in an Alternative Uses Task and the Moderating Effect of Selective Attention

“…The original answers were mostly mentioned by only 5% or less of the children, while most of the unoriginal responses were produced by 15 to 20% or more, which is clearly above the 10% threshold. However, to avoid a lack of measurement precision, intersubjective rating of originality or automated scoring is recommended in future research on creativity (for more information, see Beaty and Johnson 2020 or Dumas et al 2020 ). Second, the task instruction may have primed the children in the high stimulus condition too strongly to outward perceptual processing while a mix of outward and inward memory-based processing could have promoted originality better.…”

Naming unrelated words reliably predicts creativity

“…To address these limitations, recent efforts have moved towards using computational algorithms to score responses (10,16,25,26). Computational methods may also improve the theoretical grounding of the measures, as the assumptions required to score the responses must be made explicit in the program code (10,12).…”

“…Computational methods may also improve the theoretical grounding of the measures, as the assumptions required to score the responses must be made explicit in the program code (10,12). Researchers have successfully scored the Alternative Uses Task using semantic models, achieving scores similar to human ratings (10,16,25,26).…”