Scrutinizing the basis of originality in divergent thinking tests: On the measurement precision of response propensity estimates

Abstract

BackgroundThe originality of divergent thinking (DT) production is one of the most critical indicators of creative potential. It is commonly scored using the statistical infrequency of responses relative to all responses provided in a given sample.AimsResponse frequency estimates vary in terms of measurement precision. This issue has been widely overlooked and is addressed in the current study.Sample and methodSecondary data analysis of 202 participants was performed. A total of 900 uniquely identified responses were generated on three DT tasks and subjected to a 1‐parameter logistic model with a response as the unit of measurement which allowed for the calculation of response‐level conditional reliability (and marginal reliability as an overall summary of measurement precision).ResultsMarginal reliability of response propensity estimates ranged from .62 to .67 across the DT tasks. Unique responses in the sample (the basis for the classic uniqueness scoring) displayed the lowest conditional reliability (across tasks: ≈ .50). Reliability increased nonlinearly as a function of both the frequency of occurrence predicted by the model (conditional reliability) and sample size (conditional and marginal reliability).ConclusionsThis study indicates that the common practice of frequency‐based originality scoring with typical sample sizes (e.g., N = 100 to N = 200) yields unacceptable levels of measurement precision (i.e., in particular for highly original responses). We further offer recommendations to mitigate the lack of measurement precision of frequency‐based originality scores for DT research.

Automating creativity assessment with SemDis: An open platform for computing semantic distance

“…As LSA has been increasingly employed in creativity research, researchers have begun to identify limitations of the approach and bestpractices in data processing. In a study on the AUT, for example, Forthmann et al (2019) found that LSA values are confounded by elaboration-the more words used to describe a response, the higher LSA-based cosine similarity (i.e., lower semantic distance derived from similarity)-but this confound was partially mitigated by removing "stop words" from responses (e.g., the, an, but) prior to computing LSA. Another consideration with semantic distance-based scoring concerns the balance of novelty and usefulness (or appropriateness), the two criteria that jointly define a creative idea or product (Diedrich, Benedek, Jauk, & Neubauer, 2015).…”

Automating Creativity Assessment with SemDis: An Open Platform for Computing Semantic Distance

“…Indeed, the question of how to best capture creativity remains open and active, with a recent special issue on creativity assessment recently published in Psychology of Aesthetics, Creativity, and the Arts (Barbot, Hass, & Reiter-Palmon, 2019). Over the years, a range of assessment approaches have been developed, from methods that rely on experts to judge the creative quality of products (i.e., the Consensual Assessment Technique; (Amabile, 1983;Cseh & Jeffries, 2019) to frequency-based methods that use standardized norms (Forthmann, Paek, Dumas, Barbot, & Holling, 2019;Torrance, 1972) to subjective scoring methods that rely on layperson judgements (Silvia et al, 2008). Although each method has shown some degree of utility for creativity research, each comes with challenges and limitations.…”

How Reliable and Valid are Frequency-Based Originality Scores? More Evidence from a Two-Step Approach

“…This cross-tabulation procedure requires decisions about which responses are treated as the same and which are not. For example, should drinking milk and drinking juice be treated as the same responses for uses of a cup (Forthmann, Paek et al, 2020;Reiter-Palmon et al, 2019)?…”

“…Such problems, however, can be mitigated by employing appropriate criteria for response adequacy (Reiter-Palmon et al, 2019). As a more critical issue, the sample size dependence of frequency estimates (Silvia et al, 2008;Simpson, 1922) and their measurement precision (Forthmann, Paek, et al, 2020;Reiter-Palmon et al, 2019) need to be taken into account. In particular, the measurement precision of frequency estimates has been studied using an item response theory approach that allows for quantifying the reliability of estimates in general and specifically the reliability for given levels of response propensity (Forthmann, Paek, et al, 2020).…”

“…It was found that frequency estimates might yield the required reliability levels when 300 to 400 participants are examined. However, often research relies on way smaller samples (De Dreu et al, 2014;Wallach & Kogan, 1965), which is particularly problematic for uniqueness scoring based on frequency-estimates with the lowest conditional reliability (Forthmann, Paek, et al, 2020).…”