A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research.

Abstract

Amabile's Consensual Assessment Technique (CAT)taking the consensus opinions of domain expertsis considered a 'gold standard' of creativity assessment for research purposes. While several studies have identified how specific procedural choices impact on the CAT's reliability as a measure, researchers' depth of knowledge about procedures and their effects still remains incomplete. This paper explores gaps in the research by reviewing CAT and creativity literature, and aims to explore to what extent the creativity research community needs to revisit and reflect on the CAT and solidify protocols for its implementation. The conclusion highlights the need for new debate and a program of research to clarify, evidence, and harmonize CAT methodology, while simultaneously preserving the CAT's flexibility. This would enable the development and sophistication of the CAT, including possible new assistive technologies, to further strengthen its use within the science of creativity.

Automating Creativity Assessment with SemDis: An Open Platform for Computing Semantic Distance

“…Indeed, the question of how to best capture creativity remains open and active, with a recent special issue on creativity assessment recently published in Psychology of Aesthetics, Creativity, and the Arts (Barbot, Hass, & Reiter-Palmon, 2019). Over the years, a range of assessment approaches have been developed, from methods that rely on experts to judge the creative quality of products (i.e., the Consensual Assessment Technique; (Amabile, 1983;Cseh & Jeffries, 2019) to frequency-based methods that use standardized norms (Forthmann, Paek, Dumas, Barbot, & Holling, 2019;Torrance, 1972) to subjective scoring methods that rely on layperson judgements (Silvia et al, 2008). Although each method has shown some degree of utility for creativity research, each comes with challenges and limitations.…”

“…Although subjective methods and other manual-based approaches have shown evidence of reliability and validity (Silvia et al, 2008), they suffer from two fundamental issues: subjectivity and labor cost. Regarding subjectivity, raters don't always agree on what constitutes a creative response, and they are often given little guidance-consistent with the widely adopted guidelines of the Consensual Assessment Technique (Cseh & Jeffries, 2019)-leading to low inter-rater reliability. Moreover, raters are often asked to code hundreds or thousands of responses, leading to rater fatigue and further threatening reliability (Forthmann, Holling, Zandi, et al, 2017).…”

Measuring Tactical Innovation in Terrorist Attacks

“…The first goal was to examine how dimensions of tactical innovation manifest in terrorist attacks. Using theory and best practices from fields specialized in the measurement of innovation (see Cseh & Jeffries, 2019); we found that three characteristics of innovative products emerged from the data. First, the novelty dimension of innovation was characterized by attacks using unique weapons or methods that require a high degree of coordination and expertise.…”

“…The consensual assessment technique assumes that creativity should be assessed by consensus between domain experts. Furthermore, when sufficient agreement is reached among experts, this defines the level of the product’s creativity relative to other products within the sample (see Cseh & Jeffries, 16 for review). For this study, the attack data were coded by eight graduate students in the fields of criminology, political science, and psychology.…”

Editorial: Novel Approaches for Studying Creativity in Problem-Solving and Artistic Performance

“…Kershaw et al apply a novel originality scoring method, the Decision Tree for Originality Assessment in Design (DTOAD), to creative ideation within engineering design. Clements et al adapt Amabile's Consensual Assessment Technique (CAT; Amabile, 1982; Cseh and Jeffries, 2019) for online use so as to have a broader reach, by which they investigate the effects of varying levels of dance expertise and experience on ratings of choreographic creativity. Loesche et al's exploration of the chronometry of insight moments and Threadgold et al's construction of a normative database of rebus puzzles both treat the strength of the Eureka experience as a continuum rather than a dichotomous all-or-none phenomenon, which has generally been a more common approach; similarly, some articles, including Hill and Kemp, and Loesche et al, consider phenomenological correlates of the insight moment as continua.…”