A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity?

Jeffries, Karl K.

doi:10.1080/21650349.2015.1084893

Cited by 25 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cronbach's Alpha for each rated category was calculated using Wessa.net interrater agreement online calculator (Table 1). The combined interrater agreement for all items (Combined) of 0.68 was satisfactory, just slightly below the threshold of 0.7 for acceptable interrater agreement in design (Karl K. Jeffries, 2017). This internal consistency was interpreted to demonstrate that there was a sufficient consensus around overall quality, and good consistency in some qualities, to warrant further qualitative investigation.…”

Section: Inter-rater Agreement Of the Rated Qualitiesmentioning

confidence: 86%

Engaging Qualities: factors affecting learner attention in online design studios

Lotz¹,

Jones²,

Holden³

2018

Proceedings of DRS

View full text Add to dashboard Cite

This study looks at the qualities of learner-generated online content, as rated by experts, and how these relate to learners' engagement through comments and conversations around this content. The work uploaded to an Online Design Studio by students across a Design and Innovation Qualification was rated and analysed quantitatively using the Consensual Assessment Technique (CAT). Correlations of qualities to comments made on this content were considered and a qualitative analysis of the comments was carried out. It was observed that design students do not necessarily pay attention to the same qualities in learner-generated content that experts rate highly, except for a particular quality at the first level of study. The content that students do engage with also changes with increasing levels of study. These findings have implications for the learning design of online design courses and qualifications as well as for design institutions seeking to supplement proximate design studios with Online Social Network Services.

show abstract

Section: Inter-rater Agreement Of the Rated Qualitiesmentioning

confidence: 86%

Engaging Qualities: factors affecting learner attention in online design studios

Lotz¹,

Jones²,

Holden³

2018

Proceedings of DRS

View full text Add to dashboard Cite

show abstract

“…Once the linkographs are created, they could be evaluated using an information theoretic approach (Kan and Gero, 2017). The drawbacks of linkography and other traditional methods relying on expert raters is that they suffer from known inter-rater reliability issues (Jeffries, 2017) and cannot be automated for routine use by non-experts. To objectively and quantitatively study transcribed protocols of design conversations, here we apply a semantic analysis approach based on dynamic semantic networks of nouns.…”

Section: Introductionmentioning

confidence: 99%

Semantic Analysis of Engineering Design Conversations

Georgiev

2020

Proc. Des. Soc.: Des. Conf.

View full text Add to dashboard Cite

To objectively and quantitatively study transcribed protocols of design conversations, we apply a semantic analysis approach based on dynamic semantic networks of nouns. We examined the applicability of the approach focused on a dynamic evaluation of the design problem solving process in engineering design educational settings. Using a case of real-world case, we show that the approach is able to determine the time dynamics of semantic factors such as level of abstraction, polysemy, information content, and quantify convergence/divergence in engineering design conversations.

show abstract

“…Although good reliability has been found with as few as three judges (e.g., Rostan, 2010; Beaty et al, 2013), groups of 15 judges (c.f. e.g., Kaufman et al, 2013; Jeffries, 2017) are not uncommon. Inter-rater reliability statistics are influenced by the number of raters (Gwet, 2014), so it is possible that the agreement levels reported in published research may be inflated.…”

Section: Introductionmentioning

confidence: 99%

“…Dollinger and Shafran, 2005; Kaufman et al, 2013; Storme et al, 2014), there are applications where it has been less useful. For example, Jeffries (2017) reports varying levels of inter-rater reliability depending on the target graphic design task that is used with the CAT. While simpler tasks, such as manipulating text to creatively express one word, had high levels of inter-rater reliability, more complex tasks, such as designing a t-shirt graphic, had unacceptable levels of inter-rater reliability.…”

Section: Introductionmentioning

confidence: 99%

A Decision Tree Based Methodology for Evaluating Creativity in Engineering Design

et al. 2019

View full text Add to dashboard Cite

Multiple metrics have been proposed to measure the creativity of products, yet there is still a need for effective, reliable methods to assess the originality of new product designs. In the present article we introduce a method to assess the originality of concepts that are produced during idea generation activities within engineering design. This originality scoring method uses a decision tree that is centered around distinguishing design innovations at the system level. We describe the history and the development of our originality scoring method, and provide evidence of its reliability and validity. A full protocol is provided, including training procedures for coders and multiple examples of coded concepts that received different originality scores. We summarize data from over 500 concepts for garbage collection systems that were scored by Kershaw et al. (2015). We then show how the originality scoring method can be applied to a different design problem. Our originality scoring method, the Decision Tree for Originality Assessment in Design (DTOAD), has been a useful tool to identify differences in originality between various cohorts of Mechanical Engineering students. The DTOAD reveals cross-sectional differences in creativity between beginning and advanced students, and shows longitudinal growth in creativity from the beginning to the end of the undergraduate career, thus showing how creativity can be influenced by the curriculum. The DTOAD can be applied to concepts produced using different ideation procedures, including concepts produced both with and without a baseline example product, and concepts produced when individuals are primed to think of different users for their designs. Finally, we show how our the DTOAD compares to other measurements of creativity, such as novelty, fixation, and remoteness of association.

show abstract

A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity?

Cited by 25 publications

References 15 publications

Engaging Qualities: factors affecting learner attention in online design studios

Engaging Qualities: factors affecting learner attention in online design studios

Semantic Analysis of Engineering Design Conversations

A Decision Tree Based Methodology for Evaluating Creativity in Engineering Design

Contact Info

Product

Resources

About