Reexamining subjective creativity assessments in science tasks: An application of the rater-mediated assessment framework and many-facet Rasch model.

Abstract

Subjective creativity assessments that are originally developed as the Consensual Assessment Technique (CAT) rely on human raters to score creativity of the products. A few controversial issues in this approach are related to tasks, subjects, raters’ qualifications and performance, and methods for analyzing rating scores. This study addressed these issues under the theoretical framework of rater-mediated assessment and Rasch measurement theory. Data were collected from three groups of raters with different levels of expertise who assessed creativity of 24 6th graders in gifted and nongifted classes in two science tasks. Results indicated that (a) nongifted students showed higher creativity than gifted students in two science tasks, (b) three groups of raters had comparable scoring performances of creativity in two tasks, (c) only one rater exhibited differential rater functioning between nongifted and gifted students, but a few raters showed differential scoring between two science tasks. Implications of using rater-mediated assessment and Many-Facet Rasch measurement model to understand subjective creativity assessments are discussed, and future studies are suggested.

A Critical Review of Assessments of Creativity in Education

“…The application of different criteria seems to be a more serious issue than that of different scoring procedures or scales because consensus can be hardly reached over these criteria even among judges with similar backgrounds (Long, 2014b; Long & Pang, 2015). Therefore, given that variations in rating criteria and procedures are mainly related to raters, creativity researchers must pay more attention to issues such as rater effects, rater cognition, and how they affect the validity of the assessment (Kaufman et al, 2008; Long, 2014b; Long & Pang, 2015; Primi et al, 2019; Wang & Long, in press).…”

“…Primi et al (2019) used Many-Facet Rasch Modeling (MFRM) to model rater effects and investigate the effect of missingness on product-based creativity assessment. New methodological theories and methods have also been proposed, such as Myszkowski and Storme’s (2019) Judge Response Theory and Myszkowski’s (2019) jrt developed in R. Wang and Long (in press) recently examined the rater effects under the theoretical framework of rater-mediated assessment, which refers to “assessments in which raters evaluate test-taker performances and use rating scale categories to describe the level of performance on one or more domains” (Engelhard & Wind, 2019, p. 475).…”

“…More articles, however, deviated from these requirements and only adhered to the core characteristic of the approach, or the employment of raters’ subjective criteria of creativity. In this sense, it may be more appropriate to label this assessment approach as subjective creativity assessment than product-based assessment or CAT (Long, 2014a; Silvia et al, 2008; Wang & Long, in press). Further, when selecting judges or rating criteria, some articles selected novice as judges (e.g., Kaufman et al, 2013), while others trained graduate or undergraduate students to become judges (e.g., Kaufman et al, 2016).…”

What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?

“…In the field of creativity research, the process of selecting and training judges, enacting the coding scheme, and calculating the reliability of the ratings is most often seen as a means to an end: the main intention in the field is to study originality or creativity itself, and therefore, we aim for reliable codes in order to methodologically support our substantive work. However, a small but notable body of research has focused on the process of human rating eo ipso , empirically investigating and making recommendations on the types of tasks, participants, coders, and trainings that best support rater agreement and reliability (see Wang & Long, 2022 for a recent example). In this paper, we build on that existing literature to develop a model to inform the field's understanding of rater variance when coding the originality of children's responses to creativity assessments.…”

Effects of virtual reality on creativity performance and perceived immersion: A study of brain waves

“…In accordance with standard scoring rubrics, 30 ships were randomly selected and scored by two external senior teachers, who had more than fifteen years of teaching experience and participated in relative research more than five years. A consensual assessment technique was used in this research (Wang & Long, 2022; Weiss et al, 2021). The correlation coefficients for design concept variation, novelty, and feasibility were 0.969, 0.853, and 0.698, indicating a high level of reliability (Mugenda & Mugenda, 2003).…”