Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores

Forthmann, Boris; Holling, Heinz; Zandi, Nima; Gerwig, Anne; Çelik, Pınar; Storme, Martin; Lubart, Todd

doi:10.1016/j.tsc.2016.12.005

Cited by 63 publications

(89 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although the scale facets used in the two rating systems are conceptually similar, the prior scoring yielded only holistic rater estimates of creativity, so it is difficult to unpack how different raters weighted or integrated the 3 dimensions of creativity proposed by Wilson et al (1953) when giving their ratings (cf. Forthmann et al, 2016b). However, if the ratings generated using the new scale correlate at least moderately with the scores derived from the well-established Silvia system, it would be evidence for convergent validity between the two systems, and would stand as preliminary positive evidence for the construct validity of the new system.…”

Section: Tailoring Rating Scales To Task-specific Performancementioning

confidence: 90%

“…The instructions were identical to the above, except that these raters were told to provide a rating based on the creativity of the majority of each participant's responses. This was done to discourage a correlation between snapshot scores and fluency, such that raters were not simply awarding high snapshot scores to participants who were highly fluent (Forthmann et al, 2016b). Each participant's responses were contained in a single cell in the spreadsheet, with individual responses separated by commas.…”

Section: Current Rating Proceduresmentioning

confidence: 99%

See 1 more Smart Citation

On the dependability and feasibility of layperson ratings of divergent thinking

Hass¹,

Silvia²

2018

Preprint

View full text Add to dashboard Cite

A new system for subjective rating of responses to divergent thinking tasks was tested using raters recruited from Amazon Mechanical Turk. The rationale for the study was to determine if such raters could provide reliable (aka generalizable) ratings from the perspective of generalizability theory. To promote reliability across the Alternative Uses and Consequence task prompts often used by researchers as measures of Divergent Thinking, two parallel scales were developed to facilitate feasibility and validity of ratings performed by laypeople. Generalizability and dependability studies were conducted separately for two scoring systems: the average-rating system and the snapshot system. Results showed that it is difficult to achieve adequate reliability using the snapshot system, while good reliability can be achieved on both task families using the average-rating system and a specific number of items and raters. Additionally, the construct validity of the average-rating system is generally good, with less validity for certain Consequences items. Recommendations for researchers wishing to adopt the new scales are discussed, along with broader issues of generalizability of subjective creativity ratings.

show abstract

Section: Tailoring Rating Scales To Task-specific Performancementioning

confidence: 90%

Section: Current Rating Proceduresmentioning

confidence: 99%

On the dependability and feasibility of layperson ratings of divergent thinking

Hass¹,

Silvia²

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Divergent-thinking tasks were rated for creative quality by using snapshot scoring (Forthmann et al, 2017;Silvia, Martin & Nusbaum, 2009). Two raters scored the sets of ideas on a Likert-scale from 1 (not creative at all) to 5 (very creative).…”

Section: Divergent Thinkingmentioning

confidence: 99%

Are Speedy Brains Needed when Divergent Thinking is Speeded—or Unspeeded?

Forthmann

Lips

Szardenings

et al. 2018

Journal of Creative Behavior

Self Cite

View full text Add to dashboard Cite

In this study, we focus on mental speed and divergent thinking, examining their relationship and the influence of task speededness. Participants (N = 109) completed a set of processing speed tasks and a test battery measuring divergent thinking. We used two speeded divergent‐thinking tasks of 2 minutes and two unspeeded tasks of 8 minutes to test the influence of task speededness on creative quality and their relation to mental speed. Before each task, participants were instructed to be creative in order to optimally measure creative quality. We found a large main effect of task speededness: less creative ideas were generated when tasks were speeded as compared to unspeeded (Cohen's d = −1.64). We could also replicate a positive relationship of mental speed with speeded divergent thinking (r = .21) and mental speed with unspeeded divergent thinking (r = .25). Our hypothesis that the relation is higher for the speeded divergent‐thinking tasks was not confirmed. Importantly, variation in creative quality scores under speeded conditions was not explained by mental speed beyond the predictive power of unspeeded creative quality. The latter finding implies that measurement of creative quality under speeded conditions is not confounded by mental speed.

show abstract

“…The data set to illustrate the above ideas was taken from Forthmann et al (2016). This dataset was gathered for a large project and used in previous publications on divergent thinking assessment issues (Forthmann et al, 2016;Forthmann, Holling, Zandi, et al, 2017) and the relationship between multicultural experiences and divergent thinking performance (Forthmann, Regehr, et al, 2018). However, the analyses in this study are unique to this work and go beyond any of the issues tackled in the above-mentioned articles.…”

Section: Datamentioning

confidence: 99%

“…Another explanation for the differentiation of creative quality between low-fluency and high-fluency ideational pools could be that raters judged the quality of pools with varying numbers of responses in a slightly different way. In fact, it has been demonstrated that the amount of information that needs to be judged has a detrimental effect on rater agreement when MULTIDIMENSIONALITY OF DIVERGENT THINKING 15 divergent thinking responses are rated for creative quality (Forthmann, Holling, Zandi, et al, 2017). Thus, it should not be overlooked that processing on the side of the raters might potentially influence the results here beyond the statistical control of rater severity effects as it was applied in the current study.…”

Section: Multidimensionality Of Divergent Thinking 13mentioning

confidence: 99%

A New Perspective on the Multidimensionality of Divergent Thinking Tasks

Forthmann¹,

Bürkner²,

Benedek³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

In the presented work, a shift of perspective with respect to the dimensionality of divergent thinking tasks is introduced moving from the question of multidimensionality across divergent thinking scores to the question of multidimensionality across the scale of divergent thinking scores. We apply IRTree models to test if the same latent trait can be assumed can be assumed across the whole scale in snapshot scoring of divergent thinking tests and holds for different task instructions and varying levels of fluency. This way, multidimensionality can be explored across scale points of a Likert-type rating scale and also multidimensionality due to differences in number of responses of ideational pools can be assessed. It was found that evidence for unidimensionality across scale points was stronger with be-creative instructions as compared to be-fluent instructions which suggests better psychometric quality of ratings when be-creative instructions are used. In addition, latent variables pertaining to low-fluency and high-fluency ideational pools shared around 50% of variance which suggests both strong overlap and evidence for differentiation. The presented approach allows to further examine the psychometric quality of subjective ratings and to examine new questions with respect to within-item multidimensionality in divergent thinking.

show abstract

Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores

Cited by 63 publications

References 57 publications

On the dependability and feasibility of layperson ratings of divergent thinking

On the dependability and feasibility of layperson ratings of divergent thinking

Are Speedy Brains Needed when Divergent Thinking is Speeded—or Unspeeded?

A New Perspective on the Multidimensionality of Divergent Thinking Tasks

Contact Info

Product

Resources

About