Psychometric Evaluation of an Alternate Scoring for the Remote Associates Test

Abstract

The Remote Associates Test (RAT; Mednick, 1962; Mednick & Mednick, 1967) is a commonly employed test of creative convergent thinking. The RAT is scored with a dichotomous scoring, scoring correct answers as 1 and all other answers as 0. Based on recent research into the information processing underlying RAT performance, we argued that the dichotomous scoring may lead to a loss of potentially relevant information. Thus, we proposed an alternate scoring based on semantic similarity between the answer given by the participant and the correct solution using Latent Semantic Analysis (LSA; Landauer & Dumais, 1997). We evaluate the psychometric properties of the alternate LSA scoring and found evidence of construct validity for the LSA scoring which was comparable to findings for the standard scoring, but not better as we would have expected. Thus, our expectations that LSA-based scoring of the RAT counteracts potential information loss were not met. However, LSA based scorings appear to be a promising alternative for hardly solvable RAT items. We conducted additional analyses comparing different RAT item types with regard to their validity as well as evaluating the information uniquely contained in the LSA scoring. Implications of all finding for existing research using RAT items are discussed.

A flexible approach to modelling over‐, under‐ and equidispersed count data in IRT: The Two‐Parameter Conway–Maxwell–Poisson Model

“…These results can inform researchers' item selection. It is not uncommon to only use one type of task to measure divergent thinking ( e.g ., only AUT) in a study ( e.g ., Beisemann, Forthmann, Bürkner, & Holling, 2020). Analyses of different divergent thinking items with the 2PCMP model can indicate which items are best at discriminating between divergent thinking abilities.…”

On the Trail of Creativity: Dimensionality of Divergent Thinking and its Relation with Cognitive Abilities, Personality, and Insight

“…Insight tasks, such as items from the Remote Associates Test (Mednick, 1962(Mednick, , 1968, are commonly scored as dichotomous variables for correctness only, but further approaches that focus on scoring the originality of such answers have also been proposed (e.g. based on latent semantic analysis; Beisemann, Forthmann, Bürkner, & Holling, 2019) and should be considered in the future. In sum, our results indicate that insight is equally correlated with cognitive abilities and DT.…”

The Machines Take Over: A Comparison of Various Supervised Learning Approaches for Automated Scoring of Divergent Thinking Tasks

“…(2020) improved automated scoring by a weighting approach that relies on inverse document frequency. Notably, beyond the AUT, semantic distance scoring displayed validity evidence for the Torrance Test of Creative Thinking (Acar et al., 2021), creative verb association (Beaty & Johnson, 2021; Heinen & Johnson, 2018; Prabhakaran, Green, & Gray, 2014), the Consequences Test (LaVoie, Parker, Legree, Ardison, & Kilcullen, 2020), the Remote Associates Test (Beisemann, Forthmann, Bürkner, & Holling, 2020), or abstract figure naming (Sung, Cheng, Tseng, Chang, & Lin, 2022).…”