Problems of reliability and validity with similarity derived from category fluency

Abstract

This study aims to assess the reliability and the validity of exemplar similarity derived from category fluency tasks. A homogeneous sample of 21 healthy participants completed a category fluency task twice with an interval of one week. They also rated pairs comprised of the most frequently generated exemplars in terms of similarity. Similarities were derived from the fluency data by determining the average distance between generated exemplars and correcting it for repetitions and response sequence length. We calculated the correlation between the similarities derived from the two sessions of the fluency task and between the derived similarities and the directly rated similarities. Spatial representations of the similarities were constructed using multidimensional scaling to visualize the differences between both sessions of the fluency task and the pairwise rating task. We find that the derived similarities are not stable in time and show little correspondence with directly rated similarities. The differences between similarities derived from category fluency tasks in healthy participants, indicate that similar differences between healthy controls and patients with mental disorders, do not necessarily point to a semantic impairment of the latter, but rather reflect the unreliability of the data.

A Comparison of the Spatial Arrangement Method and the Total-Set Pairwise Rating Method for Obtaining Similarity Data in the Conceptual Domain

“…The steep expansion of the number of pairs to judge as the size of the stimulus set increases is considered the biggest drawback of PRaM. It makes the method ill-suited for large stimulus sets (Giordano et al, 2011;Kriegeskorte & Mur, 2012;Tsogo et al, 2000) and use in patient populations (White et al, 2014) where there is a genuine concern for detrimental effects of fatigue, inattention, boredom, and disengagement on data quality. Lengthy data collection protocols also increase the chance that participants will change their judgment strategy within a session (Hout et al, 2013).…”

“…Although the reliability of similarity data is not routinely assessed, it is not without consequences. Representations of unreliable average similarity data are not a good reflection of the shared structure among the participants (Ashby et al, 1994;Lee & Pope, 2003), are less likely to be reproduced (Sturidsson et al, 2006;Verheyen & Peterson, 2020;Voorspoels et al, 2014;White et al, 2014), are not necessarily representative for the individual similarity patterns (Bocci & Vichi, 2011;Okada & Lee, 2016), and limit the predictive ability of the data (White et al, 2014).…”

“…In the following section, we will compare the characteristics of the Pairwise Rating Method (PRaM) and the Spatial Arrangement Method (SpAM) for measuring similarity. The former is the predominant method for measuring similarity in the conceptual domain (e.g., De Deyne et al, 2016;Dry & Storms, 2009;Hill et al, 2015;Migo et al, 2013;White et al, 2014). According to Dry and Storms, 65% of similarity data sets in the semantic literature are obtained with this method.…”

Revealing Criterial Vagueness in Inconsistencies

“…The observation by McCloskey and Glucksberg (1978) that people provide inconsistent answers when asked to repeat a categorization task indicates that the information that is retrieved from semantic memory is not invariant. The probabilistic nature of the semantic retrieval process is corroborated by the modest reliability of repeated exemplar generation (Bellezza, 1984a; White, Voorspoels, Storms, & Verheyen, 2014), category definitions (Barsalou, 1989; Bellezza, 1984b), feature importance ratings (Hampton & Passanisi, 2016), and typicality judgments (Barsalou, 1987, 1989; Hampton & Passanisi, 2016). While these studies allow one to establish how much change to expect from one occasion to the next, they do not indicate what it is that changes over time.…”

Can we Use Conceptual Spaces to Model Moral Principles?

“…The high reliability measures indicate that the averaged similarity judgments are stable across groups. This is a requirement for the MDS configurations to be representative of a structure shared among participants (Ashby et al 1994;Lee and Pope 2003) and for the configurations to be replicable (Sturidsson et al 2006;Voorspoels et al 2014;White et al 2014). The stress tests and permutation tests conducted on the MDS configurations of the average similarity data indicate that there is structure underlying these configurations.…”