The spatial arrangement method of measuring similarity can capture high-dimensional semantic structures

Abstract

Psychologists collect similarity data to study a variety of phenomena including categorization, generalization and discrimination, and representation itself. However, collecting similarity judgments between all pairs of items in a set is expensive, spurring development of techniques like the Spatial Arrangement Method (SpAM; Goldstone, Behavior Research Methods, Instruments, & Computers, 26, 381-386, 1994), wherein participants place items on a two-dimensional plane such that proximity reflects perceived similarity. While SpAM greatly hastens similarity measurement, and has been successfully used for lower-dimensional, perceptual stimuli, its suitability for higher-dimensional, conceptual stimuli is less understood. In study 1, we evaluated the ability of SpAM to capture the semantic structure of eight different categories composed of 20-30 words each. First, SpAM distances correlated strongly (r = .71) with pairwise similarity judgments, although below SpAM and pairwise judgment splithalf reliabilities (r's > .9). Second, a cross-validation exercise with multidimensional scaling fits at increasing latent dimensionalities suggested that aggregated SpAM data favored higher (> 2) dimensional solutions for seven of the eight categories explored here. Third, split-half reliability of SpAM dissimilarities was high (Pearson r = .90), while the average correlation between pairs of participants was low (r = .15), suggesting that when different participants focus on different pairs of stimulus dimensions, reliable high-dimensional aggregate similarity data is recoverable. In study 2, we show that SpAM can recover the Big Five factor space of personality trait adjectives, and that cross-validation favors a four-or five-dimension solution on this dataset. We conclude that SpAM is an accurate and reliable method of measuring similarity for high-dimensional items like words. We publicly release our data for researchers.

A Comparison of the Spatial Arrangement Method and the Total-Set Pairwise Rating Method for Obtaining Similarity Data in the Conceptual Domain

“…Although this is an encouraging finding for researchers who intend to use aggregate SpAM data, it is curious that average SpAM data are not representative of individual SpAM data. A related observation was made by Richie et al (2020). They found that although participants can only convey two dimensions in SpAM, aggregating the data and subjecting it to multidimensional scaling could nevertheless yield more than two dimensions, presumably because different participants convey different dimensions (see also Verheyen & Storms, 2020).…”

“…In Study 3, too, the average skewness of the individual multi-arrangement SpAM dissimilarity data was positive, and averaging across participants yielded a data set with negative skewness. One reason for this might be that when one averages across participants, one might be aggregating data from individuals who employed different considerations (Hout & Goldinger, 2016;Richie et al, 2020), while when one averages across an individual's trials in multi-arrangement SpAM, the same considerations are repeatedly used. The skewness of the individual PRaM dissimilarity data was once again found to be significantly lower than the skewness of the individual SpAM dissimilarity data.…”

“…Note that the isolated presentation of stimulus pairs is not an inherent characteristic of PRaM and several researchers have in practice accommodated this potential concern by providing participants with an overview of the stimuli that will be judged prior to the pairwise similarity judgment task (e.g., Richie et al, 2020;Verheyen & Storms, 2011), with a sample of the pairs that will be judged (Goldstone, 1994), or to have ratings remain visible so that participants can refer back to previous judgments (Hutchinson & Lockhead, 1977). Hout et al (2013) have recently proposed a variant on PRaM, which they termed Total-Set PRaM.…”

“…If these concerns were to prove valid, this would not bode well for SpAM, as data exploration and the testing of structural hypotheses are among the main applications of similarity data collection methods (Borg & Groenen, 2005). Although proponents of SpAM argue that the method offers an intuitive way of providing similarity data because we tend to conceptualize similarity in a spatial manner (Hout et al, 2013;Richie et al, 2020), Verheyen et al argue that this claim does not hold across all stimulus domains.…”

“…It is warranted that SpAM be evaluated on conceptual stimuli because it is unclear whether the method is equally appropriate for perceptual and conceptual stimuli (Hout et al, 2013;Verheyen et al, 2016). The richness of conceptual stimuli might be a problem for the twodimensional SpAM because participants might want to communicate more than two dimensions of variation (Richie et al, 2020). When participants make different choices as to which dimensions to communicate and/or employ idiosyncratic strategies for conveying additional information, this might be detrimental for the reliability of the data.…”