Interpretable object dimensions in deep neural networks and their similarities to human representations

Muttenthaler, Lukas; Hebart, Martin N.

doi:10.1167/jov.22.14.4516

Cited by 19 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We next sought to identify a set of perceptual dimensions-with values for every sample-that could account for the observed pattern of similarity responses. To do this, we analysed the responses using Variational Interpretable Concept Embeddings (VICE; Muttenthaler, Zheng, McClure, Vandermeulen, Hebart, & Pereira, 2022). This algorithm takes as input the sparse (i.e., incomplete) similarity matrix obtained in the similarity rating experiment and estimates the full pairwise similarity matrix.…”

Section: Discussionmentioning

confidence: 99%

“…To test this, we performed two experiments using movies of thirty samples of different wood veneers, rotating in such a way as to reveal both non-specular and specular appearance modes. In the first experiment, we took a data-driven approach, asking participants to make relative similarity judgments in a 2AFC task, from which we sought to derive underlying dimensions using the VICE algorithm (Muttenthaler, Zheng, McClure, Vandermeulen, Hebart, & Pereira, 2022). In the second experiment, we defined a set of ten appearance characteristics and asked participants to rate each sample in terms of all ten characteristics, effectively directly stating the location of each sample in a tendimensional appearance space.…”

Section: Discussionmentioning

confidence: 99%

“…We tested the VICE model on 76 different combinations of input parameters such as et, spike, slab, pi and distribution (gaussian, laplace) (cf. Muttenthaler, Zheng, McClure, Vandermeulen, Hebart, & Pereira, 2022). Results are shown in Fig.…”

Section: Supplementary Materials 1 Vice Algorithm Trainingmentioning

confidence: 95%

“…In order to do this, we first crowd-sourced 1218 perceptual similarity judgments from 56 participants. We then applied an analysis method based on sparse, non-negative matrix factorization (Variational Interpretable Concept Embeddings;Muttenthaler, Zheng, McClure, Vandermeulen, Hebart, & Pereira, 2022) to infer a set of dimensions that can predict the similarity judgments. We show that even with a small dataset of thirty samples the method was able to derive visual dimensions that predict the similarity judgments.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Perceptual Dimensions of Wood Materials

Filip¹,

Lukavský²,

Děchtěrenko³

et al. 2023

Preprint

View full text Add to dashboard Cite

Materials exhibit an extraordinary range of visual appearances. Characterising and quantifying appearance is important not only for basic research on perceptual mechanisms, but also for computer graphics and a wide range of industrial applications. While methods exist for capturing and representing the optical properties of materials and how they vary across surfaces (Haindl & Filip., 2013), the representations are typically very high-dimensional, and how these representations relate to subjective perceptual impressions of material appearance remains poorly understood. Here, we used a data-driven approach to characterising the perceived appearance characteristics of 30 samples of wood veneer using a ‘visual fingerprint’ that describes each sample as a multidimensional feature vector, with each dimension capturing a different aspect of the appearance. Fifty-six crowd-sourced participants viewed triplets of movies depicting different wood samples as the sample rotated. Their task was to report which of the two match samples was subjectively most similar to the test sample. In another online experiment 45 participants rated ten wood-related appearance characteristics for each of the samples. The results reveal a consistent embedding of the samples across both experiments and a set of 9 perceptual dimensions capturing aspects including the roughness, directionality and spatial scale of the surface patterns. We also showed that a weighted linear combination of eleven image statistics, inspired by the rating characteristics, predicts perceptual dimensions well.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Supplementary Materials 1 Vice Algorithm Trainingmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Perceptual Dimensions of Wood Materials

Filip¹,

Lukavský²,

Děchtěrenko³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We collected an additional 10% to compensate for assumed partial exclusion of the data. A subset of 1.46 million triplets had been used in previous work 30, 41, 89 . Data quality was assessed separately across 4 batches.…”

Section: Methodsmentioning

confidence: 99%

THINGS-data: A multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

Hebart

Contier

Teichmann

et al. 2022

Preprint

View full text Add to dashboard Cite

Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here we present THINGS-data, a multimodal collection of large-scale datasets comprising functional MRI, magnetoencephalographic recordings, and 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly-annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.

show abstract

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modelling approaches that focus on psychological data.

show abstract

Interpretable object dimensions in deep neural networks and their similarities to human representations

Cited by 19 publications

References 0 publications

Perceptual Dimensions of Wood Materials

Perceptual Dimensions of Wood Materials

THINGS-data: A multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

Deep problems with neural network models of human vision

Contact Info

Product

Resources

About