Understanding the Computational Demands Underlying Visual Reasoning

Vaishnav, Mohit; Cadène, Rémi; Alamia, Andrea; Linsley, Drew; VanRullen, Rufin; Serre, T.

doi:10.1162/neco_a_01485

Cited by 25 publications

(14 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, both models only achieved acceptable performance on the rectangles dataset and poor performance on the straight lines, connected squares and connected circles datasets in the same-different task. This marked difference in task generalization is consistent with previous results by Kim et al (2018) ; see also Vaishnav et al (2021) , who reported that visual reasoning tasks that involve same-different judgments are more difficult for CNNs than other spatial reasoning tasks. Furthermore, these results provide further support to the idea that DCNNs do not form abstract representations of the relations same and different when trained on the same-different task.…”

Section: Resultssupporting

confidence: 91%

Can deep convolutional neural networks support relational reasoning in the same-different task?

Puebla

Bowers

2022

Journal of Vision

View full text Add to dashboard Cite

Same-different visual reasoning is a basic skill central to abstract combinatorial thought. This fact has lead neural networks researchers to test same-different classification on deep convolutional neural networks (DCNNs), which has resulted in a controversy regarding whether this skill is within the capacity of these models. However, most tests of same-different classification rely on testing on images that come from the same pixel-level distribution as the training images, yielding the results inconclusive. In this study, we tested relational same-different reasoning in DCNNs. In a series of simulations we show that models based on the ResNet architecture are capable of visual same-different classification, but only when the test images are similar to the training images at the pixel level. In contrast, when there is a shift in the testing distribution that does not change the relation between the objects in the image, the performance of DCNNs decreases substantially. This finding is true even when the DCNNs’ training regime is expanded to include images taken from a wide range of different pixel-level distributions or when the model is trained on the testing distribution but on a different task in a multitask learning context. Furthermore, we show that the relation network, a deep learning architecture specifically designed to tackle visual relational reasoning problems, suffers the same kind of limitations. Overall, the results of this study suggest that learning same-different relations is beyond the scope of current DCNNs.

show abstract

Section: Resultssupporting

confidence: 91%

Can deep convolutional neural networks support relational reasoning in the same-different task?

Puebla

Bowers

2022

Journal of Vision

View full text Add to dashboard Cite

show abstract

“…Bowers et al argue that the inability of DNNs to learn human-like visual strategies reflects architectural limitations. They are correct that there is a rich literature demonstrating how mechanisms inspired by neuroscience can improve the capabilities of DNNs, helping them learn perceptual grouping (Kim, Linsley, Thakkar, & Serre, 2020;Linsley, Kim, Ashok, & Serre, 2019a;Linsley, Kim, Veerabadran, Windolf, & Serre, 2018, visual reasoning (Kim, Ricci, & Serre, 2018;Vaishnav et al, 2022;Vaishnav & Serre, 2023), robust object recognition (Dapello et al, 2020), and to more accurately predict neural activity Kubilius et al, 2018;Nayebi et al, 2018). The other fundamental difference between DNNs and biological organisms is how they learn; humans and DNNs learn from vastly different types of data with presumably different objective functions.…”

Section: The Next Generation Of Dnns For Biological Visionmentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modelling approaches that focus on psychological data.

show abstract

“…Note, however, that overall generalization performance was far from optimal. Previous research on the visual reasoning capabilities of DNNs has focused heavily on the same-different task (Adeli, Ahn and Zelinsky, 2023b;Baker, Garrigan, Phillips and Kellman, 2023;Funke, Borowski, Stosio, Brendel, Wallis and Bethge, 2021;Kim, Ricci and Serre, 2018;Falchi, 2021, 2022;Puebla and Bowers, 2022;Ricci et al, 2021;Stabinger, Peer and Rodríguez-Sánchez, 2021;Tartaglini, Feucht, Lepori, Vong, Lovering, Lake and Pavlick, 2023;Vaishnav, Cadene, Alamia, Linsley, VanRullen and Serre, 2022;Webb, Mondal and Cohen, 2023c;Webb, Sinha and Cohen, 2021). This is due to the fact that the concept of sameness is considered to be fundamental to human thought (Hochmann, Wasserman and Carey, 2021), develops early in human infants (Hespos, Gentner, Anderson and Shivaram, 2021), and it is more sophisticated in humans in comparison to other species (Gentner, Shao, Simms and Hespos, 2021).…”

Section: Introductionmentioning

confidence: 99%

Can deep convolutional neural networks support relational reasoning in the same-different task?

Puebla

Bowers

2021

Preprint

View full text Add to dashboard Cite

Same-different visual reasoning is a basic skill central to abstract combinatorial thought. This fact has lead neural networks researchers to test same-different classification on deep convolutional neural networks (DCNNs), which has resulted in a controversy regarding whether this skill is within the capacity of these models. However, most tests of same-different classification rely on testing on images that come from the same pixel-level distribution as the testing images, yielding the results inconclusive. In this study we tested relational same-different reasoning DCNNs. In a series of simulations we show that models based on the ResNet-50 architecture are capable of visual same-different classification, but only when the test images are similar to the training images at the pixel-level. In contrast, even when there are only subtle differences between the testing and training images, the performance of DCNNs drops substantially. This is true even when DCNNs' training regime is augmented with images from new versions of the same-different task or through multi-task learning on the test images. Furthermore, we show that the Relation Network, a deep learning architecture specifically designed to tackle visual relational reasoning problems, suffers the same kind of limitations than ResNet-50 classifiers.

show abstract

Understanding the Computational Demands Underlying Visual Reasoning

Cited by 25 publications

References 54 publications

Can deep convolutional neural networks support relational reasoning in the same-different task?

Can deep convolutional neural networks support relational reasoning in the same-different task?

Deep problems with neural network models of human vision

Can deep convolutional neural networks support relational reasoning in the same-different task?

Contact Info

Product

Resources

About