Capsule Networks as Recurrent Models ofGrouping and Segmentation

Doerig, Adrien; Schmittwilken, Lynn; Sayim, Bilge; Manassi, Mauro; Herzog, Michael H.

doi:10.1101/747394

Cited by 4 publications

(5 citation statements)

References 50 publications

(2 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results contribute to the expanding literature showing that there is much more to vision than combining local feature detectors in a feedforward hierarchical manner (Baker et al, 2018;Brendel & Bethge, 2019;Doerig, Bornet, et al, 2019;Doerig, Schmittwilken, et al, 2019;Funke et al, 2018;Kar, Kubilius, Schmidt, Issa, & DiCarlo, 2019;Kietzmann et al, 2019;Kim, Linsley, Thakkar In line with the present findings, many studies have highlighted other fundamental differences between ffCNNs and humans in local vs. global processing. For example, Baker et al (2018) showed that ffCNNs but not humans are affected by local changes to edges and textures of objects.…”

Section: Discussionsupporting

confidence: 89%

“…For this principled reason, we propose that ffCNNs cannot produce uncrowding in general, independently of the specific ffCNN, training procedure and loss function. In support of this proposal, we showed in a separate contribution that ffCNNs specifically trained on classifying verniers and flanking shapes, as well as counting the number of flankers, do not produce global (un)crowding either (Doerig, Schmittwilken, Sayim, Manassi, & Herzog, 2019).…”

Section: Discussionsupporting

confidence: 64%

“…Hence, despite their well-known power, further aspects need to be incorporated into ffCNNs. We propose that recurrent, global grouping and segmentation is crucial to explain how the brain deals with global configurations (Doerig, Bornet, et al, 2019;Doerig, Schmittwilken, et al, 2019). Specifically, we propose that a flexible recurrent grouping process determines which elements are grouped into an object.…”

Section: Discussionmentioning

confidence: 99%

“…Deep learning models are promising because they are more flexible and can be trained to deal with any kind of stimulus. Doerig, Schmittwilken, et al (2019) showed that capsules networks (Sabour, Frosst, & Hinton, 2017), combining CNNs with a recurrent grouping and segmentation process, can explain (un)crowding, including temporal characteristics of uncrowding. Linsley, Kim, and Serre (2018) proposed recurrent grouping and segmentation modules to improve CNNs, and there are several other approaches to experiment with grouping and segmentation in recurrent network architectures (Lotter, Kreiman, & Cox, 2016;Nayebi et al, 2018;Spoerer, McClure, & Kriegeskorte, 2017).…”

Section: Discussionmentioning

confidence: 99%

See 3 more Smart Citations

Crowding reveals fundamental differences in local vs. global processing in humans and machines

Doerig¹,

Bornet²,

Choung³

et al. 2020

Vision Research

Self Cite

View full text Add to dashboard Cite

Feedforward Convolutional Neural Networks (ffCNNs) have become state-of-the-art models both in computer vision and neuroscience. However, human-like performance of ffCNNs does not necessarily imply human-like computations. Previous studies have suggested that current ffCNNs do not make use of global shape information. However, it is currently unclear whether this reflects fundamental differences between ffCNN and human processing or is merely an artefact of how ffCNNs are trained. Here, we use visual crowding as a well-controlled, specific probe to test global shape computations. Our results provide evidence that ffCNNs cannot produce human-like global shape computations for principled architectural reasons. We lay out approaches that may address shortcomings of ffCNNs to provide better models of the human visual system. texture. This training dataset biased an ffCNN (ResNet50; He, Zhang, Ren, & Sun, 2016) towards shape-level features, because textural information was no longer useful for classifying this dataset. They validated the network's shape-bias by showing increased robustness to local noise and textural changes. Alternatively, ffCNNs may be incapable of matching human global computations for principled architectural reasons. Even though Geirhos et al.'s network was able to ignore local features, it may not use global computations in the same way as humans. One difficulty in addressing this question is that there is no consensus about how to experimentally diagnose how deep networks compute global information.

show abstract

Section: Discussionsupporting

confidence: 89%

Section: Discussionsupporting

confidence: 64%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Crowding reveals fundamental differences in local vs. global processing in humans and machines

Doerig¹,

Bornet²,

Choung³

et al. 2020

Vision Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some related work on large artificial networks in linguistics (e.g., Vankov and Bowers, 2020;Jiang et al, 2021;Kim and Smolensky, 2021) suggests some strategies for combining extensive associative training with symbolic processing. In vision, capsule networks (Sabour et al, 2018) include some relational coding and have been shown to increase configural sensitivity in uncrowding effects (Doerig et al, 2020). Another recent model adds external memory to a recurrent DCNN to allow for explicit symbolic processing, resulting in rapid abstract rule learning (Webb et al, 2021).…”

Section: Discussionmentioning

confidence: 99%

Configural relations in humans and deep convolutional neural networks

Baker

Garrigan²,

Phillips³

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Deep convolutional neural networks (DCNNs) have attracted considerable interest as useful devices and as possible windows into understanding perception and cognition in biological systems. In earlier work, we showed that DCNNs differ dramatically from human perceivers in that they have no sensitivity to global object shape. Here, we investigated whether those findings are symptomatic of broader limitations of DCNNs regarding the use of relations. We tested learning and generalization of DCNNs (AlexNet and ResNet-50) for several relations involving objects. One involved classifying two shapes in an otherwise empty field as same or different. Another involved enclosure. Every display contained a closed figure among contour noise fragments and one dot; correct responding depended on whether the dot was inside or outside the figure. The third relation we tested involved a classification that depended on which of two polygons had more sides. One polygon always contained a dot, and correct classification of each display depended on whether the polygon with the dot had a greater number of sides. We used DCNNs that had been trained on the ImageNet database, and we used both restricted and unrestricted transfer learning (connection weights at all layers could change with training). For the same-different experiment, there was little restricted transfer learning (82.2%). Generalization tests showed near chance performance for new shapes. Results for enclosure were at chance for restricted transfer learning and somewhat better for unrestricted (74%). Generalization with two new kinds of shapes showed reduced but above-chance performance (≈66%). Follow-up studies indicated that the networks did not access the enclosure relation in their responses. For the relation of more or fewer sides of polygons, DCNNs showed successful learning with polygons having 3–5 sides under unrestricted transfer learning, but showed chance performance in generalization tests with polygons having 6–10 sides. Experiments with human observers showed learning from relatively few examples of all of the relations tested and complete generalization of relational learning to new stimuli. These results using several different relations suggest that DCNNs have crucial limitations that derive from their lack of computations involving abstraction and relational processing of the sort that are fundamental in human perception.

show abstract

Mixture-modeling approach reveals global and local processes in visual crowding

Jimenez

Kimchi

Yashar

2022

Sci Rep

View full text Add to dashboard Cite

Crowding refers to the inability to recognize objects in clutter, setting a fundamental limit on various perceptual tasks such as reading and facial recognition. While prevailing models suggest that crowding is a unitary phenomenon occurring at an early level of processing, recent studies have shown that crowding might also occur at higher levels of representation. Here we investigated whether local and global crowding interference co-occurs within the same display. To do so, we tested the distinctive contribution of local flanker features and global configurations of the flankers on the pattern of crowding errors. Observers (n = 27) estimated the orientation of a target when presented alone or surrounded by flankers. Flankers were grouped into a global configuration, forming an illusory rectangle when aligned or a rectangular configuration when misaligned. We analyzed the error distributions by fitting probabilistic mixture models. Results showed that participants often misreported the orientation of a flanker instead of that of the target. Interestingly, in some trials the orientation of the global configuration was misreported. These results suggest that crowding occurs simultaneously across multiple levels of visual processing and crucially depends on the spatial configuration of the stimulus. Our results pose a challenge to models of crowding with an early single pooling stage and might be better explained by models which incorporate the possibility of multilevel crowding and account for complex target-flanker interactions.

show abstract

Capsule Networks as Recurrent Models ofGrouping and Segmentation

Cited by 4 publications

References 50 publications

Crowding reveals fundamental differences in local vs. global processing in humans and machines

Crowding reveals fundamental differences in local vs. global processing in humans and machines

Configural relations in humans and deep convolutional neural networks

Mixture-modeling approach reveals global and local processes in visual crowding

Contact Info

Product

Resources

About