Beyond Bouma's window: How to explain global aspects of crowding?

Doerig, Adrien; Bornet, Alban; Rosenholtz, Ruth; Francis, Gregory; Clarke, Aaron; Herzog, Michael H.

doi:10.1371/journal.pcbi.1006580

Cited by 42 publications

(72 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies have established the involvement of such explicit grouping mechanisms during specific visual tasks. For example, different curve tracing paradigms require grouping of spatially separate contour segments [34], and recent findings by Doerig, Bornet, Rosenholtz, Francis, Clarke & Herzog [35], comparing a wide range of computational models, indicate that an explicit grouping step is crucial to explain different (un) crowding phenomena. Adding explicit segmentation mechanisms to DCNNs is promising to explain human behavior in tasks that require integrating and grouping of global features, or shape-level representations.…”

Section: Explicit Vs Implicit Models Of Grouping and Segmentationmentioning

confidence: 99%

Depth in convolutional neural networks solves scene segmentation

et al. 2020

View full text Add to dashboard Cite

Feed-forward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes. This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ('routines') that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image. Results indicate that with an increase in network depth, there is an increase in the distinction between objectand background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

show abstract

Section: Explicit Vs Implicit Models Of Grouping and Segmentationmentioning

confidence: 99%

Depth in convolutional neural networks solves scene segmentation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…We have shown previously that these global effects of crowding cannot be explained by models based 77 on the classic framework of vision, including ffCNNs (1,18,39). Here, we propose a new framework 78 to understand these global computations.…”

Section: Introduction 45mentioning

confidence: 99%

“…In crowding, objects that are easy to identify in isolation appear as jumbled and indistinct when clutter 66 is added (1,(24)(25)(26)(27)(28)(29). For example, a vernier target is presented, i.e., two vertical lines separated by a 67 horizontal offset (Figure 1a).…”

Section: Introduction 45mentioning

confidence: 99%

Capsule Networks as Recurrent Models ofGrouping and Segmentation

Doerig

Schmittwilken

Sayim

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

12Classically, visual processing is described as a cascade of local feedforward computations. Feedforward 13Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. Previously, 14 using visual crowding as a well-controlled challenge, we showed that no classic model of vision, 15including ffCNNs, can explain human global shape processing (1). Here, we show that Capsule Neural 16 Networks (CapsNets; 2), combining ffCNNs with a grouping and segmentation mechanism, solve this 17 challenge. We also show that ffCNNs and standard recurrent networks do not, suggesting that the 18 grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide 19 psychophysical evidence that grouping and segmentation is implemented recurrently in humans, and 20show that CapsNets reproduce these results well. We discuss why recurrence seems needed to 21 implement grouping and segmentation efficiently. Together, we provide mutually reinforcing 22 psychophysical and computational evidence that a recurrent grouping and segmentation process is 23 essential to understand the visual system and create better models that harness global shape 24 computations. 25 26Author Summary 27 Feedforward Convolutional Neural Networks (ffCNNs) have revolutionized computer vision and are 28 deeply transforming neuroscience. However, ffCNNs only roughly mimic human vision. There is a 29 rapidly expanding literature investigating differences between humans and ffCNNs. Several findings 30 suggest that, unlike humans, ffCNNs rely mostly on local visual features. Furthermore, ffCNNs lack 31 recurrent connections, which abound in the brain. Here, we use visual crowding, a well-known 32 psychophysical phenomenon, to investigate recurrent computations in global shape processing. 33Previously, we showed that no model based on the classic feedforward framework of vision, including 34 ffCNNs, can explain global effects in crowding. Here, we show that Capsule Networks (CapsNets), 35combining ffCNNs with recurrent grouping and segmentation, solve this challenge. Lateral and top-36 down recurrent connections do not, suggesting that grouping and segmentation are crucial for 37 human-like global computations. Based on these results, we hypothesize that one computational 38 function of recurrence is to efficiently implement grouping and segmentation. We provide 39 psychophysical evidence that, indeed, recurrent processes implement grouping and segmentation in 40 humans. CapsNets reproduce these results too. Together, we provide mutually reinforcing 41 computational and psychophysical evidence that a recurrent grouping and segmentation process is 42 essential to understand the visual system and create better models that harness global shape 43 computations. 44

show abstract

“…In crowding, objects that are easy to identify in isolation appear as jumbled and indistinct when clutter is added [9,[23][24][25][26][27][28]. Consider the following example: Fig 1A shows a vernier target, i.e., two vertical lines separated by a horizontal offset.…”

Section: Introductionmentioning

confidence: 99%

Capsule networks as recurrent models of grouping and segmentation

et al. 2020

Self Cite

View full text Add to dashboard Cite

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that Caps-Nets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

show abstract

Beyond Bouma's window: How to explain global aspects of crowding?

Cited by 42 publications

References 67 publications

Depth in convolutional neural networks solves scene segmentation

Depth in convolutional neural networks solves scene segmentation

Capsule Networks as Recurrent Models ofGrouping and Segmentation

Capsule networks as recurrent models of grouping and segmentation

Contact Info

Product

Resources

About