Improving Robustness to Texture Bias via Shape-focused Augmentation

Lee, Sang-Jun; Hwang, Inwoo; Kang, Gi-Cheon; Zhang, Byoung-Tak

doi:10.1109/cvprw56347.2022.00478

Cited by 8 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, changes in architecture (e.g., using an attention layer or the biologically inspired CORnet model) did not have a clear effect. Other methods to improve the shape bias include mixing in edge maps as training stimuli and to steer the stylization of training images (Mummadi et al, 2021), applying separate textures to the foreground object and the background (Lee et al, 2022), penalizing reliance on texture with adversarial learning (Nam et al, 2021), training on a mix of sharp and blurry images (Yoshihara et al, 2021), adding a custom drop-out layer that removes activations in homogeneous areas (Shi et al, 2020), or adding new network branches that receive preprocessed input like edge-maps (Mohla et al, 2022;Ye et al, 2022).…”

Section: Classification Of Cue Conflict Stimulimentioning

confidence: 99%

Shape-selective processing in deep networks: integrating the evidence on perceptual integration

Jarvers

Neumann

2023

Front. Comput. Sci.

View full text Add to dashboard Cite

Understanding how deep neural networks resemble or differ from human vision becomes increasingly important with their widespread use in Computer Vision and as models in Neuroscience. A key aspect of human vision is shape: we decompose the visual world into distinct objects, use cues to infer their 3D geometries, and can group several object parts into a coherent whole. Do deep networks use the shape of objects similarly when they classify images? Research on this question has yielded conflicting results, with some studies showing evidence for shape selectivity in deep networks, while others demonstrated clear deficiencies. We argue that these conflicts arise from differences in experimental methods: whether studies use custom images in which only some features are available, images in which different features compete, image pairs that vary along different feature dimensions, or large sets of images to assess how representations vary overall. Each method offers a different, partial view of shape processing. After comparing their advantages and pitfalls, we propose two hypotheses that can reconcile previous results. Firstly, deep networks are sensitive to local, but not global shape. Secondly, the higher layers of deep networks discard some of the shape information that the lower layers are sensitive to. We test these hypotheses by comparing network representations for natural images and silhouettes in which local or global shape is degraded. The results support both hypotheses, but for different networks. Purely feed-forward convolutional networks are unable to integrate shape globally. In contrast, networks with residual or recurrent connections show a weak selectivity for global shape. This motivates further research into recurrent architectures for perceptual integration.

show abstract

Section: Classification Of Cue Conflict Stimulimentioning

confidence: 99%

Shape-selective processing in deep networks: integrating the evidence on perceptual integration

Jarvers

Neumann

2023

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…By making texture-features less reliable across repetitions of the same image as well as across instances of the same object class, augmentations can be used to directly reduce the bias towards texture. Augmenting images to express object shape more vigorously has also been found to increase shape-bias (Lee et al, 2022; Gowda et al, 2022). Each of these approaches either manipulates the model architecture, implying that texture-bias is inherent to certain DNN architectures (like the commonly used ResNet50; He et al (2016)), or manipulates the input data by using augmentations that specifically target certain visual features.…”

Section: Related Workmentioning

confidence: 99%

Shape-Biased Learning by Thinking Inside the Box

Müller,

Snoek,

Groen

et al. 2024

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) may surpass human-level performance on vision tasks such as object recognition and detection, but their model behavior still differs from human behavior in important ways. One prominent example of this difference, and the main focus of our paper, is that DNNs trained on ImageNet exhibit an object texture bias, while humans are consistently biased towards shape. DNN shape-bias can be increased by data augmentation, but next to being computationally more expensive, data augmentation is a biologically implausible method of creating texture-invariance. We present an empirical study on texture-shape-bias in DNNs showcasing that high texture-bias correlates with high background-object ratio. In addition, DNNs trained on tight object bounding boxes of ImageNet images are sub-stantially more biased towards shape than models trained on the full images. Using a custom dataset of high-resolution, object annotated scene images, we show that (I) shape-bias systematically varies with training on bounding boxes, (II) removal of global object shape as a result of commonly applied cropping during training increases texture bias, (III) shape-bias is negatively correlated with test accuracy on ImageNet while being positively correlated on cue-conflict images created using bounding boxes, following the trend of humans. Overall, we show that an improved supervision signal that better reflects the visual features that truly belong to the to-be-classified object increases the shape-bias of deep neural networks. Our results also imply that simultaneous human alignment on both classification accuracy and strategy can not be achieved on default ImageNet images, suggesting the need for new assessments of both shape-bias and behavioural alignment between DNNs and humans.

show abstract

“…CL has shown great promise in self-supervised regimes [6,9,13] while recently, it has also been applied to the supervised learning domain and achieved promising results [19,22,23]. CL has been used in a self-supervised manner to help debias models [21,26,32,37]. In the fully supervised learning domain, previous works have shown that utilizing contrastive loss as an auxiliary loss can encourage learning more robust features with higher generalization abilities through careful contrastive pair construction [22,23].…”

Section: Related Workmentioning

confidence: 99%

CLAD: A Contrastive Learning based Approach for Background Debiasing

Wang¹,

Machiraju²,

Choung³

et al. 2022

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have achieved superhuman performance in multiple vision tasks, especially image classification. However, unlike humans, CNNs leverage spurious features, such as background information to make decisions. This tendency creates different problems in terms of robustness or weak generalization performance. Through our work, we introduce a contrastive learning-based approach (CLAD) to mitigate the background bias in CNNs. CLAD encourages semantic focus on object foregrounds and penalizes learning features from irrelavant backgrounds. Our method also introduces an efficient way of sampling negative samples. We achieve state-of-theart results on the Background Challenge dataset, outperforming the previous benchmark with a margin of 4.1%. Our paper shows how CLAD serves as a proof of concept for debiasing of spurious features, such as background and texture (in supplementary material).

show abstract

Improving Robustness to Texture Bias via Shape-focused Augmentation

Cited by 8 publications

References 7 publications

Shape-selective processing in deep networks: integrating the evidence on perceptual integration

Shape-selective processing in deep networks: integrating the evidence on perceptual integration

Shape-Biased Learning by Thinking Inside the Box

CLAD: A Contrastive Learning based Approach for Background Debiasing

Contact Info

Product

Resources

About