mixup: Beyond Empirical Risk Minimization

Zhang, Hongyi; Cissé, Moustapha; Dauphin, Yann N.; López-Paz, David

doi:10.48550/arxiv.1710.09412

Cited by 1,367 publications

(1,684 citation statements)

References 14 publications

Supporting

Mentioning

1,661

Contrasting

Unclassified

Order By: Relevance

“…Here, we first discuss the benefits of finetuning deep networks trained by contrastive learning over that of training randomly initialized networks, on boosting robustness against noisy labels. Then, we show that the initial robustness provided by contrastive learning enables robust training methods that are effective under small to moderate amount of noisy labels (Liu et al, 2020;Zhang et al, 2017;Mirzasoleiman et al, 2020) to achieve the state-ofthe-art performance under extreme noise levels.…”

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 87%

“…The initial level of robustness provided by contrastive learning can be leveraged by existing robust training methods to achieve a superior performance under extreme noise levels. Next, we briefly discuss three methods that prevent the pre-trained network from overfitting the noisy labels, through regularization (Liu et al, 2020;Zhang et al, 2017), or identifying clean examples (Mirzasoleiman et al, 2020).…”

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 99%

“…Mixup (Zhang et al, 2017) extends the training distribution by linear interpolations of feature vectors and their associated labels: x x x = λx x x i + (1 − λ)x x x j , ŷ y y = λy y y i + (1 − λ)y y y j , where λ ∼ Beta(α, α). In doing so, mixup makes linear transition in the decision boundary between classes and provide a smoother estimate of uncertainty.…”

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 99%

See 2 more Smart Citations

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

Xue¹,

Whitecross²,

Mirzasoleiman³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised contrastive learning has recently been shown to be very effective in preventing deep networks from overfitting noisy labels. Despite its empirical success, the theoretical understanding of the effect of contrastive learning on boosting robustness is very limited. In this work, we rigorously prove that the representation matrix learned by contrastive learning boosts robustness, by having: (i) one prominent singular value corresponding to every sub-class in the data, and remaining significantly smaller singular values; and (ii) a large alignment between the prominent singular vector and the clean labels of each subclass. The above properties allow a linear layer trained on the representations to quickly learn the clean labels, and prevent it from overfitting the noise for a large number of training iterations. We further show that the low-rank structure of the Jacobian of deep networks pre-trained with contrastive learning allows them to achieve a superior performance initially, when fine-tuned on noisy labels. Finally, we demonstrate that the initial robustness provided by contrastive learning enables robust training methods to achieve stateof-the-art performance under extreme noise levels, e.g., an average of 27.18% and 15.58% increase in accuracy on CIFAR-10 and CIFAR-100 with 80% symmetric noisy labels, and 4.11% increase in accuracy on WebVision. Noise Type Sym Asym Sym Asym Noise Ratio 20 50 80 40 20 50 80 40 F-correction 85.1 ± 0.4 76.0 ± 0.2 34.8 ± 4.5 83.6 ± 2.2 55.8 ± 0.5 43.3 ± 0.7 − 42.3 ± 0.7 Decoupling 86.7 ± 0.3 79.3 ± 0.6 36.9 ± 4.6 75.3 ± 0.8 57.6 ± 0.5 45.7 ± 0.4 − 43.1 ± 0.4 Co-teaching 89.1 ± 0.3 82.1 ± 0.6 16.2 ± 3.2 84.6 ± 2.8 64.0 ± 0.3 52.3 ± 0.4 − 47.7 ± 1.2 MentorNet 88.4 ± 0.5 77.1 ± 0.4 28.9 ± 2.3 77.3 ± 0.8 63.0 ± 0.4 46.4 ± 0.4 − 42.4 ± 0.5 D2L 86.1 ± 0.4 67.4 ± 3.6 10.0 ± 0.1 85.6 ± 1.2 12.5 ± 4.2 5.6 ± 5.4 − 14.1 ± 5.8 INCV 89.7 ± 0.2 84.8 ± 0.3 52.3 ± 3.5 86.0 ± 0.5 60.2 ± 0.2 53.1 ± 0.4 − 50.7 ± 0.2 T-Revision 79.3 ± 0.5 78.5 ± 0.6 36.2 ± 1.6 76.3 ± 0.8 52.4 ± 0.3 37.6 ± 0.3 − 32.3 ± 0.4 L DMI 84.3 ± 0.4 78.8 ± 0.5 20.9 ± 2.2 84.8 ± 0.7 56.8 ± 0.4 42.2 ± 0.5 − 39.5 ± 0.4 Crust * 85.3 ± 0.5 86.8 ± 0.3 33.8 ± 1.3 76.7 ± 3.4 62.9 ± 0.3 55.5 ± 1.1 18.5 ± 0.8 52.5 ± 0.4 Mixup 89.7 ± 0.7 84.5 ± 0.3 40.7 ± 1.1 86.3 ± 0.1 64.0 ± 0.4 53.4 ± 0.5 15.1 ± 0.1 54.4 ± 2.0 ELR * 90.6 ± 0.6 87.7 ± 1.0 69.5 ± 5.0 86.6 ± 2.9 63.6 ± 1.7 52.5 ± 4.2 23.4 ± 1.9 59.7 ± 0.1 CL+E2E * 88.8 ± 0.5 82.8 ± 0.2 72.0 ± 0.3 83.5 ± 0.5 63.5 ± 0.2 56.1 ± 0.3 36.7 ± 0.3 52.4 ± 0.2 CL+Crust * 86.5 ± 0.7 87.6 ± 0.3 77.9 ± 0.3 85.9 ± 0.4 63.0 ± 0.8 58.3 ± 0.1 34.8 ± 1.5 53.3 ± 0.7 CL+Mixup * 90.8 ± 0.2 84.6 ± 0.4 74.8 ± 0.3 87.5 ± 1.3 64.4 ± 0.4 55.5 ± 0.1 30.3 ± 0.4 55.5 ± 0.8 CL+ELR * 91.3 ± 0.0 89.1 ± 0.1 77.7 ± 0.2 89.7 ± 0.3 64.7 ± 0.2 55.6 ± 0.2 35.9 ± 0.3 63.6 ± 0.1

show abstract

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 87%

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 99%

Section: Fine-tuning the Network To Boost Performancementioning

confidence: 99%

See 1 more Smart Citation

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

Xue¹,

Whitecross²,

Mirzasoleiman³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We train different iterations for each task. We used random horizontal and vertical flips, 90 • rotation, and MixUp [109] with probability 0.5 for data augmentation. We used the Adam optimizer [38] with an initial learning rate of 2×10 −4 , which are steadily decreased to 10 −7 with the cosine annealing decay [58].…”

Section: Methodsmentioning

confidence: 99%

MAXIM: Multi-Axis MLP for Image Processing

Tu¹,

Talebi²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent progress on Transformers and multi-layer perceptron (MLP) models provide new network architectural designs for computer vision tasks. Although these models proved to be effective in many vision tasks such as image recognition, there remain challenges in adapting them for low-level vision. The inflexibility to support high-resolution images and limitations of local attention are perhaps the main bottlenecks for using Transformers and MLPs in image restoration. In this work we present a multi-axis MLP based architecture, called MAXIM, that can serve as an efficient and flexible general-purpose vision backbone for image processing tasks. MAXIM uses a UNet-shaped hierarchical structure and supports long-range interactions enabled by spatially-gated MLPs. Specifically, MAXIM contains two MLP-based building blocks: a multi-axis gated MLP that allows for efficient and scalable spatial mixing of local and global visual cues, and a cross-gating block, an alternative to cross-attention, which accounts for crossfeature mutual conditioning. Both these modules are exclusively based on MLPs, but also benefit from being both global and 'fully-convolutional', two properties that are desirable for image processing. Our extensive experimental results show that the proposed MAXIM model achieves state-of-the-art performance on more than ten benchmarks across a range of image processing tasks, including denoising, deblurring, deraining, dehazing, and enhancement while requiring fewer or comparable numbers of parameters and FLOPs than competitive models.

show abstract

“…[26] 0.1 Drop path [17] 0.1 0.1 0.15 0.3 Repeated augment [15] RandAugment [5] Mixup prob. [40] 0.8 Cutmix prob. [39] 1.0 Erasing prob.…”

Section: Methodsmentioning

confidence: 99%

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Han¹,

Guo²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

Transformer networks have achieved great progress for computer vision tasks. Transformer-in-Transformer (TNT) architecture utilizes inner transformer and outer transformer to extract both local and global representations. In this work, we present new TNT baselines by introducing two advanced designs: 1) pyramid architecture, and 2) convolutional stem. The new "PyramidTNT" significantly improves the original TNT by establishing hierarchical representations. PyramidTNT achieves better performances than the previous state-of-the-art vision transformers such as Swin Transformer. We hope this new baseline will be helpful to the further research and application of vision transformer. Code will be available at https: //github.com/huawei-noah/CV-Backbones/ tree/master/tnt_pytorch.

show abstract

mixup: Beyond Empirical Risk Minimization

Cited by 1,367 publications

References 14 publications

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

MAXIM: Multi-Axis MLP for Image Processing

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Contact Info

Product

Resources

About