Deep Long-Tailed Learning: A Survey

Zhang, Yifan; Kang, Bingyi; Hooi, Bryan; Yan, Shuicheng; Feng, Jiashi

doi:10.48550/arxiv.2110.04596

Cited by 41 publications

(62 citation statements)

References 142 publications

(226 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Others utilize two-stage training [24,31] or separate expert networks [32,33,34]. We refer the readers to [8] for an extensive survey. [35] indicated SGD momentum can contribute to the aggravation of the long-tail problem and suggested de-confounded training to mitigate its effects.…”

Section: Related Workmentioning

confidence: 99%

“…Long-tailed learning is an area heavily studied in classification settings focusing on class imbalance. We refer readers to Table 2 in [7] and the survey paper by [8] for a complete review. Most common approaches to address the long-tail problem include post-hoc normalization, data resampling, loss engineering, and learning class-agnostic representations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Taming the Long Tail of Deep Probabilistic Forecasting

Kozerawski¹,

Sharan²,

Yu³

2022

Preprint

View full text Add to dashboard Cite

Deep probabilistic forecasting is gaining attention in numerous applications ranging from weather prognosis, through electricity consumption estimation, to autonomous vehicle trajectory prediction. However, existing approaches focus on improvements on the most common scenarios without addressing the performance on rare and difficult cases. In this work, we identify a long tail behavior in the performance of state-of-the-art deep learning methods on probabilistic forecasting. We present two moment-based tailedness measurement concepts to improve performance on the difficult tail examples: Pareto Loss and Kurtosis Loss. Kurtosis loss is a symmetric measurement as the fourth moment about the mean of the loss distribution. Pareto loss is asymmetric measuring right tailedness, modeling the loss using a generalized Pareto distribution (GPD). We demonstrate the performance of our approach on several real-world datasets including time series and spatiotemporal trajectories, achieving significant improvements on the tail examples.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Taming the Long Tail of Deep Probabilistic Forecasting

Kozerawski¹,

Sharan²,

Yu³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Imbalanced Learning and Long-tailed Recognition. Real-world data typically follows a long-tailed or imbalanced distribution, which biases the learning towards head classes, and degrades performance on tail classes [16,44]. Conventional methods have focused on designing class rebalancing paradigms through data re-sampling [1,2,5,33] or adjusting the loss weights for different classes during training [3,4,9,11,20,21].…”

Section: Related Workmentioning

confidence: 99%

Targeted Supervised Contrastive Learning for Long-Tailed Recognition

Li¹,

Cao²,

Yuan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Real-world data often exhibits long tail distributions with heavy class imbalance, where the majority classes can dominate the training process and alter the decision boundaries of the minority classes. Recently, researchers have investigated the potential of supervised contrastive learning for long-tailed recognition, and demonstrated that it provides a strong performance gain. In this paper, we show that while supervised contrastive learning can help improve performance, past baselines suffer from poor uniformity brought in by imbalanced data distribution. This poor uniformity manifests in samples from the minority class having poor separability in the feature space. To address this problem, we propose targeted supervised contrastive learning (TSC), which improves the uniformity of the feature distribution on the hypersphere. TSC first generates a set of targets uniformly distributed on a hypersphere. It then makes the features of different classes converge to these distinct and uniformly distributed targets during training. This forces all classes, including minority classes, to maintain a uniform distribution in the feature space, improves class boundaries, and provides better generalization even in the presence of long-tail data. Experiments on multiple datasets show that TSC achieves state-of-the-art performance on long-tailed recognition tasks.

show abstract

“…Recently, various methods have been proposed to deal with such biased training data. Readers can refer to [19], [20], [21], [22], [23], [24] for an overall review. In this paper, we focus on the sample re-weighting approach, which is a commonly used strategy against such data bias issue and has been widely investigated started at 1950s [25].…”

Section: Introductionmentioning

confidence: 99%

CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning

Shen¹,

Yuan²,

Meng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern deep neural networks (DNNs) can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. Most current methods, however, require to manually pre-specify the weighting schemes as well as their additional hyper-parameters relying on the characteristics of the investigated problem and training data. This makes them fairly hard to be generally applied in practical scenarios, due to their significant complexities and inter-class variations of data bias situations. To address this issue, we propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data. Specifically, by seeing each training class as a separate learning task, our method aims to extract an explicit weighting function with sample loss and task/class feature as input, and sample weight as output, expecting to impose adaptively varying weighting schemes to different sample classes based on their own intrinsic bias characteristics. Synthetic and real data experiments substantiate the capability of our method on achieving proper weighting schemes in various data bias cases, like the class imbalance, feature-independent and dependent label noise scenarios, and more complicated bias scenarios beyond conventional cases. Besides, the task-transferability of the learned weighting scheme is also substantiated, by readily deploying the weighting function learned on relatively smaller-scale CIFAR-10 dataset on much larger-scale full WebVision dataset. A performance gain can be readily achieved compared with previous state-of-the-art ones without additional hyper-parameter tuning and meta gradient descent step. The general availability of our method for multiple robust deep learning issues, including partial-label learning, semi-supervised learning and selective classification, has also been validated.

show abstract

Deep Long-Tailed Learning: A Survey

Cited by 41 publications

References 142 publications

Taming the Long Tail of Deep Probabilistic Forecasting

Taming the Long Tail of Deep Probabilistic Forecasting

Targeted Supervised Contrastive Learning for Long-Tailed Recognition

CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning

Contact Info

Product

Resources

About