Deep Representation Learning on Long-Tailed Data: A Learnable Embedding Augmentation Perspective

Liu, Jialun; Sun, Yifan; Han, Chuchu; Dou, Zhaopeng; Li, Wenhui

doi:10.1109/cvpr42600.2020.00304

Cited by 165 publications

(84 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To avoid the misleading affect of some mislabeled texts, inspired by (Liu et al, 2020), we update {c k } k=K k=1 by employing the moving average with a learning rate γ:…”

Section: Implementations Of Label Angle Variancesmentioning

confidence: 99%

Semi-Supervised Text Classification with Balanced Deep Representation Distributions

Li¹,

Li²,

Ouyang³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Semi-Supervised Text Classification (SSTC) mainly works under the spirit of self-training. They initialize the deep classifier by training over labeled texts; and then alternatively predict unlabeled texts as their pseudo-labels and train the deep classifier over the mixture of labeled and pseudo-labeled texts. Naturally, their performance is largely affected by the accuracy of pseudo-labels for unlabeled texts. Unfortunately, they often suffer from low accuracy because of the margin bias problem caused by the large difference between representation distributions of labels in SSTC. To alleviate this problem, we apply the angular margin loss, and perform Gaussian linear transformation to achieve balanced label angle variances, i.e., the variance of label angles of texts within the same label. More accuracy of predicted pseudo-labels can be achieved by constraining all label angle variances balanced, where they are estimated over both labeled and pseudo-labeled texts during self-training loops. With this insight, we propose a novel SSTC method, namely Semi-Supervised Text Classification with Balanced Deep representation Distributions (S 2 TC-BDD). To evaluate S 2 TC-BDD, we compare it against the state-of-theart SSTC methods. Empirical results demonstrate the effectiveness of S 2 TC-BDD, especially when the labeled texts are scarce.

show abstract

“…To avoid the misleading affect of some mislabeled texts, inspired by (Liu et al, 2020), we update {c k } k=K k=1 by employing the moving average with a learning rate γ:…”

Section: Implementations Of Label Angle Variancesmentioning

confidence: 99%

Semi-Supervised Text Classification with Balanced Deep Representation Distributions

Li¹,

Li²,

Ouyang³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…DisAlign (Zhang et al 2021b) modifies the original logits by adding an extra learnable layer to the output layer. Apart from decoupled learning, Liu et al (2020) addresses the long-tailed problem by transferring head distributions to tail; distilling balanced student models from imbalanced trained teacher models (He, Wu, and Wei 2021;Zhang et al 2021a). Although all the above methods manifest an overall accuracy improvement, as we discussed in Section 1, they are indeed biased.…”

Section: Related Workmentioning

confidence: 99%

Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification

Zhu¹,

Niu²,

Hua³

et al. 2021

Preprint

View full text Add to dashboard Cite

We address the overlooked unbiasedness in existing longtailed classification methods: we find that their overall improvement is mostly attributed to the biased preference of "tail" over "head", as the test distribution is assumed to be balanced; however, when the test is as imbalanced as the long-tailed training data-let the test respect Zipf's law of nature-the "tail" bias is no longer beneficial overall because it hurts the "head" majorities. In this paper, we propose Cross-Domain Empirical Risk Minimization (xERM) for training an unbiased model to achieve strong performances on both test distributions, which empirically demonstrates that xERM fundamentally improves the classification by learning better feature representation rather than the "head vs. tail" game. Based on causality, we further theoretically explain why xERM achieves unbiasedness: the bias caused by the domain selection is removed by adjusting the empirical risks on the imbalanced domain and the balanced but unseen domain. Codes are available at https://github.com/BeierZhu/xERM.

show abstract

“…In recent years, it has made remarkable progress thanks to the development of deep learning [33]. To further improve person Re-ID performance, scholars tend to build network structures in a more complex way [3,13,32,37]. Generally, it will lead to an obvious growth of parameters and system latency.…”

Section: Introductionmentioning

confidence: 99%

Knowledge-Supervised Learning: Knowledge Consensus Constraints for Person Re-Identification

Wang

Fan

Guo

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

The consensus of multiple views on the same data will provide extra regularization, thereby improving accuracy. Based on this idea, we proposed a novel Knowledge-Supervised Learning (KSL) method for person re-identification (Re-ID), which can improve the performance without introducing extra inference cost. Firstly, we introduce isomorphic auxiliary training strategy to conduct basic multiple views that simultaneously train multiple classifier heads of the same network on the same training data. The consensus constraints aim to maximize the agreement among multiple views. To introduce this regular constraint, inspired by knowledge distillation that paired branches can be trained collaboratively through mutual imitation learning. Three novel constraints losses are proposed to distill the knowledge that needs to be transferred across different branches: similarity of predicted classification probability for cosine space constraints, distance of embedding features for euclidean space constraints, hard sample mutual mining for hard sample space constraints. From different perspectives, these losses complement each other. Experiments on four mainstream Re-ID datasets show that a standard model with KSL method trained from scratch outperforms its ImageNet pre-training results by a clear margin. With KSL method, a lightweight model without ImageNet pre-training outperforms most large models. We expect that these discoveries can attract some attention from the current de facto paradigm of "pre-training and fine-tuning" in Re-ID task to the knowledge discovery during model training. CCS CONCEPTS• Computing methodologies → Image representations; Object identification.

show abstract

Deep Representation Learning on Long-Tailed Data: A Learnable Embedding Augmentation Perspective

Cited by 165 publications

References 40 publications

Semi-Supervised Text Classification with Balanced Deep Representation Distributions

Semi-Supervised Text Classification with Balanced Deep Representation Distributions

Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification

Knowledge-Supervised Learning: Knowledge Consensus Constraints for Person Re-Identification

Contact Info

Product

Resources

About