Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning

Huang, Junkai; Fang, Chaowei; Chen, Weikai; Chai, Zhenhua; Wei, Xiaolin; Wei, Pengxu; Li, Lin; Li, Guanbin

doi:10.1109/iccv48922.2021.00820

Cited by 29 publications

(23 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further generalize SSL to unconstrained data without labels, there is a recent trend of developing more realistic open-set SSL methods [35], [45], [46], [47], [48], [49]. A common strategy of these works is to identify and suppress/discard OOD samples as they are considered to be less/not beneficial.…”

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

“…Curriculum learning has been used to detect and drop potentially detrimental data [46]. Besides, T2T [48] pretrains the feature model with all unlabeled data for improving OOD detection. More recently, OpenMatch [49] trains a set of one-vs-all classifiers for OOD detection and removal during SSL.…”

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

“…Whilst taking a step away from the artificial closed-set assumption, most existing open-set SSL works either focus on a simplified setting where both labeled and unlabeled sets are drawn from a single dataset, or consider only limited known classes and unlabeled data [48], [46] with all the known classes included in the unlabeled set [49], [48]. Clearly, both cases are fairly ideal and hardly valid in many practical cases.…”

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

“…Typically, there is no guarantee that the unlabeled data only contain the seen/target classes and follow the same distribution as the labeled training set, i.e., unconstrained unlabeled data with unknown distributions and classes. This is open-set semi-supervised learning, an emerging problem that has received an increasing amount of attention recently [35], [46], [49], [48]. Under this interesting context of "Knowledge distillation meets open-set semisupervised learning", we would investigate how knowledge distillation can benefit from unconstrained unlabeled data and previous open-set SSL algorithms, as well as how existing open-set SSL methods can influence the knowledge distillation process.…”

Section: Meeting Open-set Semi-supervised Learningmentioning

confidence: 99%

See 3 more Smart Citations

Knowledge Distillation Meets Open-Set Semi-Supervised Learning

Yang¹,

Zhu²,

Bulat³

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing knowledge distillation methods mostly focus on distillation of teacher's prediction and intermediate activation. However, the structured representation, which arguably is one of the most critical ingredients of deep models, is largely overlooked. In this work, we propose a novel Semantic Representational Distillation (SRD) method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. The key idea is that we leverage the teacher's classifier as a semantic critic for evaluating the representations of both teacher and student and distilling the semantic knowledge with high-order structured information over all feature dimensions. This is accomplished by introducing a notion of cross-network logit computed through passing student's representation into teacher's classifier. Further, considering the set of seen classes as a basis for the semantic space in a combinatorial perspective, we scale SRD to unseen classes for enabling effective exploitation of largely available, arbitrary unlabeled training data. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL). Extensive experiments show that our SRD outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks, as well as less studied yet practically crucial binary network distillation.Under more realistic open-set SSL settings we introduce, we reveal that knowledge distillation is generally more effective than existing Out-Of-Distribution (OOD) sample detection, and our proposed SRD is superior over both previous distillation and SSL competitors. The source code is available at https://github.com/jingyang2017/SRD ossl.

show abstract

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

Section: Open-set Semi-supervised Learningmentioning

confidence: 99%

Section: Meeting Open-set Semi-supervised Learningmentioning

confidence: 99%

See 2 more Smart Citations

Knowledge Distillation Meets Open-Set Semi-Supervised Learning

Yang¹,

Zhu²,

Bulat³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…• Huang et al [108] proposed a feature-label matching model to judge whether an example and a label from the set of target categories are matched. It can be used to identify open-set examples since they do not belong to any of target category.…”

Section: Label Distribution Mismatchmentioning

confidence: 99%

Robust Deep Semi-Supervised Learning: A Brief Introduction

Guo¹,

Zhou²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Semi-supervised learning (SSL) is the branch of machine learning that aims to improve learning performance by leveraging unlabeled data when labels are insufficient. Recently, SSL with deep models has proven to be successful on standard benchmark tasks. However, they are still vulnerable to various robustness threats in real-world applications as these benchmarks provide perfect unlabeled data, while in realistic scenarios, unlabeled data could be corrupted. Many researchers have pointed out that after exploiting corrupted unlabeled data, SSL suffers severe performance degradation problems. Thus, there is an urgent need to develop SSL algorithms that could work robustly with corrupted unlabeled data. To fully understand robust SSL, we conduct a survey study. We first clarify a formal definition of robust SSL from the perspective of machine learning. Then, we classify the robustness threats into three categories: i) distribution corruption, i.e., unlabeled data distribution is mismatched with labeled data; ii) feature corruption, i.e., the features of unlabeled examples are adversarially attacked; and iii) label corruption, i.e., the label distribution of unlabeled data is imbalanced. Under this unified taxonomy, we provide a thorough review and discussion of recent works that focus on these issues. Finally, we propose possible promising directions within robust SSL to provide insights for future research.

show abstract