Semantic-Guided Class-Imbalance Learning Model for Zero-Shot Image Classification

Ji, Zhong; Yu, Xiaozhou; Yu, Yunlong; Pang, Yanwei; Zhang, Zhongfei

doi:10.1109/tcyb.2020.3004641

Cited by 20 publications

(9 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Re-weighting [29,30] on class-sensitive learning is a simple yet effective method to address class imbalance. SCILM [31] distributed different weights to selected samples based on class representativeness to alleviate the class-imbalance issue. Ye et al [32] detailed a theoretical analysis of data imbalance, proposing a balanced mixup loss function.…”

Section: Long-tail Learningmentioning

confidence: 99%

MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

Jia,

Ye,

Mei

et al. 2024

Preprint

View full text Add to dashboard Cite

Vision-language models, such as Contrastive Language-Image Pretraining (CLIP), have demonstrated powerful capabilities in image classification under zero-shot settings. However, current Zero-Shot Learning (ZSL) relies on manually tagged samples of known classes through supervised learning, resulting in a waste of labor costs and limitations on foreseeable classes in real-world applications. To address these challenges, we propose the Mixup Long-Tail Unsupervised (MLTU) approach for open-world ZSL problems. The proposed approach employed a novel long-tail mixup loss that integrated class-based re-weighting assignments with a given mixup factor for each mixed visual embedding. To mitigate the adverse impact over time, we adopted a noisy learning strategy to filter out samples that generated incorrect labels. We reproduced the unsupervised results of existing state-of-the-art long-tail and noisy learning approaches. Experimental results demonstrate that MLTU achieves significant improvements in classification compared to these proven existing approaches on public datasets. Moreover, it serves as a plug-and-play solution for amending previous assignments and enhancing unsupervised performance. MLTU enables the automatic classification and correction of incorrect predictions caused by the projection bias of CLIP.

show abstract

Section: Long-tail Learningmentioning

confidence: 99%

MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

Jia,

Ye,

Mei

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Ji et al [47] propose a simple balanced sampling approach on top of a prototypical model, then add a more complex "feature fusion" technique to account for the fact that certain instances may be more or less representative of their class prototype. This is done by aligning the semantic features of the class with the visual features of instances, creating semantic-guided prototypes for each class.…”

Section: Zero-shot Image Classificationmentioning

confidence: 99%

“…In experiments on the three datasets listed above plus two others, the proposed model outperformed many SOTA GZSL models for the balanced datasets, and marginally outperformed all for the imbalanced datasets. On AWA2 [82], they report 62.2% accuracy on unseen classes, 76.7% on seen classes, and 68.7% on the harmonic mean of these two, significantly outperforming [47]. Additionally, on the imbalanced data, they report better training times than all but one compared model, CNZSL [83].…”

Section: Zero-shot Image Classificationmentioning

confidence: 99%

Low-shot learning and class imbalance: a survey

Billion Polak,

Prusa,

Khoshgoftaar

2024

J Big Data

View full text Add to dashboard Cite

The tasks of few-shot, one-shot, and zero-shot learning—or collectively “low-shot learning” (LSL)—at first glance are quite similar to the long-standing task of class imbalanced learning; specifically, they aim to learn classes for which there is little labeled data available. Motivated by this similarity, we conduct a survey to review the recent literature for works which combine these fields in one of two ways, either addressing the obstacle of class imbalance within a LSL setting, or utilizing LSL techniques or frameworks in order to combat class imbalance within other settings. In our survey of over 60 papers in a wide range of applications from January 2020 to July 2023 (inclusive), we examine and report methodologies and experimental results, find that most works report performance at or above their respective state-of-the-art, and highlight current research gaps which hold potential for future work, especially those involving the use of LSL techniques in imbalanced tasks. To this end, we emphasize the lack of works utilizing LSL approaches based on large language models or semantic data, and works using LSL for big-data imbalanced tasks.

show abstract

“…Shigeto et al [ 31 ] experimentally proved that the semantic-to-visual embedding is able to generate more compact and separative visual feature distribution with the one-to-many correspondence manner, thereby mitigating the hubness issue. Ji et al [ 32 ] also follow the inverse mapping direction from semantic space to visual space and proposed a semantic-guided class imbalance learning model which alleviates the class-imbalance issue in ZSIC. In addition, for the class-imbalance issue, the generative models have been introduced to learn semantic-to-visual mapping to generate visual features of unseen classes [ 33 , 34 , 35 , 36 , 37 ] for data augmentation.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Image Classification Method Based on Attention Mechanism and Semantic Information Fusion

Wang

Feng

Song

et al. 2023

Sensors

View full text Add to dashboard Cite

The zero-shot image classification (ZSIC) is designed to solve the classification problem when the sample is very small, or the category is missing. A common method is to use attribute or word vectors as a priori category features (auxiliary information) and complete the domain transfer from training of seen classes to recognition of unseen classes by building a mapping between image features and a priori category features. However, feature extraction of the whole image lacks discrimination, and the amount of information of single attribute features or word vector features of categories is insufficient, which makes the matching degree between image features and prior class features not high and affects the accuracy of the ZSIC model. To this end, a spatial attention mechanism is designed, and an image feature extraction module based on this attention mechanism is constructed to screen critical features with discrimination. A semantic information fusion method based on matrix decomposition is proposed, which first decomposes the attribute features and then fuses them with the extracted word vector features of a dataset to achieve information expansion. Through the above two improvement measures, the classification accuracy of the ZSIC model for unseen images is improved. The experimental results on public datasets verify the effect and superiority of the proposed methods.

show abstract

Semantic-Guided Class-Imbalance Learning Model for Zero-Shot Image Classification

Cited by 20 publications

References 50 publications

MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

Low-shot learning and class imbalance: a survey

Zero-Shot Image Classification Method Based on Attention Mechanism and Semantic Information Fusion

Contact Info

Product

Resources

About