CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation

Guo, Shi-Cheng; Liu, Shang-Kun; Wang, Jing-Yu; Zheng, Wei-Min; Jiang, Cheng-Yu

doi:10.3390/e25091353

Cited by 4 publications

(3 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Empirical evaluations conducted on standard datasets, PASCAL-5i and COCO-20i, showcase outstanding results. The proposed approach not only reduces training time but also maximizes performance, offering valuable insights for future research endeavors in the realm of multi-modal pretrained models for few-shot segmentation [10]. This paper presents CLIPSeg, an image segmentation approach that exhibits remarkable adaptability to new tasks through text or image prompts during inference.…”

Section: Literature Reviewmentioning

confidence: 99%

FashionMorph: Contextually Adaptive Clothing Replacement with CLIP, Segmentation, and Stable Diffusion

Goel,

Kadgi,

Gorka

et al. 2024

IJRASET

View full text Add to dashboard Cite

FashionMorph represents a groundbreaking advancement in digital image manipulation, seamlessly blending advanced segmentation techniques with the language comprehension capabilities of CLIP. This innovative approach simplifies the process of precise garment replacements while ensuring the preservation of realistic textures and boundaries. By incorporating stable diffusion, FashionMorph eliminates the need for extensive training on specific clothing styles, offering users a streamlined experience in expressing their preferences through natural language prompts. This empowers users in photo editing, granting them unparalleled creative freedom. The system comprises a comprehensive pipeline designed for the identification and inpainting of clothing elements. Leveraging CLIP and CLIPSeg models for robust image analysis, FashionMorph accurately identifies clothing elements and generates corresponding masks. The inpainting process utilizes a stable diffusion pipeline, resulting in impressive image quality that surpasses the performance of notable models like DCTransformer, StyleGAN, and ADM. Evaluations based on FID and SFID scores, along with Inception Score assessments, further validate FashionMorph's effectiveness in context-aware garment editing. Visual representations of original images, inpainted results, and masks vividly demonstrate the successful outcomes achieved by FashionMorph, establishing it as an efficient and user-friendly solution for professionals and enthusiasts alike. Among the evaluated models, Stable Diffusion emerges as the standout performer, boasting remarkable FID and SFID scores of 0.35 and 0.20, respectively, indicative of its outstanding image fidelity and diversity. Conversely, DCTransformer exhibits the highest FID and SFID scores, while StyleGAN excels in diversity but falls short in FID. ADM (dropout) strikes a balance between fidelity and diversity. These results underscore the exceptional performance of Stable Diffusion, solidifying its position as the leading approach with the lowest FID and SFID values

show abstract

Section: Literature Reviewmentioning

confidence: 99%

FashionMorph: Contextually Adaptive Clothing Replacement with CLIP, Segmentation, and Stable Diffusion

Goel,

Kadgi,

Gorka

et al. 2024

IJRASET

View full text Add to dashboard Cite

show abstract

“…Medical image segmentation plays a crucial role in a wide range of applications, as it involves dividing an entire image into a set of regions [22]. This process utilizes various image features, such as brightness, color, texture, shape, size, and location, to partition the image into multiple non-overlapping regions [23]. The resulting segmentation of these regions can provide clinicians with detailed and comprehensive image information, which can better support medical diagnosis and treatment decisions.…”

Section: Related Work a Medical Image Segmentationmentioning

confidence: 99%

A Tumor Segmentation Method Based on Mean-Teacher Reusing Pseudo-Labels

Jiang,

Liu,

Wang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Breast tumor is a common female physiological disease, and the malignant tumor is one of the main fatal diseases of women. Accurate examination and assessment of tumor shape can facilitate subsequent treatment and improve the cure rate. With the development of deep learning, automatic detection systems are designed to assist doctors in diagnosis. However, the blurry edges, poor visual quality, and irregular shapes of breast tumors pose significant challenges to design a highly efficient detection system. In addition, the lack of publicly available labeled data is a major obstacle in developing highly accurate and robust deep learning models for breast tumor detection. To overcome the aforementioned issues, we propose SRU-PMT+, a pseudo-label reusing Mean-Teacher architecture based on squeeze-and-excitation residual (SE-Res) attention. We utilize the proposed segmentation network, SRU-Net++, to generate pseudo-labels for unlabeled data, and guide the learning of the student model using the generated pseudo-labels and groundtruth, improving the accuracy and robustness of the model. Our proposed semi-supervised method has been rigorously evaluated on the available labeled dataset, i.e., Breast Ultrasound Images (BUSI) dataset. Results show that our proposed method outperforms current segmentation methods and has good performance. Importantly, our strategy of reusing pseudo-labels improves the performance of breast tumor segmentation.

show abstract

“…Therefore, how to reduce the use of resources and accurately segment the image has become an important research domain in the field of semantic segmentation. The few-shot semantic segmentation (FSS) [11] is used to solve the disadvantage of traditional methods in this paper.…”

Section: Introductionmentioning

confidence: 99%

PCNet: Leveraging Prototype Complementarity to Improve Prototype Affinity for Few-Shot Segmentation

Wang,

Liu,

Guo

et al. 2023

Electronics

Self Cite

View full text Add to dashboard Cite

With the advent of large-scale datasets, significant advancements have been made in image semantic segmentation. However, the annotation of these datasets necessitates substantial human and financial resources. Therefore, the focus of research has shifted towards few-shot semantic segmentation, which leverages a small number of labeled samples to effectively segment unknown categories. The current mainstream methods are to use the meta-learning framework to achieve model generalization, and the main challenges are as follows. (1) The trained model will be biased towards the seen class, so the model will misactivate the seen class when segmenting the unseen class, which makes it difficult to achieve the idealized class agnostic effect. (2) When the sample size is limited, there exists an intra-class gap between the provided support images and the query images, significantly impacting the model’s generalization capability. To solve the above two problems, we propose a network with prototype complementarity characteristics (PCNet). Specifically, we first generate a self-support query prototype based on the query image. Through the self-distillation, the query prototype and the support prototype perform feature complementary learning, which effectively reduces the influence of the intra-class gap on the model generalization. A standard semantic segmentation model is introduced to segment the seen classes during the training process to achieve accurate irrelevant class shielding. After that, we use the rough prediction map to extract its background prototype and shield the background in the query image by the background prototype. In this way, we obtain more accurate fine-grained segmentation results. The proposed method exhibits superiority in extensive experiments conducted on the PASCAL-5i and COCO-20i datasets. We achieve new state-of-the-art results in the few-shot semantic segmentation task, with an mIoU of 71.27% and 51.71% in the 5-shot setting, respectively. Comprehensive ablation experiments and visualization studies show that the proposed method has a significant effect on small-sample semantic segmentation.

show abstract

CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation

Cited by 4 publications

References 64 publications

FashionMorph: Contextually Adaptive Clothing Replacement with CLIP, Segmentation, and Stable Diffusion

FashionMorph: Contextually Adaptive Clothing Replacement with CLIP, Segmentation, and Stable Diffusion

A Tumor Segmentation Method Based on Mean-Teacher Reusing Pseudo-Labels

PCNet: Leveraging Prototype Complementarity to Improve Prototype Affinity for Few-Shot Segmentation

Contact Info

Product

Resources

About