SPL-MLL: Selecting Predictable Landmarks for Multi-label Learning

Li, Junbing; Zhang, Changqing; Zhu, Pengfei; Chen, Lei; Hu, Qinghua

doi:10.1007/978-3-030-58545-7_45

Cited by 9 publications

(1 citation statement)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, [39][40][41] have explored point-wise supervision and learn from only a few background/foreground pixel annotations. Unlabeled images can also be used to improve performances by considering confident predictions as annotations of these images for training [42][43][44][45][46][47][48]. However, these works assume certain forms of weak annotations are available for all classes, thus cannot generalize to a wide range of novel classes that may have no annotations at all.…”

Section: Related Workmentioning

confidence: 99%

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Huynh¹,

Kuen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

show abstract