Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts

Rahman, Shafin; Khan, Salman; Porikli, Fatih

doi:10.1007/978-3-030-20887-5_34

Cited by 104 publications

(141 citation statements)

References 51 publications

(115 reference statements)

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…The problem proposed here differs considerably from this in detecting a large set of objects in unconstrained settings and does not rely on using attributes. Comparison with recent works on ZSD: After completion of this work, we found two parallel works by Zhu et al [61] and Rahman et al [42] that target a similar problem. Zhu et al focus on a different problem of generating object proposals for unseen objects.…”

Section: Related Workmentioning

confidence: 87%

See 1 more Smart Citation

Zero-Shot Object Detection

Bansal

Sikka

Sharma

et al. 2018

Computer Vision – ECCV 2018

252

348

View full text Add to dashboard Cite

We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories as in prior works on zero-shot classification. We present a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets -MSCOCO and VisualGenome, and present extensive empirical results in both the traditional and generalized zero-shot settings to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.

show abstract

Section: Related Workmentioning

confidence: 87%

“…Zhu et al focus on a different problem of generating object proposals for unseen objects. Rahman et al [42] propose a loss formulation that combines max-margin learning and a semantic clustering loss. Their aim is to separate individual classes and reduce the noise in semantic vectors.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Object Detection

Bansal

Sikka

Sharma

et al. 2018

Computer Vision – ECCV 2018

252

348

View full text Add to dashboard Cite

show abstract

“…However, very few works have investigated the problem of few-shot object detection, where the task of recognizing instances of a category, represented by a few examples, is complicated by the presence of the image background and the need to accurately localize the objects. Recently, several interesting papers demonstrated preliminary results for the zero-shot object detection case [1,23] and for the few-shot transfer learning [5] scenario. In this work, we propose a novel approach for Distance Metric Learning (DML) and demonstrate its effectiveness on both few-shot object detection and object classification.…”

Section: Introductionmentioning

confidence: 99%

RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection

Karlinsky¹,

Shtok

Harary³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

301

214

View full text Add to dashboard Cite

Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples. In this work, we propose a new method for DML that simultaneously learns the backbone network parameters, the embedding space, and the multi-modal distribution of each of the training categories in that space, in a single end-to-end training process. Our approach outperforms state-of-theart methods for DML-based object classification on a variety of standard fine-grained datasets. Furthermore, we demonstrate the effectiveness of our approach on the problem of few-shot object detection, by incorporating the proposed DML architecture as a classification head into a standard object detection model. We achieve the best results on the ImageNet-LOC dataset compared to strong baselines, when only a few training examples are available. We also offer the community a new episodic benchmark based on the ImageNet dataset for the few-shot object detection task. Code will be released upon acceptance of the paper.

show abstract

“…Objects, events, actions and visual concepts appear with varying frequencies in real world imagery [38]. This often leads to highly skewed datasets where a few abundant classes outnumber several rare classes in a typical longtail data distribution.…”

Section: Introductionmentioning

confidence: 99%

Striking the Right Balance With Uncertainty

Khan

Hayat

Zamir

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

139

View full text Add to dashboard Cite

Figure 1: Imbalance learning with Bayesian uncertainty estimates. (a) We enforce classlevel margin penalty based on class uncertainty. This pushes boundaries further away from rare classes B and C. (b) We also consider sample-level uncertainty that is modeled as a Gaussian distribution. The learned margins consider the confidence level of classifier to re-adjust boundaries (i.e., provide more room to uncertain samples). This improves the generalization ability of the proposed model for imbalanced classes. AbstractLearning unbiased models on imbalanced datasets is a significant challenge. Rare classes tend to get a concentrated representation in the classification space which hampers the generalization of learned boundaries to new test examples. In this paper, we demonstrate that the Bayesian uncertainty estimates directly correlate with the rarity of classes and the difficulty level of individual samples. Subsequently, we present a novel framework for uncertainty based class imbalance learning that follows two key insights: First, classification boundaries should be extended further away from a more uncertain (rare) class to avoid over-fitting and enhance its generalization. Second, each sample should be modeled as a multi-variate Gaussian distribution with a mean vector and a covariance matrix defined by the sample's uncertainty. The learned boundaries should respect not only the individual samples but also their distribution in the feature space. Our proposed approach efficiently utilizes sample and class uncertainty information to learn robust features and more generalizable classifiers. We systematically study the class imbalance problem and derive a novel loss formulation for max-margin learning based on Bayesian uncertainty measure. The proposed method shows significant performance improvements on six benchmark datasets for face verification, attribute prediction, digit/object classification and skin lesion detection.

show abstract

Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts

Cited by 104 publications

References 51 publications

Zero-Shot Object Detection

Zero-Shot Object Detection

RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection

Striking the Right Balance With Uncertainty

Contact Info

Product

Resources

About