Out-Of-Distribution Detection for Generalized Zero-Shot Action Recognition

Abstract

Generalized zero-shot action recognition is a challenging problem, where the task is to recognize new action categories that are unavailable during the training stage, in addition to the seen action categories. Existing approaches suffer from the inherent bias of the learned classifier towards the seen action categories. As a consequence, unseen category samples are incorrectly classified as belonging to one of the seen action categories. In this paper, we set out to tackle this issue by arguing for a separate treatment of seen and unseen action categories in generalized zeroshot action recognition. We introduce an out-of-distribution detector that determines whether the video features belong to a seen or unseen action category. To train our out-ofdistribution detector, video features for unseen action categories are synthesized using generative adversarial networks trained on seen action category features. To the best of our knowledge, we are the first to propose an outof-distribution detector based GZSL framework for action recognition in videos. Experiments are performed on three action recognition datasets: Olympic Sports, HMDB51 and UCF101. For generalized zero-shot action recognition, our proposed approach outperforms the baseline [33] with absolute gains (in classification accuracy) of 7.0%, 3.4%, and 4.9%, respectively, on these datasets. * Authors contributed equally.

Generalized Zero-Shot Learning using Generated Proxy Unseen Samples and Entropy Separation

“…We use = 2048-dimension ResNet101 [8] features for images in AWA1 and CUB while manually defined attributes of dimension = 85 and = 312 for AWA1 and CUB, respectively. For HMDB51 and UCF101, we use the I3D [2] video feature of = 8196 dimension and = 300-dim word2vec as semantic prototypes provided by [16]. For AWA1 and CUB we use the new seen/unseen split proposed by [34] while for HMDB1 and UCF101 we use the seen/unseen splits provided by [16] where we experiment on 30 splits and report the average performance.…”

“…For HMDB51 and UCF101, we use the I3D [2] video feature of = 8196 dimension and = 300-dim word2vec as semantic prototypes provided by [16]. For AWA1 and CUB we use the new seen/unseen split proposed by [34] while for HMDB1 and UCF101 we use the seen/unseen splits provided by [16] where we experiment on 30 splits and report the average performance. We use the evaluation criterion proposed by [34] where we report the average top 1 accuracy on the seen ( ) and the unseen ( ) classes and the harmonic mean ( ) of and .…”

“…Amongst different GZSL approaches, non-generative models aim to learn deterministic or stochastic functions given the semantic and the visual spaces [9,10,15,21,23,24,29,31,[37][38][39]. On the other hand, generative techniques for GZSL focus to combat the classimbalance issue in GZSL by modeling the underlying data distributions [4,12,14,16,19,27,35,36]. Notably, [35] uses WGAN [6], which is trained to generate visual samples of the seen classes from the corresponding seen class-prototypes.…”

“…[27] proposes to match the gradients of generated and real samples to improve sample generation further. [16] extend the approach in [35] and train a separate classifier for seen and unseen classes. [14] proposes to alleviate confusion between seen and unseen class samples and introduce feature confusion scores.…”

Context-aware Feature Generation For Zero-shot Semantic Segmentation

“…Recently, a popular approach of zero-shot classification is generating synthesized features for unseen categories. For example, the method in [44] first generated features using word embeddings and random vectors, which was further improved by later works [7,22,28,40,45]. These zero-shot classification methods generated image features without involving contextual information.…”