Few-shot Action Recognition with Prototype-centered Attentive Learning

Abstract

Few-shot action recognition aims to recognize action classes with few training samples. Most existing methods adopt a meta-learning approach with episodic training. In each episode, the few samples in a meta-training task are split into support and query sets. The former is used to build a classifier, which is then evaluated on the latter using a query-centered loss for model updating. There are however two major limitations: lack of data efficiency due to the query-centered only loss design and inability to deal with the support set outlying samples and inter-class distribution overlapping problems. In this paper, we overcome both limitations by proposing a new Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional querycentered learning objective, in order to make full use of the limited training samples in each episode. Second, PAL further integrates a hybrid attentive learning mechanism that can minimize the negative impacts of outliers and promote class separation. Extensive experiments on four standard few-shot action benchmarks show that our method clearly outperforms previous state-of-the-art methods, with the improvement particularly significant (> 10%) on the most challenging fine-grained action recognition benchmark.

A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark

“…Almost all existing few-shot video classification methods [1,2,3,19,21,30,31] use pretrained weights (ImageNet or Sports-1M). However, this pre-training step might transfer the semantic knowledge learned from the ImageNet or Sports-1M to downstream few-shot learning.…”

“…One promising direction is the meta-learning paradigm [5] where transferable knowledge is learned from a collection of tasks (or episodes) to prevent over-fitting and improve generalization. Inspired by metric learning methods [23,24], the existing few-shot video classification methods [1,2,3,19,21,31] usually compare the similarity of different videos in the feature space for classification. The essential difference between videos and images is the extra temporal dimension, which makes it insufficient to represent a whole video as a single feature vector.…”

Few-shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning

“…Most state-of-the-arts adopt the episodic meta-learning strategy, which trains the model with randomly sampled support and query sets in each episode. Spurred by the success of FSL in image classification, attempts are being made to explore FSL in action recognition [3,9,23,27,48,49,51,52]. CMN [51] utilizes memory networks to compress video information into a fixed matrix, which facilitates few-shot recognition by feature matching.…”

“…As shown in Figure 4(a), most state-of-the-art few-shot action recognition methods [9,23,52] utilize the prototypical network (Pro-toNet) [34] for meta-learning. In each training episode, representations of support samples are utilized to build a classifier.…”

“…With extensive efforts devoted to few-shot learning (FSL) recently, great success has been achieved in few-shot image classification [12,34,37,40,53]. Spurred by that, attempts are being made to extend FSL to action recognition domain [3,9,23,27,48,51,52]. Most methods follow the established practice of meta-learning paradigm [40], where models are trained with randomly sampled support and query sets episodically.…”