Shuffled ImageNet Banks for Video Event Detection and Search

Abstract

This article aims for the detection and search of events in videos, where video examples are either scarce or even absent during training. To enable such event detection and search, ImageNet concept banks have shown to be effective. Rather than employing the standard concept bank of 1,000 ImageNet classes, we leverage the full 21,841-class dataset. We identify two problems with using the full dataset: (i) there is an imbalance between the number of examples per concept, and (ii) not all concepts are equally relevant for events. In this article, we propose to balance large-scale image hierarchies for pre-training. We shuffle concepts based on bottom-up and top-down operations to overcome the problems of example imbalance and concept relevance. Using this strategy, we arrive at the shuffled ImageNet bank, a concept bank with an order of magnitude more concepts compared to standard ImageNet banks. Compared to standard ImageNet pre-training, our shuffles result in more discriminative representations to train event models from the limited video event examples. For event search, the broad range of concepts enable a closer match between textual queries of events and concept detections in videos. Experimentally, we show the benefit of the proposed bank for event detection and event search, with state-of-the-art performance for both tasks on the challenging TRECVID Multimedia Event Detection and Ad-Hoc Video Search benchmarks.

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

“…The effectiveness of embedding-based methods such as the W2VV++ model used by last year's highest scoring system, SOMHunter [26], as also shown in an evaluation of SOMHunter and vitrivr [53], makes such models a valuable addition to retrieval systems. The W2VV++ model and its variants [31,34,40] was used by VIRET, SOMHunter, VBS2020 Winner, and in the form of features for image search CollageHunter. vitrivr and vitrivr-VR used a similar approach [68].…”

Is the Reign of Interactive Search Eternal? Findings from the Video Browser Showdown 2020

“…ing strategies. The SOMHunter and VIRET systems relied on the same BoW variant of the W2VV++ model [33,35], a query representation learning approach employing visual features obtained from deep networks trained with a high number of classes [43,44]. For more details about the employed W2VV++ variant and used similarity for each system, we refer to [35].…”

Object Priors for Classifying and Localizing Unseen Actions

“…Similar to the common word embedding setup, for a video v ∈ V, we seek to obtain a score for action a ∈ A using a set of global objects G. Global objects generally come from deep networks (Mettes et al 2020) pre-trained on large-scale object datasets (Deng et al 2009). We build upon current semantic matching approaches by providing three simple priors that deal with semantic ambiguity, non-discriminative objects, and object naming.…”

“…The pre-trained network includes the person class and 79 objects, such as car, chair, and tv. For the global object scores over whole videos, we apply a GoogLeNet (Szegedy et al 2015), pre-trained on 12,988 Ima-geNet categories (Mettes et al 2020). The object probability distributions are averaged over the sampled frames to obtain the global object scores.…”