Querying for video events by semantic signatures from few examples

Abstract

We aim to query web video for complex events using only a handful of video query examples, where the standard approach learns a ranker from hundreds of examples. We consider a semantic signature representation, consisting of off-the-shelf concept detectors, to capture the variance in semantic appearance of events. Since it is unknown what similarity metric and query fusion to use in such an event retrieval setting, we perform three experiments on unconstrained web videos from the TRECVID event detection task. It reveals that: retrieval with semantic signatures using normalized correlation as similarity metric outperforms a low-level bag-of-words alternative, multiple queries are best combined using late fusion with an average operator, and event retrieval is preferred over event classification when less than eight positive video examples are available.

Minimally Needed Evidence for Complex Event Recognition in Unconstrained Videos

“…2, we report our performance on the MED TEST dataset consisting of 20 events. We also compare our retrieval performance against a state of the art technique published in [17]. For clarity, we only indicate the respective average precision per event obtained using our best performing method (MNE+QR on fused motion and appearance representations).…”

“…That said, some of the noted work in recent past, pertinent to recognition in unconstrained settings include machine interpretation of either low-level features [13,26] directly extracted from human labeled event videos [18,20] or training intermediate-level semantic concepts that require expensive human annotation [1,17] or a combination of both [9,21].…”

“…Recognition of complex events in consumer videos such as "parade" or "changing a tire" is propelling a new breed of research [2,3,11,12,17,18,23,28,31] in multimedia and computer vision. This is an extremely challenging problem [2,3,8] as it involves high level machine understanding of videos that are semantically diverse, and prone to frequent illumination changes, large background clutter, and significant camera motion.…”

“…We make the following technical contributions in this paper: (a) We propose a novel game based interface to study human cognition of video events (Section 3.1), which can also be used in a variety of tasks in complex event recognition that involve human supervision [3,17,23,28,31], (b) Based on our extensive study, we demonstrate that humans can recognize complex events by seeing one or two chunks of spatially downsampled shots, less than a couple seconds in length (Section 3.2), (c) We leverage on positive and negative visual cues selected by humans for efficient retrieval (Section 4), and finally, (d) We perform conclusive experiments to demonstrate significant improvement in event retrieval performance on two challenging datasets released under TRECVID (Section 5) using off-the-shelf feature extraction techniques applied on MNEs versus evidence collected sequentially.…”

Knowledge based query expansion in complex multimedia event detection

“…This is an effective way, but it is not applicable for cases in which no or few examples are available and the models cannot give interpretation or understanding of the semantics in the event. If few examples are available, the web is a powerful tool to get more examples [24,27].…”