Video event detection using motion relativity and visual relatedness

Abstract

Event detection plays an essential role in video content analysis. However, the existing features are still weak in event detection because: i) most features just capture what is involved in an event or how the event evolves separately, and thus cannot completely describe the event; ii) to capture event evolution information, only motion distribution over the whole frame is used which proves to be noisy in unconstrained videos; iii) the estimated object motion is usually distorted by camera movement. To cope with these problems, in this paper, we propose a new motion feature, namely Expanded Relative Motion Histogram of Bag-ofVisual-Words (ERMH-BoW) to employ motion relativity and visual relatedness for event detection. In ERMH-BoW, by representing what aspect of an event with Bag-of-VisualWords (BoW), we construct relative motion histograms between visual words to depict the object activities or how aspect of the event. ERMH-BoW thus integrates both what and how aspects for a complete event description. Instead of motion distribution features, local motion of visual words is employed which is more discriminative in event detection. Meanwhile, we show that by employing relative motion, ERMH-BoW is able to honestly describe object activities in an event regardless of varying camera movement. Besides, to alleviate the visual word correlation problem in BoW, we propose a novel method to expand the relative motion histogram. The expansion is achieved by diffusing the relative motion among correlated visual words measured by visual relatedness. To validate the effectiveness of the proposed feature, ERMH-BoW is used to measure video clip similarity with Earth Mover's Distance (EMD) for event detection. We conduct experiments for detecting LSCOM events in TRECVID 2005 video corpus, and performance is improved by 74% and 24% compared with existing motion distribution feature and BoW feature respectively.

High-level event recognition in unconstrained videos

“…However, computation of trajectory descriptors requires substantial computational overhead. The first of its kind was proposed by Wang et al [149] where the authors used the wellknown Kanade-Lucas-Tomasi (KLT) tracker [79] to extract DoG-SIFT key-point trajectories, and compute a feature by modeling the motion between every trajectory pair. Sun et al [132] also applied KLT to track DoG-SIFT key-points.…”

“…Different from [149], they computed three levels of trajectory context, including point-level context which is an averaged SIFT descriptor, intra-trajectory context which models trajectory transitions over time, and inter-trajectory context which encodes proximities between trajectories. The velocity histories of key-point trajectories are modeled by Messing et al [87], who observed that velocity information is useful for detecting daily living actions in high-resolution videos.…”

Improving bag-of-visual-words model with spatial-temporal correlation for video retrieval

“…For example, Wang et al [24] proposed to incorporate a number of motion primitives of each visual word into the BovW representation of videos. Nevertheless, the above approaches usually directly include the spatial-temporal information into visual content representation, and the storage and computational cost is often high.…”

“…near identical regions being assigned to di↵erent visual words, soft-quantization of visual words [9] has been proposed to map each descriptor onto multiple neighboring visual words (in the descriptor feature space). Despite its simple structure, the BovW model has shown a promising performance in the fields such as object/event recognition [24] and image/video retrieval [20].…”

Words-of-interest selection based on temporal motion coherence for video retrieval

“…Inspired by the method proposed by Wang et al [5] which classifies the relative motion of visual words to represent the temporal patterns in a video, we propose to utilize relative motion to model the temporal relation of visual words.…”