Recognition of Complex Events: Exploiting Temporal Dynamics between Underlying Concepts

Abstract

While approaches based on bags of features excel at lowlevel action classification, they are ill-suited for recognizing complex events in video, where concept-based temporal representations currently dominate. This paper proposes a novel representation that captures the temporal dynamics of windowed mid-level concept detectors in order to improve complex event recognition. We first express each video as an ordered vector time series, where each time step consists of the vector formed from the concatenated confidences of the pre-trained concept detectors. We hypothesize that the dynamics of time series for different instances from the same event class, as captured by simple linear dynamical system (LDS) models, are likely to be similar even if the instances differ in terms of low-level visual features. We propose a two-part representation composed of fusing: (1) a singular value decomposition of block Hankel matrices (SSID-S) and (2) a harmonic signature (H-S) computed from the corresponding eigen-dynamics matrix. The proposed method offers several benefits over alternate approaches: our approach is straightforward to implement, directly employs existing concept detectors and can be plugged into linear classification frameworks. Results on standard datasets such as NIST's TRECVID Multimedia Event Detection task demonstrate the improved accuracy of the proposed method.

What are the Limits to Time Series Based Recognition of Semantic Concepts?

“…The time series recognition methods investigated are summarized in Table 2. Whether they utilise Fisher Vector √ 16 [11], 15 [17] 85 [11], 60 [17] Dynamic System √ 25 [7], 5 [18] 93( [7], [18])…”

“…Liner Dynamic System (LDS): As a natural way of modeling temporal interaction within time series, Liner Dynamic Systems [7] can characterize temporal structure with attributes extracted from within a sliding window. The time series can be arranged in a block Hankel matrix H whose elements in a column have the length of sliding window (denoted as r) and successive columns are shifted with one time step.…”

“…[6] employed an intermediate representation of semantic model vectors trained from SVMs as a basis for detecting complex events, and revealed that this representation outperforms -and is complementary to -other low-level visual descriptors for event modeling. [7] showed that concept-based temporal representations are promising in more complex event recognition. Other efforts as presented in the TRECVid event detection tasks [8,9] also demonstrated promising results for concept-based event detection.…”

Computer Vision for Lifelogging

“…The time series can be arranged in a block Hankel matrix H whose elements in a column have the length of sliding window (denoted as r) and successive columns are shifted with one time step. According to [60], singular value decomposition of H · H T has achieved comparable performance to more complex representations. We constructed the feature using the k largest singular values along with their corresponding vectors.…”

“…Therefore, the Fisher kernel can be formalized as K(X i , X j ) = U T X i I F U T X j , where I F = E X (U X U T X ) denotes the Fisher information matrix. Liner Dynamic System (LDS): As a natural way of modeling temporal interaction within time series, Liner Dynamic Systems [60] can characterize temporal structure with attributes extracted from within a sliding window. The time series can be arranged in a block Hankel matrix H whose elements in a column have the length of sliding window (denoted as r) and successive columns are shifted with one time step.…”