Exploiting multi-level parallelism for low-latency activity recognition in streaming video

Abstract

Video understanding is a computationally challenging task that is critical not only for traditionally throughput-oriented applications such as search but also latency-sensitive interactive applications such as surveillance, gaming, videoconferencing, and vision-based user interfaces. Enabling these types of video processing applications will require not only new algorithms and techniques, but new runtime systems that optimize latency as well as throughput. In this paper, we present a runtime system called Sprout that achieves low latency by exploiting the parallelism inherent in video understanding applications. We demonstrate the utility of our system on an activity recognition application that employs a robust new descriptor called MoSIFT, which explicitly augments appearance features with motion information. MoSIFT outperforms previous recognition techniques, but like other state-of-the-art techniques, it is computationally expensive -a sequential implementation runs 100 times slower than real time. We describe the implementation of the activity recognition application on Sprout, and show that it can accurately recognize activities at full frame rate (25 fps) and low latency on a challenging airport surveillance video corpus.

Tracklet Descriptors for Action Modeling and Video Analysis

“…Although many have used HoG/HoF descriptors [18,22,17,4], they aggregate them into a static signature, whereas our previous analysis and [36] suggest retaining their temporal evolution. However, rather than averaging by spatial binning (that presumes ergodicity), we prefer to use at least a crude approximation of the prior dP (g, w) in the form of samples {g(t j )}, {w(x, t j )} inferred during the training phase.…”

“…While one would want to assemble these elementary actions (dictionary elements) into a model that captures the joint spatio-temporal statistics at a more global spatial scale ("context"), in Sect. 4 we show that even a naive use of the dictionary labels as a "spatial bag" yields competitive performance in end-to-end tasks.…”

“…Different local descriptors have been proposed to capture shape [34,7] or joint motion and shape [18,17,4] by aggregating features within video cubes centered at spatio-temporal interest points into a static descriptor. In contrast, we retain in our tracklet descriptor the entire feature time series from birth to death of each tracked region.…”

“…Although "oG" in AoG stands for the gradient orientation, in analogy to HoG, any other contrast-normalizing statistic φ can be used, as in (4). Similarly, we have…”

Multimedia Analysis + Visual Analytics = Multimedia Analytics

“…For example, MoSIFT is a technique to better recognize activities in surveillance video by exploiting continuous object motion explicitly calculated from optical flow, integrated with distinctive appearance features [Chen 2010]. The value of these techniques for video retrieval are assessed through continued participation in international benchmarking forums, such as NIST TRECVID which chart progress on tasks related to video analytics.…”