Long Short-Term Relation Networks for Video Action Detection

Abstract

It has been well recognized that modeling human-object or objectobject relations would be helpful for detection task. Nevertheless, the problem is not trivial especially when exploring the interactions between human actor, object and scene (collectively as humancontext) to boost video action detectors. The difficulty originates from the aspect that reliable relations in a video should depend on not only short-term human-context relation in the present clip but also the temporal dynamics distilled over a long-range span of the video. This motivates us to capture both short-term and long-term relations in a video. In this paper, we present a new Long Short-Term Relation Networks, dubbed as LSTR, that novelly aggregates and propagates relation to augment features for video action detection. Technically, Region Proposal Networks (RPN) is remoulded to first produce 3D bounding boxes, i.e., tubelets, in each video clip. LSTR then models short-term human-context interactions within each clip through spatio-temporal attention mechanism and reasons long-term temporal dynamics across video clips via Graph Convolutional Networks (GCN) in a cascaded manner. Extensive experiments are conducted on four benchmark datasets, and superior results are reported when comparing to state-of-the-art methods. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

Representing Videos as Discriminative Sub-graphs for Action Recognition

“…For example, [4,23,44] adopts GCN to build a reasoning module to model the relations between disjoint and distant regions. [21,48] takes dense object proposals as graph nodes and learns the relations between them. [22] treats each object proposal detected in the sample frames as a graph node and then searches adaptive network structures to model the object interactions.…”

Finding Achilles' Heel

“…Multi-modal action indeed has extended its popularity to many applications including recognition [47], generative multi-view action [40], detection [16], prediction [50], egocentric action [14], video identification [57], emotion with concept selection [48], personalized recommendation [44], human-object contour [53] and electromyography-vision [41]. The development of a low-cost depth sensor (Microsoft Kinect) opens up a new dimension of tackling the tasks of human action recognition.…”

“…Deep Neural Networks have shown superior and impressive performance on many tasks and even are the state-of-the-art methods in many real-world applications such as image classification [34], Poster Session D3: Multimedia Analysis and Description & Multimedia Fusion and Embedding MM '20, October 12-16, 2020, Seattle, WA, USA object detection [16,38], video action recognition [14,47,50,57], machine translation [3,25,35], and speech recognition [28]. However, these deep neural networks are notoriously well-known for their vulnerability [2,8].…”

Video Relation Detection via Multiple Hypothesis Association

“…detection [2,12,22] etc., it is still difficult for the machine to understand video content in a fine-grained and structured level. To tackle this issue, visual relation is one of the most important and useful information that can help describe the dynamic interactions between the objects in a video.…”