Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection

Abstract

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.

Activity Driven Weakly Supervised Object Detection

“…There are on average 6.8 action labels for a video. The official Charades dataset doesn't provide object bounding box annotations and we use the annotations released by [50]. In the released annotations, 1,812 test videos are down-sampled to 1 frame per second (fps) and 17 object classes are labeled with bounding boxes on these frames.…”

“…We report per-class average precision (AP) at intersection-over-union (IoU) of 0.5 between detection and ground truth boxes, and also mean AP (mAP) as a combined metric, following the tradition of [50]. We also report CorLoc [9], a commonly-used weakly supervised detection metric.…”

“…Yuan et al [50] proposed a much more efficient actiondriven weakly supervised object detection setting which aims to learn the object appearance representation given only videos with clip-level action class labels. They proposed to first extract spatial features from object proposals.…”

“…"cup" in the action "drink from cup"). Yuan et al [50] leveraged this prop-Spatial correlation between subject and object Object appearance consistency hold vacuum fix vacuum…”

“…We conducted comprehensive experiments over two video datasets: Charades [37], EPIC KITCHENS [7] and an image dataset: HICO-DET [5]. Our method outperforms the previous methods [3,50,41] by a large margin on all datasets. Specifically, we have achieved a 6% mAP boost on Charades compared to current state-of-the-art weakly supervised models for videos.…”