Decoupled Spatial Temporal Graphs for Generic Visual Grounding

Abstract

Visual grounding is a long-lasting problem in visionlanguage understanding due to its diversity and complexity. Current practices concentrate mostly on performing visual grounding in still images or well-trimmed video clips. This work, on the other hand, investigates into a more general setting, generic visual grounding, aiming to mine all the objects satisfying the given expression, which is more challenging yet practical in real-world scenarios. Importantly, grounding results are expected to accurately localize targets in both space and time. Whereas, it is tricky to make trade-offs between the appearance and motion features. In real scenarios, model tends to fail in distinguishing distractors with similar attributes. Motivated by these considerations, we propose a simple yet effective approach, named DSTG, which commits to 1) decomposing the spatial and temporal representations to collect all-sided cues for precise grounding; 2) enhancing the discriminativeness from distractors and the temporal consistency with a contrastive learning routing strategy. We further elaborate a new video dataset, GVG, that consists of challenging referring cases with far-ranging videos. Empirical experiments well demonstrate the superiority of DSTG over stateof-the-art on Charades-STA, ActivityNet-Caption and GVG datasets. Code and dataset will be made available.

Video Referring Expression Comprehension via Transformer with Content-aware Query

“…The objective of video REC is to localize the spatial-temporal tube according to the natural language query. Most of the previous works [19,22,43,45,62] can be divided into two categories, i.e., two-stage methods and one-stage methods. However, both kinds of methods require time-consuming post-processing steps, which hinders their practical applications.…”

“…Current video REC methods can be classified into two major categories: two-stage, proposal-driven methods and one-stage, proposal-free methods. For the two-stage methods [19,20,26,62], they extract potential spatio-temporal tubes and then align these candidates to the sentence to find the best matching one. The other stream of one-stage methods [7,13,43,45,59] fuses visual-text features and directly predicts bounding boxes densely at all spatial locations.…”