Human-centric Spatio-Temporal Video Grounding With Visual Transformers

Tang, Zongheng; Liao, Yue; Liu, Si; Li, Guanbin; Jin, Xian-Min; Jiang, Hongxu; Yu, Qian; Xu, Dong

doi:10.48550/arxiv.2011.05049

Cited by 3 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Human-centric Spatio-Temporal Video Grounding (HC-STVG) [1] dataset provides 16k annotation-video pairs with different movie scenes. The duration of each video is 20 seconds.…”

Section: Datasetmentioning

confidence: 99%

See 1 more Smart Citation

2rd Place Solutions in the HC-STVG track of Person in Context Challenge 2021

Yi-Yu¹,

XinyingWang²,

WeiHu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this technical report, we present our solution to localize a spatio-temporal person in an untrimmed video based on a sentence. We achieve the second vIOU(0.30025) in the HC-STVG track of the 3rd Person in Context(PIC) Challenge. Our solution contains three parts: 1) human attributes information is extracted from the sentence, it is helpful to filter out tube proposals in the testing phase and supervise our classifier to learn appearance information in the training phase. 2) we detect humans with YoloV5 and track humans based on the DeepSort framework but replace the original ReID network with FastReID. 3) a visual transformer is used to extract cross-modal representations for localizing a spatio-temporal tube of the target person.

show abstract

“…The Human-centric Spatio-Temporal Video Grounding (HC-STVG) [1] dataset provides 16k annotation-video pairs with different movie scenes. The duration of each video is 20 seconds.…”

Section: Datasetmentioning

confidence: 99%

“…Tube trimming is to localize the temporal boundary of the target person since tube proposals may contain redundant transition frames. We follow [1] to apply tube trimming. During the experiment, we find that the performance with tube trimming is even worse.…”

Section: Tube Trimmingmentioning

confidence: 99%

2rd Place Solutions in the HC-STVG track of Person in Context Challenge 2021

Yi-Yu¹,

XinyingWang²,

WeiHu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding

Wang

et al. 2022

AAAI

View full text Add to dashboard Cite

Temporal grounding aims to localize a video moment which is semantically aligned with a given natural language query. Existing methods typically apply a detection or regression pipeline on the fused representation with the research focus on designing complicated prediction heads or fusion strategies. Instead, from a perspective on temporal grounding as a metric-learning problem, we present a Mutual Matching Network (MMN), to directly model the similarity between language queries and video moments in a joint embedding space. This new metric-learning framework enables fully exploiting negative samples from two new aspects: constructing negative cross-modal pairs in a mutual matching scheme and mining negative pairs across different videos. These new negative samples could enhance the joint representation learning of two modalities via cross-modal mutual matching to maximize their mutual information. Experiments show that our MMN achieves highly competitive performance compared with the state-of-the-art methods on four video grounding benchmarks. Based on MMN, we present a winner solution for the HC-STVG challenge of the 3rd PIC workshop. This suggests that metric learning is still a promising method for temporal grounding via capturing the essential cross-modal correlation in a joint embedding space. Code is available at https://github.com/MCG-NJU/MMN.

show abstract

Decoupled Spatial Temporal Graphs for Generic Visual Grounding

Feng¹,

Wei²,

Cheng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Visual grounding is a long-lasting problem in visionlanguage understanding due to its diversity and complexity. Current practices concentrate mostly on performing visual grounding in still images or well-trimmed video clips. This work, on the other hand, investigates into a more general setting, generic visual grounding, aiming to mine all the objects satisfying the given expression, which is more challenging yet practical in real-world scenarios. Importantly, grounding results are expected to accurately localize targets in both space and time. Whereas, it is tricky to make trade-offs between the appearance and motion features. In real scenarios, model tends to fail in distinguishing distractors with similar attributes. Motivated by these considerations, we propose a simple yet effective approach, named DSTG, which commits to 1) decomposing the spatial and temporal representations to collect all-sided cues for precise grounding; 2) enhancing the discriminativeness from distractors and the temporal consistency with a contrastive learning routing strategy. We further elaborate a new video dataset, GVG, that consists of challenging referring cases with far-ranging videos. Empirical experiments well demonstrate the superiority of DSTG over stateof-the-art on Charades-STA, ActivityNet-Caption and GVG datasets. Code and dataset will be made available.

show abstract

Human-centric Spatio-Temporal Video Grounding With Visual Transformers

Cited by 3 publications

References 44 publications

2rd Place Solutions in the HC-STVG track of Person in Context Challenge 2021

2rd Place Solutions in the HC-STVG track of Person in Context Challenge 2021

Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding

Decoupled Spatial Temporal Graphs for Generic Visual Grounding

Contact Info

Product

Resources

About