Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions

Woo, Sangmin; Noh, Junhyug; Kim, Kangil

doi:10.48550/arxiv.2106.08543

Cited by 1 publication

(1 citation statement)

References 50 publications

(126 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The challenging and open-ended nature of the task lends itself to various forms. It includes VRD [10,12] which is the generic form of visual relational reasoning tasks, Human-Object Interaction (HOI) [27,28,29] which mainly focuses on human and considers them as the subject, and Scene Graph Generation (SGG) [26,30,31,32,33,34] which is a structured form of VRD. Most of the works are mainly conducted in the static image domain.…”

Section: B Relation Detectionmentioning

confidence: 99%

What and When to Look?: Temporal Span Proposal Network for Video Visual Relation Detection

Woo¹,

Noh²,

Kim³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Identifying relations between objects is central to understanding the scene. While several works have been proposed for relation modeling in the image domain, there have been many constraints in the video domain due to challenging dynamics of spatio-temporal interactions (e.g., Between which objects are there an interaction? When do relations occur and end?). To date, two representative methods have been proposed to tackle Video Visual Relation Detection (VidVRD): segment-based and windowbased. We first point out the limitations these two methods have and propose Temporal Span Proposal Network (TSPN), a novel method with two advantages in terms of efficiency and effectiveness. 1) TSPN tells what to look: it sparsifies relation search space by scoring relationness (i.e., confidence score for the existence of a relation between pair of objects) of object pair.2) TSPN tells when to look: it leverages the full video context to simultaneously predict the temporal span and categories of the entire relations. TSPN demonstrates its effectiveness by achieving new state-of-the-art by a significant margin on two VidVRD benchmarks (ImageNet-VidVDR and VidOR) while also showing lower time complexity than existing methods -in particular, twice as efficient as a popular segment-based approach.

show abstract