Decoding Language Spatial Relations to 2D Spatial Arrangements

Radevski, Gorjan; Collell, Guillem; Moens, Marie‐Francine; Tuytelaars, Tinne

doi:10.18653/v1/2020.findings-emnlp.408

Cited by 8 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In HICO-DET the gain is less outspoken but note that because of their architecture CNN features already encode some spatial information. 11 A multi-layer perceptron provided similar results as a classifier. Late fusion was additionally considered, which fared slightly worse than early fusion with either type of classifier.…”

Section: Spatial Plus Visualmentioning

confidence: 85%

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

2021

Self Cite

View full text Add to dashboard Cite

Humans often leverage spatial clues to categorize scenes in a fraction of a second. This form of intelligence is very relevant in time-critical situations (e.g., when driving a car) and valuable to transfer to automated systems. This work investigates the predictive power of solely processing spatial clues for scene understanding in 2D images and compares such an approach with the predictive power of visual appearance. To this end, we design the laboratory task of predicting the identity of two objects (e.g., "man" and "horse") and their relationship or predicate (e.g., "riding") given exclusively the ground truth bounding box coordinates of both objects. We also measure the performance attainable in Human Object Interaction (HOI) detection, a real-world spatial task, which includes a setting where ground truth boxes are not available at test time. An additional goal is to identify the principles necessary to effectively represent a spatial template, that is, the visual region in which two objects involved in a relationship expressed by a predicate occur. We propose a scale-, mirror-, and translation-invariant representation that captures the spatial essence of the relationship, i.e., a canonical spatial representation. Tests in two benchmarks reveal:(1) High performance is attainable by using exclusively spatial information in all tasks. (2) In HOI detection, the canonical template outperforms the rest of spatial, visual, and several state-of-the-art baselines. (3) Simple fusion of visual and spatial features substantially improves performance. (4) Our methods fare remarkably well with a small amount of data and rare categories. Our results obtained on the Visual Genome (VG) and the Humans Interacting with Common Objects -Detection (HICO-DET) datasets indicate that great predictive power can be obtained from spatial clues alone, opening up possibilities for performing fast scene understanding at a glance.INDEX TERMS Spatial understanding, spatial layout, computer vision, vision and scene understanding. I. INTRODUCTION 1A Well-researched concept in cognitive science is the 2 gist, or the initial representation of a scene obtained 3 in a brief glance. The gist may include semantic content 4 (e.g., "is a classroom"), the identity of a few objects (e.g., 5"there are books"), and the spatial layout [1]. Humans can 6 categorize scenes in a fraction of a second (∼13-250 ms) 7[1], [2]. Generally, more detailed scenes and finer-grained 8

show abstract

Section: Spatial Plus Visualmentioning

confidence: 85%

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…When predicting the pose offset for an object, it only has information about the language instruction and previously predicted objects. This baseline is similar to the previous transformer models used in floor plan generation and clip-art generation work [51], [52], where the placements of objects are less spatially constrained. • No Structure: This variant of our pose generator directly predicts 6-DoF pose offset of each object in the world frame without predicting and using the virtual structure frame.…”

Section: A Model Component Testingmentioning

confidence: 91%

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Liu¹,

Paxton²,

Hermans³

et al. 2021

Preprint

View full text Add to dashboard Cite

Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Put objects in a line Set the table Put objects in a line Make a circle

show abstract

“…The applications of MHA in computer vision are rapidly expanding [22]. So far, MHA has been applied in conjunction with a CNN [6,15,31,42], as a stand-alone MHA over low level, raw image pixels [5,10,13,47], for vision + text tasks [28,43], or tasks involving spatial reasoning [36] to name a few. In contrast, we apply MHA: (i) over high level, abstract, spatio-temporal layouts, and (ii) to fuse the features of two distinct modalities (layout and appearance).…”

Section: Related Workmentioning

confidence: 99%

“…Some of the limitations include: (i) With a two-branch model, fusion is performed by concatenation, not fully exploiting the complementarity of the spatio-temporal layouts and the video appearance [4,30,49], and (ii) the layout module is treated as a peripheral component [20,49], so it remains unclear to what extent in different evaluation settings (compositional, few-shot, background cluttered videos), a well assembled layout-based model can recognize human actions. At the same time, a multi-head attention model [45] has been demonstrated to be a powerful common-sense reasoning tool over sets of spatially distributed objects in images for visual question-answering [28,43], layout generation [36], etc. By applying multiple heads of beyond-pairwise spatial reasoning, it encapsulates the scene's global spatial context, which is indicative of its semantics, to a certain extent.…”

Section: Introductionmentioning

confidence: 99%

Revisiting spatio-temporal layouts for compositional action recognition

Radevski¹,

Moens²,

Tuytelaars³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recognizing human actions is fundamentally a spatio-temporal reasoning problem, and should be, at least to some extent, invariant to the appearance of the human and the objects involved. Motivated by this hypothesis, in this work, we take an object-centric approach to action recognition. Multiple works have studied this setting before, yet it remains unclear (i) how well a carefully crafted, spatio-temporal layout-based method can recognize human actions, and (ii) how, and when, to fuse the information from layoutand appearance-based models. The main focus of this paper is compositional/few-shot action recognition, where we advocate the usage of multi-head attention (proven to be effective for spatial reasoning) over spatio-temporal layouts, i.e., configurations of object bounding boxes. We evaluate different schemes to inject video appearance information to the system, and benchmark our approach on background cluttered action recognition. On the Something-Else and Action Genome datasets, we demonstrate (i) how to extend multi-head attention for spatio-temporal layout-based action recognition, (ii) how to improve the performance of appearance-based models by fusion with layout-based models, (iii) that even on non-compositional background-cluttered video datasets, a fusion between layout-and appearance-based models improves the performance.

show abstract

Decoding Language Spatial Relations to 2D Spatial Arrangements

Cited by 8 publications

References 34 publications

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Revisiting spatio-temporal layouts for compositional action recognition

Contact Info

Product

Resources

About