Spatial Commonsense Graph for Object Localisation in Partial Scenes

Giuliari, Francesco; Skenderi, Geri; Cristani, Marco; Wang, Yiming; Bue, Alessio Del

doi:10.1109/cvpr52688.2022.01891

Cited by 12 publications

(22 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These predictions also highlight the difficulty of the task, as many predictions are plausible (such as a drawer keeping its state or the second chairs also changing) but are not realized. architectures, the GNN of 3DSSG [7], where we replace the PointNet input features with the SG embeddings of [7] to operate directly on a SG, and the recent architecture of Spatial Commonsense Graphs (SCG) [19] based on graph transformers [50]. We find that GNNs perform significantly better than equivalent fully connected architectures, as the MLP is not able to identify local context and barely predicts any state or instance changes.…”

Section: B Variable Scene Graph Prediction Performancementioning

confidence: 96%

“…Rosinol et al [8] introduce dynamic SGs, providing rich hierarchical abstractions of the SG and accounting for short-term moving agents, such as humans. Recently, Giuliari et al [19] presented Spatial Commonsense Graphs (SCG), embedding additional nodes from knowledge graphs such as "used for reading", which they call commonsense concepts. Different approaches have been proposed to estimate SGs from images [20,21] or dense reconstructions [7,8].…”

Section: A 3d Semantic Scene Representationsmentioning

confidence: 99%

See 1 more Smart Citation

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

Looper¹,

Rodríguez-Puigvert²,

Siegwart³

et al. 2022

Preprint

View full text Add to dashboard Cite

Numerous applications require robots to operate in environments shared with other agents such as humans or other robots. However, such shared scenes are typically subject to different kinds of long-term semantic scene changes. The ability to model and predict such changes is thus crucial for robot autonomy. In this work, we formalize the task of semantic scene variability estimation and identify three main varieties of semantic scene change: changes in the position of an object, its semantic state, or the composition of a scene as a whole.To represent this variability, we propose the Variable Scene Graph (VSG), which augments existing 3D Scene Graph (SG) representations with the variability attribute, representing the likelihood of discrete long-term change events. We present a novel method, DeltaVSG, to estimate the variability of VSGs in a supervised fashion. We evaluate our method on the 3RScan long-term dataset, showing notable improvements in this novel task over existing approaches. Our method DeltaVSG achieves a precision of 72.2% and recall of 66.8%, often mimicking human intuition about how indoor scenes change over time. We further show the utility of VSG predictions in the task of active robotic change detection, speeding up task completion by 62.4% compared to a scene-change-unaware planner. We make our code available as open-source.

show abstract

Section: B Variable Scene Graph Prediction Performancementioning

confidence: 96%

Section: A 3d Semantic Scene Representationsmentioning

confidence: 99%

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

Looper¹,

Rodríguez-Puigvert²,

Siegwart³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…• We present extensive evaluation of our proposed solution, as well as an in-depth analysis of the internal working of the proposed solution to explain the utility of commonsense reasoning. This work is an extended version of [4]. The novel contributions with respect to our previous findings are: i) We propose D-SCG that describes the spatial information on the proximity edges via directional relative positions, which improves the previous formulation of the SCG that represents the proximity edges as distances.…”

Section: Introductionmentioning

confidence: 93%

“…It then converts the relative positions into absolute ones which agree on a single final position of the target object. Differently from the previous work that localises object with Spatial Commonsense Graph (SCG-OL) [4], our new D-SCG represents the proximity edges with directional relative positions, instead of the undirected relative distances. This allows us to regress and estimate the target position in an end-to-end trainable manner, without requiring a non-differentiable multilateration procedure for localising the target object, thus contributing to a more effective loss calculation and training procedure.…”

Section: Introductionmentioning

confidence: 99%

“…The novel contributions with respect to our previous findings are: i) We propose D-SCG that describes the spatial information on the proximity edges via directional relative positions, which improves the previous formulation of the SCG that represents the proximity edges as distances. The new formulation removes the need for a non-differentiable multilateration phase when computing the position of the target object as in [4], leading to a 8x speed-up for training and a much better generalisation capability when applied in the 3D domain. ii) We propose a new GNN-based solution with D-SCG to address the object localisation task, achieving a 5.9% increase in the localisation success rate.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Leveraging commonsense for object localisation in partial scenes

Giuliari¹,

Skenderi²,

Cristani³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose an end-to-end solution to address the problem of object localisation in partial scenes, where we aim to estimate the position of an object in an unknown area given only a partial 3D scan of the scene. We propose a novel scene representation to facilitate the geometric reasoning, Directed Spatial Commonsense Graph (D-SCG), a spatial scene graph that is enriched with additional concept nodes from a commonsense knowledge base. Specifically, the nodes of D-SCG represent the scene objects and the edges are their relative positions. Each object node is then connected via different commonsense relationships to a set of concept nodes. With the proposed graph-based scene representation, we estimate the unknown position of the target object using a Graph Neural Network that implements a novel attentional message passing mechanism. The network first predicts the relative positions between the target object and each visible object by learning a rich representation of the objects via aggregating both the object nodes and the concept nodes in D-SCG. These relative positions then are merged to obtain the final position. We evaluate our method using Partial ScanNet, improving the state-of-the-art by 5.9% in terms of the localisation accuracy at a 8x faster training speed.

show abstract

A Survey on 3D Scene Graphs: Definition, Generation and Application

Bae

Shin

et al. 2023

Robot Intelligence Technology and Applications 7

View full text Add to dashboard Cite

Spatial Commonsense Graph for Object Localisation in Partial Scenes

Cited by 12 publications

References 22 publications

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

Leveraging commonsense for object localisation in partial scenes

A Survey on 3D Scene Graphs: Definition, Generation and Application

Contact Info

Product

Resources

About