Learning 3D Semantic Scene Graphs with Instance Embeddings

Abstract

A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG, a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.

Deep Neural Networks in Video Human Action Recognition: A review

“…The model investigated here is combined with transformer for generating the scene graph by inputting video data and exploring the relationship between humans, the environment(places), and objects. As the novel proposed research, a scene graph mentioned the relationship between objects in the 3D environment [142]. It built the 3DSSG on top of 3RScan, aiming to extract the 3D geometry and depth information of 3D space.…”

“…Rather than the 2D scene graph, the involvement of a diagram is more densely connected and informative in the 3D scene graph. As mentioned in the work [142] of scene graph generation, it allows spatialtemporal analysis of moving objects and human behaviors in a stereoscopic environment. D. RNNs: Gated Recurrent Units(GRUs) and LSTM GRU are utilized for predicting the time sequential movements with multiple variants, including GRU-SVM and GRU-D models.…”

Image Semantic Segmentation Approach for Studying Human Behavior on Image Data

“…At present, it is a hot topic to consider the real dynamic situation in time and space dimensions. Ji et al [27] proposed to capture spatial and temporal information from video and Wald et al [28] presented a new neural network architecture of 3D data to explore the relationship between entities, and returned semantics from a given 3D scene through learning. Fernando et al [29] combined motion dynamics into the image and feed the image into any standard CNN for end-to-end learning.…”

Deep Neural Networks in Video Human Action Recognition: A review

“…The model investigated here is combined with a transformer for generating the scene graph by inputting video data and exploring the relationship between humans, the environment(places), and objects. As the novel proposed research, a scene graph mentioned the relationship between objects in the 3D environment [130]. It built the 3DSSG on top of 3RScan, aiming to extract the 3D geometry and depth information of 3D space.…”

“…Rather than the 2D scene graph, the involvement of a diagram is more densely connected and informative in the 3D scene graph. As mentioned in the work [130] of scene graph generation, it allows video analysis of moving objects and human behaviors in a stereoscopic environment.…”