Visual Graphs from Motion (VGfM): Scene Understanding with Object Geometry Reasoning

Abstract

Recent approaches on visual scene understanding attempt to build a scene graph -a computational representation of objects and their pairwise relationships. Such rich semantic representation is very appealing, yet difficult to obtain from a single image, especially when considering complex spatial arrangements in the scene. Differently, an image sequence conveys useful information using the multi-view geometric relations arising from camera motions. Indeed, object relationships are naturally related to the 3D scene structure. To this end, this paper proposes a system that first computes the geometrical location of objects in a generic scene and then efficiently constructs scene graphs from video by embedding such geometrical reasoning. Such compelling representation is obtained using a new model where geometric and visual features are merged using an RNN framework. We report results on a dataset we created for the task of 3D scene graph generation in multiple views.

Learning 3D Semantic Scene Graphs with Instance Embeddings

“…Only a few works have explored scene graphs in 3D. Gay et al (2018) propose a 2.5D graph dataset based on ScanNet (Dai et al 2017), Armeni et al (2019), on the other hand, suggest hierarchical 3D scene graphs. They split the different components of a scene into 4 different layers: cameras, objects, buildings and rooms.…”

“…Additionally to the data, Armeni et al (2019) and Gay et al (2018) propose graph prediction methods. Armeni et al (2019) sample images from a panoramic camera and apply a regularization technique to 2D mask predictions aiming to obtain improved 3D object nodes.…”

“…Although, 3D graphs have been used in computer graphics for decades to store 3D mesh data, the respective edges usually do not represent semantic connections but rather relative transformations such that when a parent node is relocated, the change is applied in a hierarchical fashion to all child nodes. Only recently, semantic scene graphs have started to emerge in the 3D context (Gay et al 2018;Armeni et al 2019;Rosinol et al 2020b). Armeni et al construct graphs for buildings, including rooms, major objects, camera views and the relations between these entities (Armeni et al 2019).…”

“…Rosinol et al (2020b) incorporates dynamics to this representation by additionally considering moving humans. Both (Armeni et al 2019) and (Gay et al 2018) propose multiview graph prediction methods based on 2D masks (Armeni et al 2019) and object detection networks (Gay et al 2018). They estimate graphs from images while we operate on 3D data directly.…”

A System of Associated Intelligent Integration for Human State Estimation

“…Additionally, we integrate the results of classifiers by using rule-based architecture for estimating multi-modal information. This rule-based integration architecture is inspired by the methods which utilize structured knowledge to learn from small amounts of experience for scene understanding [7]. Fig.…”