Semantic 3D Reconstruction of a Scene and Its Effective Visualisation

Abstract

Single-image 3D scene reconstruction is required in multiple challenging tasks including mobile robotics, industrial monitoring and reconstruction of lost cultural heritage. While modern models demonstrate robust resolution of scene in real time with resolution up to 128 x 128 x 128 voxels, visualization of such detailed of a such detailed voxel model is challenging. A model with 128<sup>3</sup> voxels contains 2097152 simple cubes 16M vertices. It is unfeasible for modern hardware to perform visualization of such voxel models in real-time. Hence a voxel model simplification technique is required to demonstrate reconstruction results in real-time. In this paper, we propose a new algorithm for voxel model simplification using predefined camera views. The algorithm reduces a rigid-body voxel model to a shell voxel model. It keeps only the voxels that are visible from the required view. We demonstrate the effectiveness of the proposed algorithm using a case study with a mobile robot and a state-of-the-art SSZ single-photo 3D reconstruction neural network. We generated a real and a virtual scene with various objects including a statue. We use a mobile robot equipped with a single camera to collect real and synthetic data. We train the SSZ model using the collected data. We developed a dedicated visualization software that implements our algorithm. The comparison of the visualization performance for the full model and its reduced version demonstrates that our algorithm allows to increase the performance by 420 times.