A novel no-sensors 3D model reconstruction from monocular video frames for a dynamic environment

Abstract

Occlusion awareness is one of the most challenging problems in several fields such as multimedia, remote sensing, computer vision, and computer graphics. Realistic interaction applications are suffering from dealing with occlusion and collision problems in a dynamic environment. Creating dense 3D reconstruction methods is the best solution to solve this issue. However, these methods have poor performance in practical applications due to the absence of accurate depth, camera pose, and object motion.This paper proposes a new framework that builds a full 3D model reconstruction that overcomes the occlusion problem in a complex dynamic scene without using sensors’ data. Popular devices such as a monocular camera are used to generate a suitable model for video streaming applications. The main objective is to create a smooth and accurate 3D point-cloud for a dynamic environment using cumulative information of a sequence of RGB video frames. The framework is composed of two main phases. The first uses an unsupervised learning technique to predict scene depth, camera pose, and objects’ motion from RGB monocular videos. The second generates a frame-wise point cloud fusion to reconstruct a 3D model based on a video frame sequence. Several evaluation metrics are measured: Localization error, RMSE, and fitness between ground truth (KITTI’s sparse LiDAR points) and predicted point-cloud. Moreover, we compared the framework with different widely used state-of-the-art evaluation methods such as MRE and Chamfer Distance. Experimental results showed that the proposed framework surpassed the other methods and proved to be a powerful candidate in 3D model reconstruction.

Parallel Implementation of 3D Model Reconstruction of Monocular Video Frames in a Dynamic Environment

“…The work in this paper presents the CUDAparallel implementation of monocular 3D reconstruction, which has been proposed in our previous study [14]. The framework aims to generate a smooth and accurate 3D point cloud from a sequence of monocular video frames containing existent moving objects.…”

“…The serial implementation is proposed in [14]. The per-frame point cloud 𝑝 𝑖 is reconstructed by 𝑫 𝒊 is the depth for a single frame, the final Motion 𝑬 𝒎 𝑭 consists of individual moving objects 𝜓𝑀 𝑜 and Camera motion 𝜓𝐸 𝒊→𝒋 .…”

“…Another approach is presented using unsupervised learning and point cloud fusion to reconstruct a 3D scene in a dynamic environment (M3DMR) [14]. This approach focuses on applying unsupervised learning to predict the depth map, camera position, and object motion through a sequence of video frames.…”

“…This work aims to improve the computational time of the M3DMR framework that is proposed in our previous work [14]. Profile and complexity analyses were performed to select the bottleneck time of serial implementation.…”

“…Figure. 1 3D point cloud mapped to 2D image (a) selected input frame (b) Ground truth (c) predicted points[14] …”