3D Object Reconstruction from Imperfect Depth Data Using Extended YOLOv3 Network

Abstract

State-of-the-art intelligent versatile applications provoke the usage of full 3D, depth-based streams, especially in the scenarios of intelligent remote control and communications, where virtual and augmented reality will soon become outdated and are forecasted to be replaced by point cloud streams providing explorable 3D environments of communication and industrial data. One of the most novel approaches employed in modern object reconstruction methods is to use a priori knowledge of the objects that are being reconstructed. Our approach is different as we strive to reconstruct a 3D object within much more difficult scenarios of limited data availability. Data stream is often limited by insufficient depth camera coverage and, as a result, the objects are occluded and data is lost. Our proposed hybrid artificial neural network modifications have improved the reconstruction results by 8.53% which allows us for much more precise filling of occluded object sides and reduction of noise during the process. Furthermore, the addition of object segmentation masks and the individual object instance classification is a leap forward towards a general-purpose scene reconstruction as opposed to a single object reconstruction task due to the ability to mask out overlapping object instances and using only masked object area in the reconstruction process.

Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired

“…By comparing the proposed method with similar visual substitution approaches, optimal performances have been demonstrated, in terms of computational speed in different operative conditions, such as different point-of-view, presence of occlusion, frame rotation, and different illumination. Finally, in [43], the authors described a novel 3D object reconstruction method, based on a modified hybrid artificial neural network, obtaining more precise filling of partial object images and reducing the process noise compared to the YOLOv3 algorithm. Furthermore, the obtained reconstruction is more stable than the results obtained with other reconstruction techniques.…”

Autonomous 3D geometry reconstruction through robot-manipulated optical sensors

“…This means that many more points, compared to those required to fulfil the target sampling density, are collected in some regions of an object, making the merged point cloud difficult to process in timely fashions and to store in physical memories. For these reasons, solutions to down-sample the collected points and obtain a uniform point density across the resulting point cloud are typically found in many works [14,32]. Although down-sampling algorithms have been presented elsewhere, it is worth describing what downsampling and merging algorithms were implemented in this work, for the sake of making the entire incremental 3D reconstruction pipeline as clear as possible.…”

“…Finally, the set B new (subset of B), which contains cubes with only new points, is defined as the difference between set B and set C (Equ. (14)). The cubes belonging to these sets are represented in Figure 2b.…”

Advanced Intelligent Control through Versatile Intelligent Portable Platforms

“…Object segmentation masks instance classification, together with YOLOv3 with improved design. 3D object reconstruction and prediction with an extended YOLOv3 network is addressed through intelligent versatile applications using full 3D, depth-based two streams, especially in the scenarios of intelligent remote control and communications, where virtual and augmented reality will soon become outdated and are forecast to be replaced by point-cloud streams, providing explorable 3D environments of communication and industrial data [ 2 ]. A hybrid artificial neural network for reconstructing polygonal meshes using a single RGB D frame and prior knowledge is proposed.…”