Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

Abstract

Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error.

Special Issue on “Augmented Reality, Virtual Reality & Semantic 3D Reconstruction”

“…With the help of mini Unmanned Aerial Vehicles (UAVs), multiview highresolution aerial images of large-scale scenes can be conveniently collected. In [10], Wei et al propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. The graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the reprojection error.…”

3D Reconstruction of a Complex Grid Structure Combining UAS Images and Deep Learning

“…CNN methods [53][54][55][56][57][58][59] have also demonstrated their potential for detecting a numerous number of elements in the images and then boost the processing pipeline in terms of constrained tie point extraction or semantic multi-view stereo [60][61][62]. The advantages of image masking for dense point cloud generation are well known in the literature [62][63][64]. While there are multiple readily available segmenation models for oblique aerial photos [63] or buildings [64,65], the generation of pixel-level semantic segmentation for sparse wire objects is challenging.…”

“…The advantages of image masking for dense point cloud generation are well known in the literature [62][63][64]. While there are multiple readily available segmenation models for oblique aerial photos [63] or buildings [64,65], the generation of pixel-level semantic segmentation for sparse wire objects is challenging. The analysis of repetitive patterns [66,67] allows to partly solve this problem for opaque objects (e.g., skyscrapers).…”

3DTV Mass Adoption in the United States and the National Football League

“…They allowed the viewer to experience emotional bonding when, for example, the own team was playing, thus creating a form of social presence (Shin, 2012). Another scholar perceived real-time conversion from 2D to 3D as a method to ensure enough availability of 3DTV programming (Limbachiya, 2014). For an interim period before full 3DTV mass adoption, it would be thinkable to broadcast both a 2D and 3D version of important live events in parallel to minimize commercial risks.…”