3D Reconstruction of a Complex Grid Structure Combining UAS Images and Deep Learning

Abstract

The latest advances in technical characteristics of unmanned aerial systems (UAS) and their onboard sensors opened the way for smart flying vehicles exploiting new application areas and allowing to perform missions seemed to be impossible before. One of these complicated tasks is the 3D reconstruction and monitoring of large-size, complex, grid-like structures as radio or television towers. Although image-based 3D survey contains a lot of visual and geometrical information useful for making preliminary conclusions on construction health, standard photogrammetric processing fails to perform dense and robust 3D reconstruction of complex large-size mesh structures. The main problem of such objects is repeated and self-occlusive similar elements resulting in false feature matching. This paper presents a method developed for an accurate Multi-View Stereo (MVS) dense 3D reconstruction of the Shukhov Radio Tower in Moscow (Russia) based on UAS photogrammetric survey. A key element for the successful image-based 3D reconstruction is the developed WireNetV2 neural network model for robust automatic semantic segmentation of wire structures. The proposed neural network provides high matching quality due to an accurate masking of the tower elements. The main contributions of the paper are: (1) a deep learning WireNetV2 convolutional neural network model that outperforms the state-of-the-art results of semantic segmentation on a dataset containing images of grid structures of complicated topology with repeated elements, holes, self-occlusions, thus providing robust grid structure masking and, as a result, accurate 3D reconstruction, (2) an advanced image-based pipeline aided by a neural network for the accurate 3D reconstruction of the large-size and complex grid structured, evaluated on UAS imagery of Shukhov radio tower in Moscow.

Handling Critical Aspects in Massive Photogrammetric Digitization of Museum Assets

“…Moreover, as artefacts are different from one another, segmentation models based on U-Net, Mask R-CNN, etc. (He et al, 2020;Knyaz et al, 2020;Grilli et al, 2021;Minaee et al, 2021) are not suitable.…”

Automatic Anthropological Landmarks Recognition and Measurements

“…To incorporate information about 3D coordinates of the landmark a spatial consistency loss function LSC (A, L0) is introduced. Specifically, similarly to (Kniaz et al, 2020, Knyaz et al, 2020b we add information about predicted location of detecting points as masked image M containing epipolar constrains for NL landmarks detected in the reference image A0.…”

Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques

“…In most cases, the main aim is to carry out an automatic clustering or regrouping of the considered elements in classes characterised by certain common characteristics. More and more frequently, the researchers are exploring these strategies and establishing new methodologies to perform a semantic classification both on images (Knyaz, Kniaz, Remondino, Zheltov, & Gruen, 2020;Stathopoulou & Remondino, 2019) and 3D data, such as 3D models (George, Xie, & Tam, 2018) or point clouds (Grilli, Farella, Torresani, & Remondino, 2019;Grilli, Özdemir, & Remondino, 2019;Pierdicca et al, 2020).…”