Multi-Sensor Data Fusion for 3D Reconstruction of Complex Structures: A Case Study on a Real High Formwork Project

Abstract

As the most comprehensive document types for the recording and display of real-world information regarding construction projects, 3D realistic models are capable of recording and displaying simultaneously textures and geometric shapes in the same 3D scene. However, at present, the documentation for much of construction infrastructure faces significant challenges. Based on TLS, GNSS/IMU, mature photogrammetry, a UAV platform, computer vision technologies, and AI algorithms, this study proposes a workflow for 3D modeling of complex structures with multiple-source data. A deep learning LoFTR network was used first for image matching, which can improve matching accuracy. Then, a NeuralRecon network was employed to generate a 3D point cloud with global consistency. GNSS information was used to reduce search space in image matching and produce an accurate transformation matrix between the image scene and the global reference system. In addition, to enhance the effectiveness and efficiency of the co-registration of the two-source point clouds, an RPM-net was used. The proposed workflow processed the 3D laser point cloud and UAV low-altitude multi-view image data to generate a complete, accurate, high-resolution, and detailed 3D model. Experimental validation on a real high formwork project was carried out, and the result indicates that the generated 3D model has satisfactory accuracy with a registration error value of 5 cm. Model comparison between the TLS, image-based, data fusion 1 (using the common method), and data fusion 2 (using the proposed method) models were conducted in terms of completeness, geometrical accuracy, texture appearance, and appeal to professionals. The results denote that the generated 3D model has similar accuracy to the TLS model yet also provides a complete model with a photorealistic appearance that most professionals chose as their favorite.

An Improved TransMVSNet Algorithm for Three-Dimensional Reconstruction in the Unmanned Aerial Vehicle Remote Sensing Domain

“…During the algorithmic research, after replacing the FPN feature extraction network model using the AFPN combined with the ASFF feature extraction network model, considering updating the FMT feature matching module of the original network, after checking the related literature, we found that the local feature transform LoFTR [43] network model can satisfy our needs, and this network model also uses the self-and cross-attention layer. This network model also uses a Transformer with self-and cross-attention layers to handle the dense local features extracted from the backbone network.…”

Computer Vision-Based Measurement Techniques for Livestock Body Dimension and Weight: A Review

“…The reconstruction technique based on RGB images generally involves employing a multi-source image fusion approach to acquire the target's geometric shape and texture information [39]. Its procedure typically entails the following steps: (1) capture a set of two-dimensional images containing the target object from various angles; (2) identify and extract features such as key points, corners, edges, etc.…”

Multi-Sensor Data Fusion Method Based on Self-Attention Mechanism

“…where 1 f is a standard 1D convolution operation with a kernel size of one and stride of one. The convolution operation does not alter the size of the feature map; however, the number of channels changes from 3 to 256.…”

“…Its primary objective is to restore the 3D structure of a target object or scene. The continuous evolution of sensor technologies has made it possible to obtain high-quality and accurate point cloud and image data [1]. However, due to the rapid development of digital information and the increasing complexity of neural networks, relying solely on a single sensor is insufficient for capturing the rich features of target objects.…”