Deep Learning-Based Damage Detection from Aerial SfM Point Clouds

Abstract

Aerial data collection is well known as an efficient method to study the impact following extreme events. While datasets predominately include images for post-disaster remote sensing analyses, images alone cannot provide detailed geometric information due to a lack of depth or the complexity required to extract geometric details. However, geometric and color information can easily be mined from three-dimensional (3D) point clouds. Scene classification is commonly studied within the field of machine learning, where a workflow follows a pipeline operation to compute a series of engineered features for each point and then points are classified based on these features using a learning algorithm. However, these workflows cannot be directly applied to an aerial 3D point cloud due to a large number of points, density variation, and object appearance. In this study, the point cloud datasets are transferred into a volumetric grid model to be used in the training and testing of 3D fully convolutional network models. The goal of these models is to semantically segment two areas that sustained damage after Hurricane Harvey, which occurred in 2017, into six classes, including damaged structures, undamaged structures, debris, roadways, terrain, and vehicles. These classes are selected to understand the distribution and intensity of the damage. The point clouds consist of two distinct areas assembled using aerial Structure-from-Motion from a camera mounted on an unmanned aerial system. The two datasets contain approximately 5000 and 8000 unique instances, and the developed methods are assessed quantitatively using precision, accuracy, recall, and intersection over union metrics.

GNSS/INS-Assisted Structure from Motion Strategies for UAV-Based Imagery over Mechanized Agricultural Fields

“…Since for the relative orientation, the translation vector can only be determined up to an arbitrary scale, the correction to one of the translation components (say, ∆r x assuming that the baseline between the two images is mainly aligned along the x-axis of the left camera coordinate system) can be set to zero. Moreover, assuming good approximate values for the rotation angles, the incremental rotation matrix can be represented as in Equation (7). Substituting Equation (7) into Equation (6), while ignoring second-order incremental terms, results in a linear equation in five unknown parameters.…”

“…Moreover, assuming good approximate values for the rotation angles, the incremental rotation matrix can be represented as in Equation (7). Substituting Equation (7) into Equation (6), while ignoring second-order incremental terms, results in a linear equation in five unknown parameters. Consequently, given five or more conjugate features, a least-squares solution for the unknown corrections can be derived.…”

“…Digital agriculture, also known as smart farming, aims at increasing food production by enhancing decision making through the analysis of digitally collected data and/or information over agricultural fields [4]. In this regard, remotely sensed data acquired by satellite [5], manned aircraft [6], and, most recently, unmanned aerial vehicle (UAV) [7] platforms, are among the most popular digital data sources that are starting to replace many of the traditional in-field manual trait measurements [8]. The easier deployment and possibility of equipping UAVs with a variety of advanced imaging sensors, along with their capability to collect high temporal and spatial resolution data have increased the use of UAVs for digital agriculture, in applications such as phenotyping [9][10][11][12][13], crop monitoring [14][15][16][17], and yield estimation/prediction [18][19][20].…”

Deep Learning Classification of 2D Orthomosaic Images and 3D Point Clouds for Post-Event Structural Damage Assessment

“…During the training process, the pre-trained weights of these models were further modified to match the user-defined classes. Moreover, a 3D fully convolutional network (3D FCN) with skip connections is developed based on expanding the 3D FCN model proposed by Mohammadi et al [7]. While the goal of 2DCNN is to classify the aerial images based on the most prominent object observed in the images, the 3D FCN with skip connections semantically classifies the SfM derived point cloud.…”

Measuring Transverse Displacements Using Unmanned Aerial Systems Laser Doppler Vibrometer (UAS-LDV): Development and Field Validation

“…There have been several attempts to use a camera-mounted UAS for bridge SHM [ 35 , 36 ]. 3D image correlation on aerial images captured by cameras on the UAS [ 37 ] and close-range photogrammetry or structure-from-motion (SfM) using a UAS [ 38 , 39 , 40 ] has also been investigated for SHM. Using cameras and other devices mounted on UAS for SHM holds the potential to solve the problems related to accessibility in remote locations and hazardous conditions.…”