Fusion of thermal and RGB images for automated deep learning based crack detection in civil infrastructure

Abstract

Research has been continually growing toward the development of image-based structural health monitoring tools that can leverage deep learning models to automate damage detection in civil infrastructure. However, these tools are typically based on RGB images, which work well under ideal lighting conditions, but often have degrading performance in poor and low-light scenes. On the other hand, thermal images, while lacking in crispness of details, do not show the same degradation of performance in changing lighting conditions. The potential to enhance automated damage detection by fusing RGB and thermal images together within a deep learning network has yet to be explored. In this paper, RGB and thermal images are fused in a ResNET-based semantic segmentation model for vision-based inspections. A convolutional neural network is then employed to automatically identify damage defects in concrete. The model uses a thermal and RGB encoder to combine the features detected from both spectrums to improve its performance of the model, and a single decoder to predict the classes. The results suggest that this RGB-thermal fusion network outperforms the RGB-only network in the detection of cracks using the Intersection Over Union (IOU) performance metric. The RGB-thermal fusion model not only detected damage at a higher performance rate, but it also performed much better in differentiating the types of damage.

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

“…Due to the rarity of thermal images in image processing, largely attributed to the high cost of recording devices, thermal imaging (TI) is often combined with the RGB modality to enhance resolution [21] or for segmentation and detection purposes [22,23]. However, synchronizing and superimposing the different streams is costly [24] and may not achieve perfect alignment.…”

Sensor Fusion for the Robust Detection of Facial Regions of Neonates Using Neural Networks

“…In each case, the images are taken from a long distance [ 16 ]. The network used by Alexander et al is used for civil infrastructure applications and states that the robustness increases for fusion images [ 17 ]. Jung et al use a neural network for the creation of fusion images out of near infrared (NIR) and noisy RGB images [ 18 ].…”

Thermal image analytics for industrial anomaly detection

“…based on the application and the industry. For instance, work has been done on detection of hotspots in solar panels [19,20,21], detection of cracks in roads, metals and rooftops [22,23,24,25], detection of hotspots in electric equipment or machinery, etc. The motivation for this being the automatic early identification of an oncoming failure, for prompting timely repair, thereby preventing financial, human and resource losses.…”