Reliable Estimation of Deterioration Levels via Late Fusion Using Multi-View Distress Images for Practical Inspection

Abstract

This paper presents reliable estimation of deterioration levels via late fusion using multi-view distress images for practical inspection. The proposed method simultaneously solves the following two problems that are necessary to support the practical inspection. Since maintenance of infrastructures requires a high level of safety and reliability, this paper proposes a neural network that can generate an attention map from distress images and text data acquired during the inspection. Thus, deterioration level estimation with high interpretability can be realized. In addition, since multi-view distress images are taken for single distress during the actual inspection, it is necessary to estimate the final result from these images. Therefore, the proposed method integrates estimation results obtained from the multi-view images via the late fusion and can derive an appropriate result considering all the images. To the best of our knowledge, no method has been proposed to solve these problems simultaneously, and this achievement is the biggest contribution of this paper. In this paper, we confirm the effectiveness of the proposed method by conducting experiments using data acquired during the actual inspection.

Estimation of Degradation Degree in Road Infrastructure Based on Multi-Modal ABN Using Contrastive Learning

“…The studies [ 5 , 6 , 7 , 24 , 25 , 26 , 27 , 28 , 29 ] achieved their tasks by extracting the features of distress images by introducing CNNs. Other previous studies [ 5 , 8 , 9 , 10 , 30 ] target the classification of distresses (e.g., crack, efflorescence or corrosion) that have occurred in infrastructures. In particular, the studies [ 8 , 9 , 10 ] attempted to estimate the degree of degradation in road infrastructures.…”

“…Other previous studies [ 5 , 8 , 9 , 10 , 30 ] target the classification of distresses (e.g., crack, efflorescence or corrosion) that have occurred in infrastructures. In particular, the studies [ 8 , 9 , 10 ] attempted to estimate the degree of degradation in road infrastructures. The estimation of the degradation degrees leads to a judgment of the distress which should be repaired as a high priority and the realization of efficient field operations.…”

“…Various techniques have been proposed to support engineers using supervised deep learning [ 5 , 6 , 7 , 8 , 9 , 10 ]. Studies in [ 5 , 6 , 7 ] have constructed methods based on the convolutional neural network (CNN) [ 11 ] that achieves high performance in generic object recognition.…”

“…Researchers have actively attempted to construct CNN-based supervised models that can automatically inspect distresses in images [ 5 , 6 , 7 ]. Other studies [ 8 , 9 , 10 ] have constructed methods based on the attention branch network (ABN) [ 12 ] to solve the problem encountered by many supervised learning models, including CNN, that is, the failure to determine the reason for the classification results. ABN has a CNN-based backbone and generates an attention map that indicates the regions of interest in distress images.…”

“…ABN has a CNN-based backbone and generates an attention map that indicates the regions of interest in distress images. For road infrastructure distresses, ABN-based methods improve the estimation performance of the degradation degree considering the highlighted regions [ 8 , 9 , 10 ]. Supervised classification methods, such as ABN, were originally proposed for classification in the field of general object recognition, in which a large-scale training dataset, such as ImageNet [ 13 ], is available.…”