Correlation-Aware Attention Branch Network Using Multi-Modal Data For Deterioration Level Estimation Of Infrastructures

Abstract

This paper presents a correlation-aware attention branch network (CorABN) using multi-modal data for deterioration level estimation of infrastructures. CorABN can collaboratively use visual features from distress images and text features from text data recorded at the inspection to improve the estimation performance of deterioration levels. Specifically, by maximizing correlation between the visual and text features that provide useful information for the deterioration level estimation, a correlation-aware attention map can be generated. Besides, the text features are also utilized in the final estimation along with visual features improved by the attention mechanism to achieve higher estimation performance. With the losses based on both the estimation performance and the correlation, CorABN can train the entire model in an end-to-end manner. Experiments using distress images and their corresponding text data show the effectiveness of the proposed method.

Estimation of Degradation Degree in Road Infrastructure Based on Multi-Modal ABN Using Contrastive Learning

“…The studies [ 5 , 6 , 7 , 24 , 25 , 26 , 27 , 28 , 29 ] achieved their tasks by extracting the features of distress images by introducing CNNs. Other previous studies [ 5 , 8 , 9 , 10 , 30 ] target the classification of distresses (e.g., crack, efflorescence or corrosion) that have occurred in infrastructures. In particular, the studies [ 8 , 9 , 10 ] attempted to estimate the degree of degradation in road infrastructures.…”

“…Other previous studies [ 5 , 8 , 9 , 10 , 30 ] target the classification of distresses (e.g., crack, efflorescence or corrosion) that have occurred in infrastructures. In particular, the studies [ 8 , 9 , 10 ] attempted to estimate the degree of degradation in road infrastructures. The estimation of the degradation degrees leads to a judgment of the distress which should be repaired as a high priority and the realization of efficient field operations.…”

“…Various techniques have been proposed to support engineers using supervised deep learning [ 5 , 6 , 7 , 8 , 9 , 10 ]. Studies in [ 5 , 6 , 7 ] have constructed methods based on the convolutional neural network (CNN) [ 11 ] that achieves high performance in generic object recognition.…”

“…Researchers have actively attempted to construct CNN-based supervised models that can automatically inspect distresses in images [ 5 , 6 , 7 ]. Other studies [ 8 , 9 , 10 ] have constructed methods based on the attention branch network (ABN) [ 12 ] to solve the problem encountered by many supervised learning models, including CNN, that is, the failure to determine the reason for the classification results. ABN has a CNN-based backbone and generates an attention map that indicates the regions of interest in distress images.…”

“…ABN has a CNN-based backbone and generates an attention map that indicates the regions of interest in distress images. For road infrastructure distresses, ABN-based methods improve the estimation performance of the degradation degree considering the highlighted regions [ 8 , 9 , 10 ]. Supervised classification methods, such as ABN, were originally proposed for classification in the field of general object recognition, in which a large-scale training dataset, such as ImageNet [ 13 ], is available.…”