Improving visual question answering for bridge inspection by pre‐training with external data of image–text pairs

Abstract

This paper explores the application of visual question answering (VQA) in bridge inspection using recent advancements in multimodal artificial intelligence (AI) systems. VQA involves an AI model providing natural language answers to questions about the content of an input image. However, applying VQA to bridge inspection poses challenges due to the high cost of creating training data that requires expert knowledge. To address this, we propose leveraging existing bridge inspection reports, which already include image–text pairs, as external knowledge to enhance VQA performance. Our approach involves training the model on a large collection of image–text pairs, followed by fine‐tuning it on a limited amount of training data specifically designed for the VQA task. The results demonstrate a significant improvement in VQA accuracy using this approach. These findings highlight the potential of AI models for VQA as valuable tools for assessing the condition of bridges.

Deep learning‐based automatic classification of three‐level surface information in bridge inspection

“…This is because even identical defects can have varying impacts on the bridge's health depending on the structural components involved. Chun et al (2022) proposed a method that can innovatively output integrated information on defects and members in the form of subtitles, showing great application potential (Kunlamai et al, 2023). Although it includes various types of members and defects, it does not consider the issue of future incremental learning.…”