Image‐based monitoring of bolt loosening through deep‐learning‐based integrated detection and tracking

Abstract

Structural bolts are critical components used in different structural elements, such as beam-column connections and friction damping devices. The clamping force in structural bolts is highly influenced by the bolt rotation. Much of the existing vision-based research about bolt rotation estimation relies on traditional computer vision algorithms such as Hough transform to assess static images of bolts. This requires careful image preprocessing, and it may not perform well in the situation of complicated bolt assemblies, or in the presence of surrounding objects and background noise, thus hindering their real-world applications. In this study, an integrated real-time detect-track method, namely, RTDT-bolt, is proposed to monitor the bolt rotation angle. First, a real-time convolutionalneural-networks-based object detector, named YOLOv3-tiny, is established and trained to localize structural bolts. Then, the target-free object tracking algorithm based on optical flow is implemented to continuously monitor and quantify the rotation of structural bolts. In order to enhance the tracking performance against background noise and potential illumination changes during tracking, the YOLOv3-tiny is integrated with the optical flow tracking algorithm to redetect the bolts when the tracking gets lost. Extensive parameter studies were conducted to identify optimal tracking performance and examine the potential limitations. The results indicate that the RTDT-bolt method can greatly enhance the tracking performance of bolt rotation, which can achieve over 90% accuracy using the recommended range for the parameters.

3D vision‐based out‐of‐plane displacement quantification for steel plate structures using structure‐from‐motion, deep learning, and point‐cloud processing

“…For these reasons, a multi‐view vision‐based 3D object detection method is developed to detect structural components from an unorganized 3D scene cloud. This method builds upon a CNN‐based object detector, YOLOv3‐tiny, which is adopted from Pan and Yang (2021). Compared to traditional image‐based object detection algorithms, one advantage of CNN‐based object detection algorithms is their robustness in localizing objects of interest in relatively complicated external environments, where many irrelevant objects and background noise are present.…”

“…In recent years, both DL‐based and non‐DL‐based computer vision methods have been proposed for damage detection of different types of structural systems, such as reinforced concrete structures (Gao & Mosalam, 2018; Liang, 2019; Pan & Yang, 2020; Sirca & Adeli, 2018), steel structures (Cha et al., 2018; Kong & Li, 2018; Yeum & Dyke, 2015; Yun et al., 2017), masonry structures (Wang et al., 2018, 2019), structural bolted assemblies (Cha et al., 2016; Pan & Yang, 2021; Ramana et al., 2019). In general, compared to non‐DL‐based methods, DL‐based methods provide higher accuracy and are much more robust against background noise in image classification, object detection, and semantic segmentation.…”

Vision-Based Detection of Bolt Loosening Using YOLOv5

“…The average accuracy of the method is respectively 100% and 98.1%. Pan et al [ 47 ] proposed an RTDT-bolt method by combining YOLOv3-tiny with optical flow method. The method achieved real-time detection and tracking of bolt rotation with an accuracy of more than 90%.…”

Post‐disaster damage classification based on deep multi‐view image fusion

“…Cha et al (2017) proposed a vision-based DL model to detect damage in the form of cracks in concrete buildings. Pan and Yang (2021) developed a real-time DL model for detection and tracking of bolt loosening and rotation assessment. Liang (2019) performed the post-disaster damage assessment of bridges using DL.…”