Continuous video stream pixel sensor: A CNN‐LSTM based deep learning approach for mode shape prediction

Abstract

Modal analysis has emerged as a globally accepted tool to formulate and optimize the behavioral functions of engineering structures, which assists in assessing structural failure and laying out a plan for their maintenance. Modal analysis aims at determining the frequencies, damping ratios, and mode shapes of the system under excitation. However, conventional mode shape measurement methods like contact sensors are prone to precision and accuracy issues owing to the sensor's weight and low spatial resolution. In this paper, we improve upon various existing methods for mode shape determination and introduce the idea of a full‐field pixel sensor for mode shape prediction. The proposed computer vision‐based deep learning architecture predicts the mode shape of a vibrating structure with significant precision. Besides, a ModeShape dataset consisting of the vibration recording video and finite element analysis (FEA) based label has been curated. Specifically, we introduce a convolutional neural network, long short‐term memory (CNN‐LSTM) computer vision‐based non‐contact vibration measurement technique for automated mode shape prediction. The key idea is to use each pixel of a RGB camera as a sensor and process the captured spatio‐temporal data to enable mode shape prediction. Our CNN‐LSTM model takes the video streams of a vibrating structure as input and yields the fundamental mode shapes. The proposed technique is non‐invasive and can extract information at relatively high spatial density. The CNN‐LSTM model is proficient by utilizing experimental outcomes. The robustness of the deep learning model has been scrutinized by utilizing specimens of an assortment of different materials and fluctuating dimensions.

Classification of External Vibration Sources through Data-Driven Models Using Hybrid CNNs and LSTMs

“…Te hybrid model of CNN and LSTM can be expressed in three forms, as shown in Figure 3. Te frst one can be called CNN−LSTM [26,46], which indicates that the CNN is used to extract the local features of the signal frst and the LSTM is used to further process the extracted features. Te second one can be called LSTM−CNN [47,48], which means that the LSTM is frst used for the extraction of the overall features of the signal, and the CNN is subsequently used for further processing of the signal.…”

“…Considering that the hyperparameter settings in the models could signifcantly afect the model results, the random search method is used to optimize the hyperparameter combinations for each model. With reference to existing relevant research [26,66], six parameters were set as hyperparameters to be optimized: the number of units in the LSTM, the number of CNN modules, the size and number of convolutional kernels in the CNN modules, the size of the pooling layer, and the number of FCNN units. Considering the capacity of computational devices, each hyperparameter was set at 2 or 3 levels and the sampling space of each hyperparameter is shown in Table 2.…”

“…In signal processing using deep learning methods, the most well-known models include a convolutional neural network (CNN) [19], long short-term memory (LSTM) network [20], and deep residual network [21]. Specifcally, CNN can focus on the local features in the signal, while LSTM can focus on the sequence features in the signal, and both of them have achieved great success in signal identifcation tasks, such as damage identifcation of mechanical components [22,23] and structural health monitoring [24][25][26]. In recent years, many researchers have used hybrid models based on both models to conduct research in the feld of image or time series signal processing, such as daily energy consumption prediction [27,28], daily air and water quality prediction [29,30], fnancial asset price volatility prediction [31,32], and biological and structural health monitoring [33][34][35].…”

Structural seismic response prediction based on convolutional neural networks

“…In recent years, artificial intelligence and deep learning have developed rapidly, with convolutional neural networks (CNNs) being one of the most popular deep learning algorithms used in image classification, object detection, and text recognition [2]. The academic and engineering communities are increasingly applying CNNs to the field of civil engineering, such as structural modal identification, structural health monitoring, and seismic response prediction [3]- [5]. Compared to finite element analysis, CNNs can establish a neural network mapping relationship between the structure and the seismic response to predict the structural response without design data.…”