A Deep Learning‐Based Approach to Enable Action Recognition for Construction Equipment

Abstract

In order to support smart construction, digital twin has been a well-recognized concept for virtually representing the physical facility. It is equally important to recognize human actions and the movement of construction equipment in virtual construction scenes. Compared to the extensive research on human action recognition (HAR) that can be applied to identify construction workers, research in the field of construction equipment action recognition (CEAR) is very limited, mainly due to the lack of available datasets with videos showing the actions of construction equipment. The contributions of this research are as follows: (1) the development of a comprehensive video dataset of 2,064 clips with five action types for excavators and dump trucks; (2) a new deep learning-based CEAR approach (known as a simplified temporal convolutional network or STCN) that combines a convolutional neural network (CNN) with long short-term memory (LSTM, an artificial recurrent neural network), where CNN is used to extract image features and LSTM is used to extract temporal features from video frame sequences; and (3) the comparison between this proposed new approach and a similar CEAR method and two of the best-performing HAR approaches, namely, three-dimensional (3D) convolutional networks (ConvNets) and two-stream ConvNets, to evaluate the performance of STCN and investigate the possibility of directly transferring HAR approaches to the field of CEAR.

A Motion Capture Data-Driven Automatic Labanotation Generation Model Using the Convolutional Neural Network Algorithm

“…e CNN-based method for generating dance spectrum of lower limbs is proposed in this study, in which the network structure consists of two convolution layers, a pool layer, three fully connected layers, the output of the softmax function is the probability value of the human action category, and finally, the category with the largest probability value is taken as the result of network prediction [29]. In order to prevent overfitting, a dropout layer is connected behind each fully connected layer.…”

Pose estimation method for construction machine based on improved AlphaPose model

“…It has more advantages than sensor technology (Pradhananga and Teizer, 2013;Yang et al, 2011;Brilakis et al, 2011;Teizer et al, 2010). At present, pieces of literature have studied the identification and classification of construction entities (Memarzadeh et al, 2013;Tajeen and Zhu, 2014), the activity recognition of construction entities (Yang et al, 2014;Akhavian and Behzadan, 2015;Kim and Chi, 2019;Zhang et al, 2020;Kim et al, 2018a), the location of relevant entities at the construction site (Kim, 2018;Golparvar-Fard et al, 2013) and the capture of their trajectory (Angah and Chen, 2020;Roberts and Golparvar-Fard, 2019), and the automatic identification of hazards based on the distance and spatial information between identified entities (Fang et al, 2020). Hyojoo et al (2019) presented a vision-based collision warning system based on the automated 3D position estimation of each worker with monocular vision, intending to protect equipment workers from potentially dangerous situations, such as collisions between the equipment and workers in a certain proximity.…”

“…, 2013; Tajeen and Zhu, 2014), the activity recognition of construction entities (Yang et al. , 2014; Akhavian and Behzadan, 2015; Kim and Chi, 2019; Zhang et al. , 2020; Kim et al ., 2018a), the location of relevant entities at the construction site (Kim, 2018; Golparvar-Fard et al.…”

A digital twin-driven deformation monitoring system for deep foundation pit excavation

“…Greif et al (2020) developed a digital twin of silos to optimize in-situ logistics. Zhang et al (2020) created the DT model of construction equipment to enable action recognition. However, the application of DT in deformation monitoring of deep foundation pit excavation is immature and still has some limitations.…”