Using Synthetic Data to Improve and Evaluate the Tracking Performance of Construction Workers on Site

Abstract

Vision-based tracking systems enable the optimization of the productivity and safety management on construction sites by monitoring the workers’ movements. However, training and evaluation of such a system requires a vast amount of data. Sufficient datasets rarely exist for this purpose. We investigate the use of synthetic data to overcome this issue. Using 3D computer graphics software, we model virtual construction site scenarios. These are rendered for the use as a synthetic dataset which augments a self-recorded real world dataset. Our approach is verified by means of a tracking system. For this, we train a YOLOv3 detector identifying pedestrian workers. Kalman filtering is applied to the detections to track them over consecutive video frames. First, the detector’s performance is examined when using synthetic data of various environmental conditions for training. Second, we compare the evaluation results of our tracking system on real world and synthetic scenarios. With an increase of about 7.5 percentage points in mean average precision, our findings show that a synthetic extension is beneficial for otherwise small datasets. The similarity of synthetic and real world results allow for the conclusion that 3D scenes are an alternative to evaluate vision-based tracking systems on hazardous scenes without exposing workers to risks.

An efficient method for producing deep learning point cloud datasets based on BIM 3D model and computer simulation

“…10. This problem may be solved by adding virtual cameras and sensors to the 3D model to simulate Lidar scanning through point cloud simulation software (Blender et al) [29], and setting the location of the virtual cameras and scanning parameters to maximize the restoration of the real scanning scene.…”

Scientometric analysis and critical review on the application of deep learning in the construction industry

“…This cluster encompasses the studies that use existing deep-learning models, such as YOLO, CNN, R-CNN, CNN+LSTM, and Faster R-CNN, for the recognition of diverse construction site activities (Luo et al 2018, detection of unsafe worker behaviour on the construction site (Ding et al 2018), detection of workers wearing PPE (e.g., safety helmet, safety waist) (Fang et al 2018b, Mneymneh et al 2019, Neuhausen et al 2020, and performance evaluation of construction workers on site. Each of these deep-learning models has been shown to achieve high accuracy and speed.…”

Fake it till you make it: Training Deep Neural Networks for Worker Detection using Synthetic Data

“…Generated synthetic images are proven to be effective in improving DNN training. Since labels are extracted automatically in this approach, data generation is a seamless effort (Neuhausen et al, 2020); thus, an unlimited number of synthetic data is accessible as seen in the SYNTHIA dataset (Ros et al, 2016).…”

“…Following the same approach, Mahmood et al (Mahmood et al, 2022) developed a synthetic dataset for training DNN models that estimated the 3D pose of an excavator. In a very recent study by Neuhausen et al (Neuhausen et al, 2020), synthetically generated images are leveraged to train DNNs with a focus on human worker detection and tracking model. A comparative study of the developed model on both synthetic and real-world data identified the synthetic images as a viable solution for vision-based DNN training.…”